CrackerBoy,
Well... I assume you opened the firewall to all yourself access to the POP or IMAP, but I am guessing you did not enable public access. When working int he GUI interface (server-manager) you are offered the ability to email pubic access, but because security is important to all of us, the protocols are POP3S and IMAP3S.
Did you enable these on your server?
Don't want to do that? Then try establishing a VPN before retrieving your email.