perhaps with VPN...
In server-manager/E-mail, set 'POP3/Imap' to 'private' and Webmail to 'disabled'
Then give the 'selected users' VPN access (see wiki docs). Assuming your server is on the gateway.
Regards
[Edit]
'users' & 'selected users' will have to use an email client to access mail.