The reasonable safe way to do it from all internet is to connect via putty and then to set up a ssh tunnel from putty and then log in via the internal address.
It's also quite easy to make the server-manager panel public available for all internet but this info will make moderators and others quite mad. (Have posted it before so you could xxxxxx for it.)
A ssh tunnel should be a relatively safe way, and it works without a problem.
By the way if you want to make access from only one ip and not the whole internet I think this one ip option is allso supported from the server-manager panel ..