It should quite well be possible to access your server manager with IE7. At work I am able to access the server manager on my SME box at home.
From what you describe, it seems that the ip address from which you are trying to access your server manager from, is not allowed to access your server manager remotely.
On the remote location check your external ip address (for example go to
http://www.whatismyip.com and check this with the list of ip addresses at "Remote access" --> "Remote Management".