Hi there,
Firstly, you can only access the server-manager via HTTPS, not standard HTTP.
Secondly, you need to allow external access to the server-manager via the server-manager from within your local network. This can be done through the server panel by going to Remote Access > Remote Management and adding the IP address of the machine you wish to use when accessing it externally. I currently have mine set to only allow access from my workplace's IP.
As an alternative, you can always enable VPN and VPN into your network in order to gain access to the server-manager. The choice is yours.
I believe that this is all covered in the documentation

Hope this helps,
Arthur