No, you're quite right, Zoi
The user-manager is a contrib.
And putty (or equivalent) is required to create the secure link. It's parallel to the server-manager access.
The only remedy I know of, is to add your external users' networks to local networks in server-manager, but that is NOT recommendable.
Lets see what possibilities 6.5 or later versions offer, once the kinks have been untangled.