Maybe my networking knowledge is worse than I feared... but if I'm restricting remote server access to a single IP address, and with a secure password, then isn't that plenty safe? I know IPs can be spoofed, but they'd have to know the exact IP address and the password to gain access wouldn't they?
Don't get me wrong, the SSH + elinks is a great tip, but I'd rather use the web interface any day.