mbachmann is correct. Under server manager/security/remote access, check the secure shell access. You should have yours set to Allow access only from local networks. I just tested this on a server I remotely manage. I changed the settings to only allow local and I was denied ssh access to the box.
Now, if you have added any networks in the local access section, then those networks will still be able to ssh to the server.
GL,
JB