I can't speak for anybody else but my setup seems to work as expected.
If I log into server-manager, close down my browser (IE 6) and then attempt to get into server manager again, it prompts me for the password again.
This seems inline with any other session based password protection scheme.
PS: Please note that in the case of some browsers, all windows need to be closed before the browser will dispose of the login details.
Regards,
Luke