I have this problem too,
Rebooting or even reinstalling the server won't solve the problem.
I've had the problem since version 4.1.2 when i started using e-smith/sme-server
I have no clue of what is causing these problems, they just occur once in a while, after a few minutes evrything is fine again. Rebooting the service doesn't do much, it has the same result as being patient.
I use xs4all btw, hwat isp do you have?