I had two experiences similar each one triggered by other reason
1. please check the memory
I had the same behaviour with low memory
when the swap is full the server become extremely slow
any mail which must be parsed by spamassasin and clamav require even 10 minutes
2. look for high mail traffic generated by spam/virus
also you can log to the logs of the email flows (IN and OUT)
(I recommend to look at all in the same time - several putty connections)
for incoming/outgoing
tail -f /var/log/qmail/current
here you must look at
status: local 0/10 remote 2/20
mine for example has no mail to send to local an d 2 for remote recipients
do not mix this with queue (this are the actual jobs)
also one, some or all of this
tail -f /var/log/imap/current
tail -f /var/log/imaps/current
tail -f /var/log/pop3/current
tail -f /var/log/pop3s/current
this depend on your mail client configuration (POP3, POP3s, IMAP, IMAPS)
you must look after this kind of info
tcpsvd: info: status 60/80 - number of session of that protocol from maximum allowed
tcpsvd: info: concurrency 16540 a.b.c.d 41/60 - number of session of that protocol from the same IP
from this you can see if your server has to many jobs to do, or it is "aggressed" by an computer which could be virused
to see if an local virus is not involved you must look also in the log of spamassasin and clamav