Thank you for the response
The slowdown was actually without any RBL lookups.
Now I've increased SA to level 5 and 3 RBLs and the speed is the same as without any lookup and default 8 for SA.
As for virus scanning, I only have it turned on for incoming emails not outgoing, so I assuem it's not scanning those, even though I wish there was a place for me to see total scans, not just found virii so I can see that it's truly scanning everything.
I'll verify myself when I'm onsite tomorrow that local users do not experience delay. As for me, my remote location is 20 Mbps fibre and the server is on a dedicated T-1, so it's for sure not net speed.
So, I think I tried everything else on your list.
Does anyone else have this sort of problem still? Is this problem reported in 6.5RC1? If not, then maybe I'll upgrade ... again just to get rid of this issue.