This shouldn't be a problem. I'm doing it already for some of my users who either won't use IMAP, or won't leave the messages on the server.
Visuses shouldn't be a problem, or are a pretty minor concern, in regards to the server. Most of the email viruses are scripting viruses that affect MS Windows boxes, and therefore can't effect Linux based ditros. That isn't to say there aren't viruses targeted at Linux machines, but the threat is low in the configuration you are talking about.