As configured by default SME is very secure.
If you have changed any of the default settings you should tell us so that we can give you specific recommendations on how to see what might have been compromised by those changes.
If you are having a specific problem you should tell us what it is so we can discuss possible causes & solutions.
Any spam delivered by your SME server should generate an entry in /var/log/qmail/current.
"rkhunter" is scheduled to run daily to check for rootkits - results are emailed to the local admin, and can also be found in /var/log/rkhunter.log
(both of these logs can be viewed from the server-manager).