I have actually replaced qmail in my systems (at ScanMailX) basically since I could modify them as I wanted. I have written an outbound mailer based on qpsmtpd with queues in MySQL to give me 100% control.
Somewhere someone mentioned that using TLS outbound would be a challange with self-signed certificated. I am not so sure that is a problem as they seemed to work (as I remember).
Again, inbound the SME supports SPF, DKIM and DMARC check and allows for TLS connected with PFS (I think its enabled in SME 9.x not?). To go all crazy one can add DNSSEC and the DANE but the two latter properly wont add a lot of extra security as it is not widely accepted/supported.
I can find this patch for qmail but do not have a build environment.
http://inoa.net/qmail-tls/What you ultimately want it all green like this:
https://ssl-tools.net/mailservers/swerts-knudsen.dkHaving made all this, I think that the biggest challanges are not whether we are being listened in on (hence TLS) but whether the mail system allows for "bad" emails with nasty stuff to come through. The standard SME with all the available DNSBL cannot prevent attachments to come through and the users are extremele stup.... - they keep on opening "Invoice XXX" or "Itenary XXX" XLS/DOCX files. The system can also use an improved attachment filter beyound the one we have today.
So maybe a community based attachment HASH system (alá VirusTotal) where an attachment can be checked up against would be awsome. I did start the smeoptimizer (
https://wiki.contribs.org/SMEOptimizer) to stimulate us to help each other. Initially with a DNSBL but this could be extended...