Outbound traffic
If you have only one ACTUAL server then it is up to pfsense to route outbound to the smarthost.
You have to use a multi WAN setup for that eg
https://www.netgate.com/docs/pfsense/routing/multi-wan.htmlThere are several different options to do that with load balancing/failover etc etc.
That's all fairly straightforward.
The only issue is if each different ISP requires you be using their actual link when you login/send email. If they don't care where the connection comes from because it has to be authenticated then you are OK.
Inbound traffic
Your MX records should be OK. If an upstream server fails to deliver to 1.2.3.4 then it will pick the next IP, and then the next. Your server knows it hosts mail for domain.tld so should accept it (mine seems to)
The tricky part is with www or imap/mail
If say 1.2.3.4 goes down, where will enquiries go for
www.domain.tld, or email clients connecting to imap.domain.tld ?
The only way to solve that is with round robin DNS as I mentioned earlier. It seems you can use CNAMEs but you just have to be careful with MX records.
For example, assume two different A records, example.com pointed to 1.2.3.4 and example.com pointed to 1.2.3.5. We would then have a www.example.com CNAME record that aliases to the root records of the domain
So you would have:
A 1.2.3.4 domain.tld
A 1.2.3.5 domain.tld
A 1.2.3.6 domain.tld
CNAME
www.domain.tldCNAME imap.domain.tld
Then your own records as you mentioned:
Record A
server.domain.tld 1.2.3.4
server2.domain.tld 1.2.3.5
server3.domain.tld 1.2.3.6
MX record
10 server.domain.tld
20 server2.domain.tld
30 server3.domain.tld
Something like that should fix it.