I think you should focus on get your e-mail delivered where you wanna to be... not to just receive from 4 IP addressess.
BUT, if you choose that, you should do this on firewall rules. Are you using your SME as server-gateway with internet directly connected to external NIC ?
If not, you should block IPs on firewall/gateway/router/???