Yes I have read this.
No, you didn't. Because:
However I can't find an IPSEC contrib for 10.1.
smeserver-libreswan contrib page says:
Libreswan is a free software implementation of the most widely supported and standarized VPN protocol based on
("IPsec")
<sigh>
In getting to this mess I did the following...
installed smeserver-bridge-interface + dependencies
installed smeserver-softethervpn-server + dependencies
installed smeserver-openvpn-bridge + dependencies
installed smeserver-libreswan + dependencies
installed smeserver-phpki-ng + dependencies
So instead of trying to solve a situation you just threw the kitchen sink at it?
Uninstall them all. Remove the ipsec key.
L2TP is (should be) deprecated as seriously insecure. Do not use it.
You should use openvpn (probably the routed version) or wireguard which is available on v10.
I want to have remote access to my server to access email on the server as it is only accessible when on my local network. (This is deliberate)
So you will allow vpn access but not secure email access using imaps? Ok....
No idea what mode your server runs or if you have a router so can't advise you on how to set it up.
You'll need to read the docs properly. In server gateway it will set up the correct ports.
In server only you need to understand what ports you should permit on your router.