What are you trying to accomplish? If you're trying to access a remote SME server from a specific workstation you'd want to set up the PPtP connection from that workstation, not from the local SME server. Or is the workstation non-Windows? As for IPSec, I've had a production VPN running across half a continent since shortly after 5.1.2 came out. It didn't work until I got *everything* exactly right in the setup, then it did work and has continued to do so, reconnecting automagically after all outages scheduled & unscheduled (knock wood!). The only thing that doesn't work is NetBIOS browsing but we've gotten around that.
Looking back I see you have a dialup on one end. How about a few more details? Do you have SME boxes acting as gateways/routers at both ends, or is there something else doing the routing? You'll be happiest if the SME boxes have that role. Believe me, it will work when you find the magic combination of settings in the IPSec VPN servermanager panel.