If your internet connection is via a dynamic IP address, most recipient mail servers will reject your direct connections, so you have to use your ISP's SMPT server. What is wrong with that?
If your users are connecting to you server remotely, then they will not be able to send mail via your server - they will have to connect to their ISP's SMTP server - this is to prevent mail relaying and is a documented feature of SME Server