moo2
....the SME Firewall is down while using "server only" mode. Is it possible to get more security?
If you use a properly & correctly configured firewall then all should be well
The firewall in sme server can be enabled/tweaked as you wish/desire. There have been extensive discussions about this in the forums.
Any time you digress from standard configurations, you take on the onus of responsibility to ensure everything is done correctly & securely.
For example using fail2ban (block ip addresses after x failed login attempts) to avoid brute force attacks which could try to get the password?
SME server is very robust & security is taken very seriously by the developers and there are many "behind the scenes features" that are protecting you, if you understand & use sme server correctly.
If you wish to use additional layers of security, then perhaps a "better" approach would be to reconfigure your sme server in server & gateway mode, and use the static IP option to create a dmz.
It is covered in the manual and has been discussed various times in the forums, so do a search.
As far as "brute force attacks" go, you are much better off to only use secure login methods that will resist brute force attacks eg for ssh only use Public Private keys & do not allow password only login. This will be a far better security model than allowing password login & then monitoring IP's and banning them.
For server manager access, simply do not enable external access, not even for specific IP's. Configure remote administration to use ssh/Putty to create a secure tunnel to port 443, then login via browser to localhost. That way it is impossible for external users to access server manager without first establishing a secure ssh login, which itself is only accessible to users having the PPKey setup.
For email, use only secure IMAP, configured in server manager.
If users want access to ibays etc, then setup VPN access for them, or use ssh access (WinSCP etc) via PPKey and jail the users to a specified ibay using the remoteuseraccess contrib.
Avoid insecure ftp, just do not allow users to have that sort of access.
...And so on for whatever access you require, generally speaking use secure methods that do not rely upon passwords only, so no amount of password cracking attempts will ever be successful.
And of course, have long & secure passwords for user accounts, a minimum of 7 characters long (the longer the better), using upper & lower case, numerals & special characters, and not "plain english" words or phrases or common items eg birthdays, phone numbers etc.
Another area that will have far more security implications than the core sme server, is installed applications, particularly php contribs/apps. These must be kept up to date with the latest bug fixes etc, to avoid security loopholes etc. There can also occasionally be issues where a security "issue" in one app, utilises a security issue in a component installed on sme server. Neither on it's own would allow a security breach, but when combined a clever hacker can bring down your server. It is indeed rare, but has happened in the past. The moral of the story is to keep your sme server operating system fully updated at all times, and subscribe to mailing lists for any installed apps, and keep them updated as soon as you receive notification that updates & bug fixes are available.
Following these protocols will be a far more effective way to stop hacking than say using fail2ban. I'm not saying not to use it, but just saying make sure you have firstly addressed other more "likely to be vulnerable" aspects of server usage.