Topic: SME Server 6.0.1 hacked - Help would be appreciated

[jonroberts]

I have a client with an SME Server (6.0.1) that was hacked and used to relay Spam.  It has been left with the var/log directory deleted and I get a Segmentation Fault when running certain commands (e.g. mkdir).

The server runs a number of contribs.  I guess the least standard of them being my own Cyrus-Imap contrib.  Apart from that the other contibs are:
RAID monitor, BackupToWorkstation, SpamAssassin, ClamAV, System Monitor & Sarg.

The Server connects to the Internet with a fixed IP through a Netgear ADSL firewall / router which is set up to only allow these services to reach the SME Server:
PPTP (1723), HTTP, HTTPS amd SMTP.  Additionally SSH and FTP are allowed but only from a specific IP address range (Our own network for support).

Because of that the SME server was set to allow public SSH and VPN access, although only 1 account was set to VPN enabled in Server Manager.

If its a vulnerability in the Cyrus (not that I'm aware of one) & they got in via port 25 then I could understand access for Spam, but they must have got root access (I assume) to delete the var/log directory.

I have a recovery to do, but without knowing how they got in, how do I protect against this in future?

(all anti-virus is up to date on the client site, as far as I am aware)

Any comments / advice gratefully received.  I understand the sensitivities of discussing security issues in a public forum, so I'll also e-mail contribs directly.

I can also be contacted directly at jon AT westcountrybusiness.com if there's anything you don't want to write via the forums (& for anyone concerned about this, I will update this post with a safe version of any tips I find useful if that happens)

[gordonr]

I have a client with an SME Server (6.0.1) that was hacked and used to relay Spam.

Please report security issues to security@contribs.org and not the boards in the first instance.

Thanks,

Gordon

[jonroberts]

Following on from the above problem, I now have the server for a rebuild.  On reboot, the system would not restart, listing errors like:

unable to aquire log\supervise\lock: read-only file system

(The same message repeats for lots of different files)

I've tried booting in single user & tried an update from the original 6.0.1 CD, but no luck.

I know the data is OK as it was working before the reboot, but I can't get at it.  The data is on a pair of  IDE drives configured as RAID (using SME raid).  

How do I mount them on another server?  I'm really having trouble here, so any tips would be very welcome.

[raem]

jonroberts

> The server runs a number of contribs.  I guess the > least standard of them being my own Cyrus-Imap
> contrib.  Apart from that the other contibs are:
> RAID monitor, BackupToWorkstation, SpamAssassin,
> ClamAV, System Monitor & Sarg.

You didn't mention if you run any php type applicatiosn eg phpmyadmin, phpBB forums, OsCommerce etc etc. Those type of apps are a likely source of the breach.



> I have a recovery to do, but without knowing how
> they got in, how do I protect against this in future?

Initially by sending all information to the security address gordon gave you and see if something needs to be fixed in sme server rather than in applications.

Also see if you are running versions of apps that have known or unknown(at present) vulnerabilities. They should all be updated immediately where appropriate.


Your server is compromised and you should NOT continue using it and I assume it has been disconnected from the Internet already.

A complete rebuild is the only safe answer with restoration of data ONLY from known good backups.
 
If there is data you need to get from the compromised server it should be minimal only and carefully screened to ensure no lurking hacker code time bombs are in the data.

On a practical note remove and put aside one of the RAID drives, you can look at that later for forensic follow up or vital data recovery.

Get a replacement second drive and do a clean rebuild of the server, at least the users will have something to use in the meantime. Unless you have a good backup (tape or otherwise) that you can fully restore your system from, then just tell the users that data & services will gradually be restored over the next few days/hours ?

I've been through it recently and you can waste a lot of precious time trying to fix/diagnose the old box, it's not worth the effort and still will have security risks associated with it. A clean install is by far the best approach. Keep a drive though to look at when time permits after the emergency has finished.

When the new box is up and running disable as much external access as possible, use public private keys rather than normal passwords for ssh access, disable root ssh access (only allow user ssh access via public private keys) and also only allow access from local network & VPN for remote connections.

Good luck

[jonroberts]

Gordon / Ray,

Thanks for the tips.  The server is up and working again now.  Unfortunately I did this before seeing Ray's tip on keeping one of the drives - Ah well ...

I have been exchanging e-mails with Gordon via security AT contribs & will update him with full details of the breach as I understand it.  

The server was not running any other PHP apps, only the basic 6.0.1 install & the contribs listed (plus backup2ws, which I forgot to list).

The rebuilt server has been set as secure server & gateway & I have also set the Netgear ADSL firewall to block all incoming.  We can live with that, but I would still like to understand how this could have happened.

If I ever find out, I'll update the post (assuming its not too sensitive ...)

[thedude]

Another route of entry is for someone to have simply guessed one of the user/passwords on the box. If they guess that then they have access to the box - ftp, ssh, etc. How secure were your passwords?

There have been a lot of reports lately of ssh "attacks" where someone sends repeated login attempts with different usernames. I've watched that happen on several servers.

It could also be possible that someone gained access through another computer on your network, i.e. a windows computer infected with a trojan. And of course, we won't even get into the keystroke loggers snagging passwords, etc. etc.

Also, don't underestimate the possibility of an infected windows box blasting out emails through your sme server. That could make the server look like it is relaying. I have a network where several windows boxes got the sober bug, and blasted out so many emails that it somehow corrupted qmail on the sme server.

[jonroberts]

thedude ..

Thanks for the suggestions.  I don't think it was a virus problem as all PCs were running up to date anti-virus (I checked as I've seen this happen on another installation).  Also, if it was only relaying mail from an internal PC, I can't see how this would have resulted in files & directories being deleted from the server.

I think guessing a password may have been more likely, as the passwords used on this site were generally only 4 - 7 letter words.  The server was set to allow PPTP, but only 1 user was set to allow entry this way (via Server Manager). That would mean guessing a user name & password combination - or is root also allowed as a user for PPTP access?  

I guess that SSL access would be most likely, but the firewall rules in the ADSL router (linking the SME server to the Internet) blocked SSL access to all except 1 IP address - our own external IP - for support access.  So I'm not sure how this would be possible.

[raem]

jonroberts

> I don't think it was a virus problem as all PCs
> were running up to date anti-virus (I checked as
> I've seen this happen on another installation).  

Did you actually run a full virus scan on each workstation ?
You should also update and run a full scan using both Adaware & Spybot (or something better if you have it) on all workstations.
Just because the virus definitions are up to date doesn't mean there is no virus or malware on one or more of your workstations.
A local root breach may have been obtained that way.

You need to do all of the above to rule out the possibilities, so the focus of attention can go fairly to the server.


> the passwords used on this site were generally
> only 4 - 7 letter words.  

Short "words" only are a very bad idea. Ideally you should include capitals and numerics and even special characters

On the subject of passwords have you changed all user and admin/root passwords etc ?
As your breach method seems to still be unknown, it may be a very good idea, and make them stronger at the same time.


>...is root also allowed as a user for PPTP access?  

No as far as I know, only VPN enabled users.


> I guess that SSL access would be most likely, but > the firewall rules in the ADSL router (linking the > SME server to the Internet) blocked SSL access to > all except 1 IP address - our own external IP -
> for support access.


It sounds like you have a reasonably tight (security wise) system except for the weak passwords, but as you say access is limited. Are you sure there were no web sites using php (or similar) with weak insecure coding that could have been breached ?

[smeghead]

.. I assume any and all security updates had been applied, specifically ssh springs to mind ...

[jonroberts]

Smeghead,

Could you let me know which security updates are required?  The server was running the packages from the 6.0.1 ISO and I have just double-checked the 6.0.1 updates section & there aren't any updates for SSL.

Is there some other source of updates for the SME server that I should be monitoring & if so, does that mean that the current 6.0.1 verion lacks current security updates?

I would be surprised if this were the case as I have searched the forums quite extensively as a result of this problem & haven't found anything that would indicate known problems or suggesting alternative sources of updates apart from contribs.org, but I'd appreciate a 'heads up' if I've misunderstood the situation.

[smeghead]

ahh, Torquay, the holidays of my youth

Check out the SME6.x update script:

http://no.longer.valid/phpwiki/index.php/SME6.0.1Contribs

This updates lots of modules and also adds quite a few nice extensions.  You can use the script to d/l and then just install what you want manually or just let the script do its stuff.

HTH

[thedude]

I don't think it was a virus problem as all PCs were running up to date anti-virus (I checked as I've seen this happen on another installation).

One of the fun things I've discovered in fighting viruses in the last few years is never to rely on the installed antivirus scanner. It can be completely up to date and still miss dozens of viruses.

You need to scan a PC with a minimum of 3 different scanners to be only mostly sure that it isn't infected. We pull harddrives, scan in another pc, and we use 3 different scanners - Kaspersky, Panda, and BitDefender. And then we may also use TrendMicro's. Only the first is installed, the others are online scanners. Many times each scanner will find additional viruses that the others didn't.

Also, if it was only relaying mail from an internal PC, I can't see how this would have resulted in files & directories being deleted from the server.

Relaying mail may not have been the only thing it was doing. It could have been relaying, and giving remote access. And remote access would allow a hacker to delete directories.

[soprom]

Hi Jon,

Patches were indeed needed, but beware! Using the smeplus script will break cyrus authentification. Search the forums about this (DB3 vs DB4 I think). Dungog has a some rpms for updating and also sme.swerts-knudsen.dk

My personal approach with cyrus on sme is to set it up in a box doing nothing else but mail serving. Anything else being served from a diffirent machine using almost all the security updates I can find.

I wonder if the 6.5 could be put in a production role though.

[irule]

I installed the update script today on my 6.0.1 server-gateway, with some packages installed.
and it broke my mail!!

mail coming in is refused, and mail going out get an smtp-server error.

everything else seems to work.....

I disabled the virus scan and spam filter, and deleted everything in the rbl list.

rebooted several times, but no succes.....

***mmm, posted in the wrong place ****

[ngomes]

The SME Server development (aka, the new releases) and maintenance (aka, the updates) depends entirely on the Contribs.org community.

Just to keep all of you up to date, Ian Wells, Floyd Hartog, Dave Kainer and Matthew Copple (sorry if I left someone out) are the people trying to give to this community the SME Server 6.x maintenance and bugfix updates and  bring to live the SME Server 6.5 final stable release.

Contribs.org needs your help on this project.
What can you do for Contribs.org?

# Read the maintenance process:
http://no.longer.valid/phpwiki/index.php/Maintenance%20Process

# Join the devinfo mailing list and offer your help to test, debug, etc:
http://lists.contribs.org/mailman/listinfo/devinfo

# Go to the Contribs.org Bug Tracker and study some of the listed bugs with new or feedback status, simulate them, give your feedback, try to find some sort of solution. Also if you have some packager skills try to build some rpm packages to the listed bugs with resolved or closed status:
http://no.longer.valid/mantis/view_all_bug_page.php

Finally, take these thoughts into seriously consideration (taken from Charlie Brady, a SME core developer):

If maintenance of distribution updates is not a "core role" for contribs.org, then what is?

Don't ask what contribs.org can do for me, ask what I can do for contribs.org.

-Nuno