Topic: Lot of smtp activity but i can't send mail...

[cipandales]

Hello !

I can not send mail outside local network but i can receive all mail (gmail, yahoo etc).
I checked /var/log/qpsmtpd/current and there's a lot of strange mail addreses (like in capture below).
I didn't find an internal ip (in the /var/log/qpsmtpd/current) wich can make spam and now all my network computers are stopped. And i have just apple stuff.

Please, help me with this.

Thank you

2011-06-08 20:22:48.000082500 12998 queue::qmail_2dqueue plugin (queue): (for 12763 ) Queuing qp 12998 to /var/qmail/bin/qmail-queue
2011-06-08 20:22:48.018777500 12763 250 Queued! 1307553768 qp 12998 <>
2011-06-08 20:22:48.227387500 12852 dispatching RCPT TO:<dvdcormack@yahoo.co.uk>
2011-06-08 20:22:48.227389500 12852 250 <dvdcormack@yahoo.co.uk>, recipient ok
2011-06-08 20:22:48.227390500 12909 dispatching RCPT TO:<dvdlanduk@hotmail.co.uk>
2011-06-08 20:22:48.227391500 12909 250 <dvdlanduk@hotmail.co.uk>, recipient ok
2011-06-08 20:22:48.227393500 12864 dispatching RCPT TO:<dvdhanlon@hotmail.com>
2011-06-08 20:22:48.227883500 12864 250 <dvdhanlon@hotmail.com>, recipient ok
2011-06-08 20:22:48.598879500 12851 dispatching RCPT TO:<dvd13@btopenworld.com>
2011-06-08 20:22:48.598881500 12851 250 <dvd13@btopenworld.com>, recipient ok
2011-06-08 20:22:48.614345500 12763 dispatching QUIT
2011-06-08 20:22:48.614348500 12763 221 mydomain.com closing connection. Have a wonderful day.
2011-06-08 20:22:48.614349500 12763 click, disconnecting
2011-06-08 20:22:48.775488500 4296 cleaning up after 12763
2011-06-08 20:22:48.967671500 12852 dispatching RCPT TO:<dvdcrll999@live.co.uk>
2011-06-08 20:22:48.967673500 12909 dispatching RCPT TO:<dvdlenehan@yahoo.com>
2011-06-08 20:22:48.967675500 12909 250 <dvdlenehan@yahoo.com>, recipient ok
2011-06-08 20:22:48.967676500 12864 dispatching RCPT TO:<dvdhgh@hotmail.com>
2011-06-08 20:22:48.967677500 12864 250 <dvdhgh@hotmail.com>, recipient ok
2011-06-08 20:22:48.967894500 12852 250 <dvdcrll999@live.co.uk>, recipient ok
2011-06-08 20:22:49.338521500 12851 dispatching RCPT TO:<dvd2k2009@hotmail.co.uk>
2011-06-08 20:22:49.338523500 12851 250 <dvd2k2009@hotmail.co.uk>, recipient ok
2011-06-08 20:22:49.706684500 12852 dispatching RCPT TO:<dvddvd95@ntlworld.com>
2011-06-08 20:22:49.706686500 12852 250 <dvddvd95@ntlworld.com>, recipient ok
2011-06-08 20:22:49.706687500 12909 dispatching RCPT TO:<dvdlghrn@hotmail.co.uk>
2011-06-08 20:22:49.706688500 12909 250 <dvdlghrn@hotmail.co.uk>, recipient ok
2011-06-08 20:22:49.706690500 12864 dispatching RCPT TO:<dvdholyhead@yahoo.co.uk>
2011-06-08 20:22:49.706691500 12864 250 <dvdholyhead@yahoo.co.uk>, recipient ok
2011-06-08 20:22:50.077213500 12851 dispatching RCPT TO:<dvd2lp@hotmail.com>
2011-06-08 20:22:50.077864500 12851 250 <dvd2lp@hotmail.com>, recipient ok
2011-06-08 20:22:50.386749500 13026 Accepted connection 4/40 from 127.0.0.1 / localhost
2011-06-08 20:22:50.386842500 13026 Connection from localhost [127.0.0.1]
2011-06-08 20:22:50.388633500 13026 tls plugin (init): ciphers: ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
2011-06-08 20:22:50.393521500 13026 220 mail.mydomain.com ESMTP
2011-06-08 20:22:50.444953500 12852 dispatching RCPT TO:<dvdebor@yahoo.co.uk>
2011-06-08 20:22:50.444955500 12852 250 <dvdebor@yahoo.co.uk>, recipient ok
2011-06-08 20:22:50.444956500 12909 dispatching RCPT TO:<dvd-magic@tesco.net>
2011-06-08 20:22:50.444958500 12909 250 <dvd-magic@tesco.net>, recipient ok
2011-06-08 20:22:50.444959500 12864 dispatching RCPT TO:<dvdhopewell@yahoo.co.uk>
2011-06-08 20:22:50.444960500 12864 250 <dvdhopewell@yahoo.co.uk>, recipient ok
2011-06-08 20:22:50.829477500 12851 dispatching DATA
2011-06-08 20:22:50.829479500 12851 354 go ahead
2011-06-08 20:22:50.835043500 13026 dispatching HELO User
2011-06-08 20:22:50.835428500 13026 250 mydomain.com Hi localhost [127.0.0.1]; I am so happy to meet you.
2011-06-08 20:22:51.195926500 12909 dispatching RCPT TO:<dvdmakowski@yahoo.co.uk>
2011-06-08 20:22:51.195929500 12909 250 <dvdmakowski@yahoo.co.uk>, recipient ok
2011-06-08 20:22:51.200961500 12852 dispatching DATA
2011-06-08 20:22:51.200963500 12852 354 go ahead
2011-06-08 20:22:51.206695500 12864 dispatching DATA
2011-06-08 20:22:51.206697500 12864 354 go ahead
2011-06-08 20:22:51.567719500 12851 spooling message to disk
2011-06-08 20:22:51.572809500 13026 dispatching RSET
2011-06-08 20:22:51.572810500 13026 250 OK
2011-06-08 20:22:51.957061500 12909 dispatching RCPT TO:<dvdman@blueyonder.co.uk>
2011-06-08 20:22:51.957063500 12909 250 <dvdman@blueyonder.co.uk>, recipient ok
2011-06-08 20:22:51.957065500 12864 spooling message to disk
2011-06-08 20:22:52.325844500 12852 spooling message to disk
2011-06-08 20:22:52.343522500 13026 dispatching MAIL FROM:<paper-free@online-documents.halifax-online.co.uk>
2011-06-08 20:22:52.343524500 13026 full from_parameter: FROM:<paper-free@online-documents.halifax-online.co.uk>
2011-06-08 20:22:52.343525500 13026 getting mail from <paper-free@online-documents.halifax-online.co.uk>
2011-06-08 20:22:52.343527500 13026 250 <paper-free@online-documents.halifax-online.co.uk>, sender OK - how exciting to get mail from you!
2011-06-08 20:22:52.717789500 12909 dispatching RCPT TO:<dvdman147@yahoo.co.uk>
2011-06-08 20:22:52.718284500 12909 250 <dvdman147@yahoo.co.uk>, recipient ok
2011-06-08 20:22:53.127800500 13026 dispatching RCPT TO:<dvdmckenna79@yahoo.co.uk>
2011-06-08 20:22:53.127803500 13026 250 <dvdmckenna79@yahoo.co.uk>, recipient ok
2011-06-08 20:22:53.530126500 12909 dispatching RCPT TO:<dvdman99@hotmail.co.uk>
2011-06-08 20:22:53.530820500 12909 250 <dvdman99@hotmail.co.uk>, recipient ok
2011-06-08 20:22:53.935701500 13026 dispatching RCPT TO:<dvdmisc@ntlworld.com>
2011-06-08 20:22:53.941231500 13026 250 <dvdmisc@ntlworld.com>, recipient ok
2011-06-08 20:22:54.339913500 12909 dispatching RCPT TO:<dvdmccr@aol.com>
2011-06-08 20:22:54.340612500 12909 250 <dvdmccr@aol.com>, recipient ok
2011-06-08 20:22:54.537358500 13026 dispatching RCPT TO:<dvdmixes@hotmail.com>
2011-06-08 20:22:54.537360500 13026 250 <dvdmixes@hotmail.com>, recipient ok
2011-06-08 20:22:54.909095500 12909 dispatching DATA
2011-06-08 20:22:54.909097500 12909 354 go ahead
2011-06-08 20:22:55.314317500 13026 dispatching RCPT TO:<dvdmoranlizmoran@yahoo.co.uk>
2011-06-08 20:22:55.314319500 13026 250 <dvdmoranlizmoran@yahoo.co.uk>, recipient ok
2011-06-08 20:22:56.034530500 12909 spooling message to disk

[cipandales]

.....and in the mail log files/list outgoing messages i have thousands of lines:

.............
8 Jun 2011 13:22:47 GMT  #53872003  71394    bouncing
   remote   dart2darts@yahoo.co.uk
   remote   dartagatere@hotmail.co.uk
   remote   dartagnan7@lycos.co.uk
  done   remote   dartagnan77165@neuf.fr
   remote   dartann71@hotmail.co.uk
   remote   dartapping@btinternet.com
   remote   dartboy180@hotmail.com
   remote   darted2002@aol.com
   remote   darterace.uk@btinternet.com
   remote   dartess2002@yahoo.co.uk
   remote   dartfordbabe@hotmail.com
   remote   dartfordjfc@hotmail.co.uk
   remote   dartgirl45@hotmail.com
   remote   darth.lord@gwadanews.com
  done   remote   darth.mctaggart@ntlworld.com
8 Jun 2011 08:56:32 GMT  #47892601  71395   
   remote   chez1-@hotmail.co.uk
   remote   chez1026@hotmail.com
   remote   chez110@hotmail.com
   remote   chez118@hotmail.com
   remote   chez128@hotmail.com
   remote   chez1301xxx@hotmail.com
   remote   chez1308@hotmail.co.uk
   remote   chez144@hotmail.com
  done   remote   chez1477@gmail.com
   remote   chez1477@hotmail.com
   remote   chez-15@hotmail.co.uk
   remote   chez1521@hotmail.com
   remote   chez16_xox@hotmail.com
   remote   chez18_7@hotmail.com
   remote   chez1950_50@hotmail.com
8 Jun 2011 14:16:17 GMT  #47894625  71394    bouncing
   remote   davidwebster@msn.com
   remote   davidwebster_29@hotmail.com
   remote   davidwebster44@yahoo.co.uk
   remote   davidweigh@hotmail.co.uk
   remote   davidweightdesignartwork@yahoo.co.uk
   remote   davidwelham@hotmail.com
  done   remote   davidweller6@blueyonder.co.uk
  done   remote   davidwellings@bmf.demon.co.uk
   remote   davidwellings9@hotmail.com
  done   remote   davidwells@mowers1a.fsnet.co.uk
  done   remote   davidwells32@googlemail.com
   remote   davidwemmerson@aol.com
   remote   davidwendy@hotmail.co.uk
  done   remote   davidwest@beeb.net
   remote   davidwest@uhns.nhs.uk
7 Jun 2011 13:45:27 GMT  #47880733  71394   
  done   remote   cat_meyrick_123@hotmail.com
  done   remote   cat_moss@hotmail.co.uk
   remote   cat_n30@yahoo.co.uk
   remote   cat_nam@yahoo.co.uk
  done   remote   cat_parker@hotmail.com
  done   remote   cat_parker1@hotmail.com
  done   remote   cat_pers@hotmail.com
  done   remote   cat_pringle_pinklady@hotmail.com
  done   remote   cat_queen_11@hotmail.com
  done   remote   cat_renshaw@hotmail.com
  done   remote   cat_rescue@msn.com
  done   remote   cat_rex@hotmail.com
  done   remote   cat_robb@hotmail.com
   remote   cat_rochford@yahoo.co.uk
   remote   cat_s100@yahoo.co.uk
7 Jun 2011 14:40:58 GMT  #47882688  71395    bouncing
  done   remote   drmih@hvmail.co.uk
  done   remote   drmike@mikeoshea.co.uk
  done   remote   drmikemaloney@hotmail.com
   remote   drmiller599@yahoo.co.uk
  done   remote   drminem@aol.com
  done   remote   drminer@embarqmail.com
  done   remote   drmittenspider@hotmail.co.uk
  done   remote   drmkcenko@aol.com
  done   remote   drmkensington@gmail.com
  done   remote   drmlad@hotmail.co.uk
   remote   drmohdnazir@yahoo.co.uk
  done   remote   drmohdrafi@doctors.net.uk
  done   remote   drmoo45@hotmail.com
  done   remote   drmoonshine@tiscali.co.uk
  done   remote   drmount@bigpond.com.dele.te
8 Jun 2011 02:14:48 GMT  #45483788  71395    bouncing
   remote   barry@racing2profit.com
  done   remote   barry@rainbowoffice.co.uk
  done   remote   barry@rainbowweb.freeserve.co.uk
  done   remote   barry@reading3559.fsnet.co.uk
  done   remote   barry@reboot.wanadoo.co.uk
  done   remote   barry@red-baron.fsnet.co.uk
  done   remote   barry@redrose1.fsnet.co.uk
  done   remote   barry@regent-estates.co.uk
  done   remote   barry@rogers1175.freeserve.co.uk
  done   remote   barry@safe2connect.co.uk
  done   remote   barry@scicam.co.uk
  done   remote   barry@scottb.demon.co.uk
   remote   barry@securityguardcompany.co.uk
  done   remote   barry@shaftfield.co.uk
  done   remote   barry@shenton3.fsnet.co.uk
7 Jun 2011 23:31:29 GMT  #45482684  71395    bouncing
  done   remote   andy.dudley@blueyonder.co.uk
  done   remote   andy.dunn37@ntlworld.com
  done   remote   andy.durnion@blueyonder.co.uk
  done   remote   andy.duval@siemens.com
   remote   andy.dwyer@nmigroup.com
  done   remote   andy.eakins@medinn.co.uk
   remote   andy.easteal@parkerbaines.co.uk
  done   remote   andy.easteal@radiuslondon.com
  done   remote   andy.easton@intechnology.co.uk
  done   remote   andy.ellman-brown@blueyonder.co.uk
  done   remote   andy.else@ntlworld.com
  done   remote   andy.elson@hotmail.com
  done   remote   andy.engeluk@tiscali.co.uk
  done   remote   andy.english-revill@ntlworld.com
  done   remote   andy.eyres@blueyonder.co.uk
8 Jun 2011 11:50:13 GMT  #47894349  71395    bouncing
  done   remote   crmarlow@tiscali.co.uk
  done   remote   crmarno@hotmail.com
   remote   crmarshall@v21.me.uk
  done   remote   crmeh42@tiscali.co.uk
  done   remote   crmerrick@hotmail.co.uk
  done   remote   crmiller@emohawk.eclipse.co.uk
   remote   crmillward@ukonline.co.uk
  done   remote   crmit@mypostoffice.co.uk
   remote   crmlkissez@yahoo.co.uk
  done   remote   crmoriarty@hotmail.com
  done   remote   crmpicco@hotmail.com
  done   remote   crmspencer@fsmail.net
   remote   crmwlg1@aol.com
  done   remote   crni2910@gmail.com
   remote   crnjoan7@aol.com
7 Jun 2011 13:30:57 GMT  #47880066  71394    bouncing
  done   remote   bodyshrine@hotmail.com
  done   remote   bodytape@hotmail.com
  done   remote   bodytek@tiscali.co.uk
  done   remote   bodythief22@hotmail.com
  done   remote   bodyworkonsite@hotmail.com
  done   remote   bodyworks_4u@hotmail.com
  done   remote   bodyworkshop123@hotmail.com
  done   remote   bodz2009@hotmail.co.uk
  done   remote   bodzio@always.uk
   remote   bodzio_c@yahoo.co.uk
  done   remote   boe@millfactory.dk
   remote   boedicayy@yahoo.co.uk
  done   remote   boehead13@hotmail.com
  done   remote   boehiggs22@hotmail.co.uk
  done   remote   boeing777@fleetbuzz.com
...........

[cactus]

First things you need to do is stop qmail:

Code: [Select]

sv d /service/qmail
After that you will have to analyze the messages in the queue, particularly the header information to find out from which host they are coming, the Received: header should show you the machine the mail originated from.

You can find your queue in the following location:

/var/qmail/queue/local (for mail originating from your network)
/var/qmail/queue/remote (for mail not from your network)

Most likely it is a local system that is used to send spam. Isolate the affected machine and then clean out the queue, be sure to not remove all messages but carefully select them. qmHandle might help you with that. Information on qmHandle can be found in the wiki:

http://wiki.contribs.org/Qmhandle_mail_queue_manager

Examples on how to use it are also in this thread: http://forums.contribs.org/index.php/topic,40959.0.html

After cleaning the queue and isolating the systems you will have to restart qmail again so the mail starts flowing again:

Code: [Select]

sv u /service/qmail
After that it is time to clean up the affected system still keeping it isolated and disconnected from your network, only after you are sure it is clean you can add it to the network again.

[CharlieBrady]

After that you will have to analyze the messages in the queue, particularly the header information to find out from which host they are coming, the Received: header should show you the machine the mail originated from.

We already know that from the log message we have been shown - it is 127.0.0.1. So some program running on the server itself is injecting those mail messages.

My first guess is that this is a compromised password being used to access webmail. /var/log/httpd/access.log should provide evidence of that. Identify the account and lock it/change its password

[cipandales]

I tried to read /var/log/httpd/access_log but it is empty.

I installed qmHandle and stopped qmail but when i use it with qmHandle -D and after "Calling system script to terminate qmail..." queue mails are deleted.

Then restarted qmail and it happens again. There are no computers running now in local network.

There are many mails in /var/qmail/queue/remote and none in var/qmail/queue/local.

How can i see who or what program send this spam ?

Please, help !

Thank you

[cactus]

I tried to read /var/log/httpd/access_log but it is empty.

Also look through the older ones (logs are being rotated), although I can't hardly imagine it is empty.

[cipandales]

i looked through older ones and there are many external ip's but no specific account.

How can i see wich account are using webmail and sending spam ?

Now i stopped qmail but the queue is growing....

Also i looked all accounts and the queue is growing either...

Please help....

[Stefano]

are there any web application running on your server and exposed to wan? php applications? are they up-to-date?

please tell us more on your server, thank you

[cipandales]

i have no web application installed.

last night, spam traffic stopped. now i'm looking in the log files and i see just the normal mail traffic.

i don't know why.... nothing in local network had changed.

is there any log files where i can see who or what generated that spam, except those above ? can SME Server be infected itself ?

thank you

[mmccarn]

You may want to check /var/log/sshd/* to see if anyone has been logging in to your system remotely.

It seems inconceivable that /var/log/httpd/current would be empty -- browse to an ibay, then check it again.  If it's still empty, then possibly your web server has been reconfigured to use a different log file.

Does your SME server have the SMTP transparent proxy enabled (config show smptd)?  If so, this would be intercepting all outbound SMTP traffic for all LAN hosts - perhaps (and I really have no idea about this) the resulting traffic would appear to come from "127.0.0.1" when viewed by qpsmtpd (this would depend on the firewall rules used to enable the transparent proxy).  You could test this by firing up a LAN workstation and sending an email from an email client that is trying to use an off-site SMTP server while watching /var/log/qpsmtpd/current.



 

[Stefano]

i have no web application installed.

last night, spam traffic stopped. now i'm looking in the log files and i see just the normal mail traffic.

i don't know why.... nothing in local network had changed.

is there any log files where i can see who or what generated that spam, except those above ? can SME Server be infected itself ?

thank you

IMHO your server has been compromised.. you should make a backup, optionally a disk image for investigation, then format, re-install, restore..

it's just a guess, I (we) don't know how/who/when it happened

my 2c

[CharlieBrady]

Does your SME server have the SMTP transparent proxy enabled (config show smptd)?  If so, this would be intercepting all outbound SMTP traffic for all LAN hosts - perhaps (and I really have no idea about this) the resulting traffic would appear to come from "127.0.0.1" when viewed by qpsmtpd...

No, it would not. The true source address is logged. The transparent proxy only changes the destination of the connections (terminates them locally rather than passed them out to the  Internet).

[CharlieBrady]

IMHO your server has been compromised.. you should make a backup, optionally a disk image for investigation, then format, re-install, restore..

Then use different stronger passwords for all accounts, and do not enable SSH access from the Internet.

[cipandales]

What do you mean my server has been compromised ?
I have a lot of files/imap mail hosted on this server and it will take a long time to backup and restore on other server.
It is so unsecured ? Even i'll change my root and all accounts passwords, stop ssh access and remote admin access ?

Please, be more specific.

Thank you for your patience.

[mmccarn]

"compromised" means that there is a chance that someone has had an unauthorized level of access to your system.

Possible side-effects could range from unwanted email relay to replacement of binary files on your system.

A common early linux/unix attack (mid 90's), for example, involved replacing the 'login' program with another that would collect user credentials and periodically send them to the attacker. 

Or an attacker could replace your copy of 'qpsmtpd' with another binary that does everything usually done by qpsmtpd in addition to relaying spam.

Combine either of the above with changes that prevent logging of the relevant information in the system log files, and you may never know exactly what has been done to a "compromised" system, or what it is doing.

You can choose to be optimistic, and assume the intrusion was minor and can be corrected by changing user passwords etc, but this may result in continued unauthorized access to (and use of) your system. 

As with all aspects of network and data security, the decision is a trade-off between convenience and consequences -- if you go for the easier, more convenient cleanup, can you support the future consequences if your system really has been compromised and the cleanup isn't completely successful?  Only you can answer this question...