Topic: 7.6 server spewing out spam- where to start??

[ber]

SME server 7.6 updates installed.

getting failure of email notices with myriads of addresses that the q-mail is trying to send to but being rejected. Ive stopped the  qmail service which has stopped the error messages.
here is a sample of the message with myriads of addresses that is being rejected.

Hi. This is the qmail-send program at ber.local.
I tried to deliver a bounce message to this address, but the bounce bounced!

<fahad.alsaeed198@gmail.com>:
173.194.79.26 failed after I sent the message.
Remote host said: 550-5.7.1 [119.224.106.1      12] Our system has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail,
550-5.7.1 this message has been blocked. Please visit
550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131 for
550 5.7.1 more information. so1si4664466pab.90 - gsmtp

--- Below this line is the original bounce.

Return-Path: <>
Received: (qmail 7451 invoked for bounce); 16 Jul 2013 21:38:00 -0000
Date: 16 Jul 2013 21:38:00 -0000
From: MAILER-DAEMON@ber.local
To: fahad.alsaeed198@gmail.com
Subject: failure notice

Hi. This is the qmail-send program at ber.local.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

I've shutdown all the internal PC's and scanned to spam- all clean. Shut down all PC's- restarted qmail and the errors began to pour in again.
I suspect the server has been hacked??

here is the qmail log

2013-07-17 23:56:38.415934500 starting delivery 10405: msg 5424493 to remote menno2000@ayna.com
2013-07-17 23:56:38.415935500 status: local 0/10 remote 20/20
2013-07-17 23:56:38.417339500 delivery 10404: deferral: Sorry,_I_wasn't_able_to_establish_an_SMTP_connection._(#4.4.1)/
2013-07-17 23:56:38.417341500 status: local 0/10 remote 19/20
2013-07-17 23:56:38.417342500 starting delivery 10406: msg 5424493 to remote meno2000@ayna.com
2013-07-17 23:56:38.417344500 status: local 0/10 remote 20/20
2013-07-17 23:56:38.452580500 delivery 10405: deferral: Sorry,_I_wasn't_able_to_establish_an_SMTP_connection._(#4.4.1)/
2013-07-17 23:56:38.452582500 status: local 0/10 remote 19/20
2013-07-17 23:56:38.452583500 starting delivery 10407: msg 5424493 to remote meno_ya_layaly@hotmail.com
2013-07-17 23:56:38.452584500 status: local 0/10 remote 20/20
2013-07-17 23:56:38.454607500 delivery 10406: deferral: Sorry,_I_wasn't_able_to_establish_an_SMTP_connection._(#4.4.1)/
2013-07-17 23:56:38.454609500 status: local 0/10 remote 19/20
2013-07-17 23:56:38.454610500 starting delivery 10408: msg 5424493 to remote menome@yahoo.com
2013-07-17 23:56:38.454611500 status: local 0/10 remote 20/20
2013-07-17 23:56:38.976165500 delivery 10401: success: 65.54.188.94_accepted_message./Remote_host_said:_250__<20130712024632.19992vn7dgsb3ha8@119.224.106.1>_Queued_mail_for_delivery/
2013-07-17 23:56:38.976167500 status: local 0/10 remote 19/20
2013-07-17 23:56:38.976168500 starting delivery 10409: msg 5424493 to remote mensaf@37.com
2013-07-17 23:56:38.976169500 status: local 0/10 remote 20/20
2013-07-17 23:56:39.261931500 delivery 10403: failure: 65.55.92.136_does_not_like_recipient./Remote_host_said:_550_Requested_action_not_taken:_mailbox_unavailable/Giving_up_on_65.55.92.136./
2013-07-17 23:56:39.261933500 status: local 0/10 remote 19/20
2013-07-17 23:56:39.261934500 starting delivery 10410: msg 5424493 to remote Mensur.Tahirovic@tr.ey.com
2013-07-17 23:56:39.261935500 status: local 0/10 remote 20/20
2013-07-17 23:56:39.471366500 delivery 10402: success: 65.54.188.94_accepted_message./Remote_host_said:_250__<20130712024632.19992vn7dgsb3ha8@119.224.106.1>_Queued_mail_for_delivery/
2013-07-17 23:56:39.471368500 status: local 0/10 remote 19/20
2013-07-17 23:56:39.471369500 starting delivery 10411: msg 5424493 to remote ment@ayna.com
2013-07-17 23:56:39.471371500 status: local 0/10 remote 20/20

/var/log/qpsmtpd/current

2013-07-17 20:09:44.309564500 23766 Plugin tls, hook mail returned DECLINED,
2013-07-17 20:09:44.309604500 23766 running plugin (mail): require_resolvable_fromhost
2013-07-17 20:09:44.309678500 23766 trying to get config for invalid_resolvable_fromhost
2013-07-17 20:09:44.324664500 23766 trying to get config for require_resolvable_fromhost
2013-07-17 20:09:44.326035500 23766 Plugin require_resolvable_fromhost, hook mail returned DECLINED,
2013-07-17 20:09:44.326080500 23766 running plugin (mail): check_badmailfrom
2013-07-17 20:09:44.326152500 23766 trying to get config for badmailfrom
2013-07-17 20:09:44.338554500 23766 Plugin check_badmailfrom, hook mail returned DECLINED,
2013-07-17 20:09:44.338668500 23766 getting mail from <clubwarebackup@asfc.co.nz>
2013-07-17 20:09:44.338739500 23766 250 <clubwarebackup@asfc.co.nz>, sender OK - how exciting to get mail from you!
2013-07-17 20:09:44.375412500 23766 dispatching RCPT TO:<john@ber.net.nz>
2013-07-17 20:09:44.375414500 23766 to email address : [<john@ber.net.nz>]
2013-07-17 20:09:44.375415500 23766 running plugin (rcpt): tls
2013-07-17 20:09:44.375417500 23766 Plugin tls, hook rcpt returned DECLINED,
2013-07-17 20:09:44.375418500 23766 running plugin (rcpt): check_badmailfrom
2013-07-17 20:09:44.375419500 23766 Plugin check_badmailfrom, hook rcpt returned DECLINED,
2013-07-17 20:09:44.375420500 23766 running plugin (rcpt): check_badrcptto_patterns
2013-07-17 20:09:44.375431500 23766 trying to get config for badrcptto_patterns
2013-07-17 20:09:44.376745500 23766 Plugin check_badrcptto_patterns, hook rcpt returned DECLINED,
2013-07-17 20:09:44.376787500 23766 running plugin (rcpt): check_badrcptto
2013-07-17 20:09:44.376851500 23766 trying to get config for badrcptto
2013-07-17 20:09:44.391750500 23766 Plugin check_badrcptto, hook rcpt returned DECLINED,
2013-07-17 20:09:44.391794500 23766 running plugin (rcpt): check_goodrcptto
2013-07-17 20:09:44.391879500 23766 check_goodrcptto plugin (rcpt): stripping '-' extensions
2013-07-17 20:09:44.391922500 23766 trying to get config for goodrcptto
2013-07-17 20:09:44.558480500 23766 check_goodrcptto plugin (rcpt): address includes extn '-', checking users: john
2013-07-17 20:09:44.572745500 23766 Plugin check_goodrcptto, hook rcpt returned DECLINED,
2013-07-17 20:09:44.572806500 23766 running plugin (rcpt): rcpt_ok
2013-07-17 20:09:44.572890500 23766 trying to get config for rcpthosts
2013-07-17 20:09:44.591180500 23766 Plugin rcpt_ok, hook rcpt returned OK,
2013-07-17 20:09:44.591321500 23766 250 <john@ber.net.nz>, recipient ok
2013-07-17 20:09:44.627796500 23766 dispatching DATA
2013-07-17 20:09:44.627908500 23766 running plugin (data): tls
2013-07-17 20:09:44.627998500 23766 Plugin tls, hook data returned DECLINED,
2013-07-17 20:09:44.628037500 23766 running plugin (data): check_earlytalker
2013-07-17 20:09:44.628155500 23766 Plugin check_earlytalker, hook data returned DECLINED,
2013-07-17 20:09:44.628305500 23766 354 go ahead
2013-07-17 20:09:44.628381500 23766 trying to get config for databytes
2013-07-17 20:09:44.646044500 23766 max_size: 50000000 / size: 0
2013-07-17 20:09:44.646205500 23766 trying to get config for timeout
2013-07-17 20:09:44.675130500 23766 spooling message to disk
2013-07-17 20:09:44.757011500 23766 max_size: 50000000 / size: 1270
2013-07-17 20:09:44.757013500 23766 running plugin (data_post): check_basicheaders
2013-07-17 20:09:44.757014500 23766 Plugin check_basicheaders, hook data_post returned DECLINED,
2013-07-17 20:09:44.757016500 23766 running plugin (data_post): virus::pattern_filter
2013-07-17 20:09:44.757017500 23766 trying to get config for pattern_filter
2013-07-17 20:09:44.757018500 23766 trying to get config for signatures_patterns
2013-07-17 20:09:44.757019500 23766 Plugin virus::pattern_filter, hook data_post returned DECLINED,
2013-07-17 20:09:44.757031500 23766 running plugin (data_post): tnef2mime
2013-07-17 20:09:44.780146500 23766 Plugin tnef2mime, hook data_post returned DECLINED,
2013-07-17 20:09:44.780196500 23766 running plugin (data_post): spamassassin
2013-07-17 20:09:44.780280500 23766 spamassassin plugin (data_post): check_spam
2013-07-17 20:09:44.780749500 23766 spamassassin plugin (data_post): check_spam: connected to spamd
2013-07-17 20:09:44.781588500 23766 spamassassin plugin (data_post): check_spam: finished sending to spamd
2013-07-17 20:09:48.467078500 23766 spamassassin plugin (data_post): check_spam: spamd: SPAMD/1.1 0 EX_OK
2013-07-17 20:09:48.467080500 23766 spamassassin plugin (data_post): check_spam: spamd: Content-length: 42
2013-07-17 20:09:48.467082500 23766 spamassassin plugin (data_post): check_spam: spamd: Spam: False ; 3.1 / 5.0
2013-07-17 20:09:48.467083500 23766 spamassassin plugin (data_post): check_spam: spamd:
2013-07-17 20:09:48.467084500 23766 spamassassin plugin (data_post): check_spam: finished reading from spamd
2013-07-17 20:09:48.467085500 23766 spamassassin plugin (data_post): check_spam: No, hits=3.1, required=5.0, tests=MISSING_MID,RCVD_IN_BRBL_LASTEXT,RDNS_NONE
2013-07-17 20:09:48.467110500 23766 Plugin spamassassin, hook data_post returned DECLINED,
2013-07-17 20:09:48.467111500 23766 running plugin (data_post): spamassassin
2013-07-17 20:09:48.474081500 23766 Plugin spamassassin, hook data_post returned DECLINED,
2013-07-17 20:09:48.474083500 23766 running plugin (data_post): virus::clamav
2013-07-17 20:09:48.474084500 23766 virus::clamav plugin (data_post): Changing permissions on file to permit scanner access
2013-07-17 20:09:48.474086500 23766 virus::clamav plugin (data_post): Running: /usr/bin/clamdscan --stdout  --config-file=/etc/clamd.conf --no-summary /var/spool/qpsmtpd/1374048584:23766:0 2>&1
2013-07-17 20:09:48.541338500 23766 virus::clamav plugin (data_post): clamscan results: /var/spool/qpsmtpd/1374048584:23766:0: OK
2013-07-17 20:09:48.541340500 23766 Plugin virus::clamav, hook data_post returned DECLINED,
2013-07-17 20:09:48.541341500 23766 running plugin (queue): logging::logterse
2013-07-17 20:09:48.541343500 23766 logging::logterse plugin (queue): ` 219.88.131.228   Unknown   asfc.local   <clubwarebackup@asfc.co.nz>   <john@ber.net.nz>   queued      <>   No, hits=3.1 required=5.0_
2013-07-17 20:09:48.541344500 23766 Plugin logging::logterse, hook queue returned DECLINED,
2013-07-17 20:09:48.541361500 23766 running plugin (queue): queue::qmail_2dqueue
2013-07-17 20:09:48.541362500 23776 queue::qmail_2dqueue plugin (queue): (for 23766 ) Queuing qp 23776 to

/var/qmail/bin/qmail-queue

2013-07-17 20:09:48.899772500 23766 Plugin queue::qmail_2dqueue, hook queue returned OK, Queued! 1374048588 qp 23776 <>
2013-07-17 20:09:48.899774500 23766 250 Queued! 1374048588 qp 23776 <>
2013-07-17 20:09:48.949051500 23766 dispatching QUIT

I am at a loss how to fix this...What is the best way to resolve it?

[Stefano]

any web application running on the server and available from WAN?

[Frank VB]

You can find a good pointer here: http://forums.contribs.org/index.php/topic,49785.msg249289.html#msg249289

Be aware that SME 7.6 is no longer supported, it is obsolete. After you've resolved your spam issue, you should consider upgrading to SME 8.0 as soon as possible.

Good luck!

[CharlieBrady]

I am at a loss how to fix this...What is the best way to resolve it?

The only way to resolve it is to identify the cause of the problem. We've given advice before here how to troubleshoot an issue like this - I'm surprised you didn't find it.

You've done the right thing to stop qmail. You now need to identify the source of the spam, fix that problem, clean the qmail queue and then restart qmail.

It's good that you've looked in the qpsmtpd log file, but the example you've shown appears to be normal incoming email, not a spam message being relayed to an external IP address.

Your best approach to identify the source of the problem is to find spam messages in your qmail queue, and then look at the Received headers in the message and identify where the messages came from. Then you need to identify how the messages are being injected. Two possibilities are an exploitable web application, and the compromise of one or more account passwords.

[ber]

Charlie, Ive made efforts to track and see where the problem lies. The server hosts 5 domains (email and websites). Imap service is running accessible from the inter net.

I have looked into this post which seems helpful.

http://forums.contribs.org/index.php/topic,49785.msg249289.html#msg249289

This post describes my problem- My server is being used to pump out spam. I am not sure whether its a web application or a account hack.

I have gone as far as I can and am not sure what to do next. I have progrssed to the stage where i have been able to get some info on the email that has been sent as spam. I have a UID from the email. running grep and it displays the following output.


<rainbowarts_uae@yahoo.com>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

<rainy9699@yahoo.com.cn>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

<rainujac@yahoo.com>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

<rainy_desai@yahoo.co.in>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

<raisa.merchant@unilver.com>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

<raissa_reyes@yahoo.com>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

<raisarivne@yahoo.com>:
Connected to 127.0.0.1 but greeting failed.
Remote host said: 451 Upstream SMTP server not available
I'm not going to try again; this message has been in the queue too long.

--- Below this line is a copy of the message.

Return-Path: <fahad.alsaeed198@gmail.com>
Received: (qmail 17332 invoked by uid 453); 6 Jul 2013 13:13:39 -0000
X-Virus-Checked: Checked by ClamAV on ber.local
Received: from localhost (HELO localhost) (127.0.0.1)
    by ber.local (qpsmtpd/0.83) with ESMTP; Sun, 07 Jul 2013 01:13:39 +1200
Received: from 120.141.193.8 ([120.141.193.8]) by 119.224.106.1 (Horde
 Framework) with HTTP; Sun,  7 Jul 2013 01:13:30 +1200
Message-ID: <20130707011330.17513dn4l9nvq22o@119.224.106.1>
Date: Sun,  7 Jul 2013 01:13:30 +1200
From: Fahad Al Saeed <fahad.alsaeed198@gmail.com>
Reply-to: fahad.alaeed23@yahoo.com
To: undisclosed-recipients:;
Subject: Re: Al Saeed Fahad From Libya ( Not A Junk Mail)
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) H3 (4.3.6)



Dear Friend,

It's my pleasure to contact you for a business investment which I want to
establish in your country, though I have not met you before but wishes to
contact you through email which I think is the fastest and easy way of
communication these days.

I am personal aid to Late Col Muammar Gaddafi, who Died with the NATO
coalition Since I observed that the war has taken a drastic shape and it
will be very difficult to resolve the crises and restore my Boss Col Muammar
Gaddafi to Power.

I have been able to divert a Crude Oil payment to unknown destination for
safe keeping. This will now serve as my gratuity and establishment in life,
since he has been very cruel and wicked to all his men.

I have been subjected to so many difficult tasks by his tyrant attitude and
neglect to mankind. That is why I am soliciting for your kind understand to
assist me receive in your custody the sum of Eleven Million and Seven Hundred
and Sixty Thousand United States Dollars ( US$11.760 Million) for my
establishment and subsequent disbursement in your country as I do not
intend to return back even after the war.

Right now I have been able to move out of my country and wish to have a quick
response from you, so that I could give out to you all the vital documents and
information concerning the money.

I will give the security Company your name and contact information as the
bonafide beneficiary of the funds for immediate release.

This piece of information requires absolute confidentiality as you can
understand that I am just hanging out here without much on me.
I need your truthful cooperation, so that I can take care of my family and
many other dependants scattered all over Italy , Tunisia and other countries.

Your swift response will be of great importance to me, because we do not
really need to waste time over this matter.

I need your contact number and mailing address for better correspondence. I
look forward towards a healthier business opportunities with you in any area
that will be profitable.

This will be 100% risk free transaction.
Kindly Contact me Direct to my email address:fahad.alaeed23@yahoo.com

Mr.Fahad Al Saeed



[root@server ~]# grep 453 /etc/passwd
qpsmtpd:x:453:453:qpsmtpd system user:/var/service/qpsmtpd:/bin/false
[root@server ~]#


Can anyone advise what the information is and what are my next steps to pinpoint the cause of the spam.

[Brave Dave]

Do you have Webmail Accessible from the Internet ?

I think - based on the header

• Your External IP is : 119.224.106.1
• The Perpetrators IP is : 120.141.193.8
• Which is in Kuala Lumpur (http://www.geoiptool.com/en/?IP=120.141.193.8)
• They did this on : Sun, 07 Jul 2013 01:13:30 +1200

It looks like someone has gained access to the webmail system and pumped out the messages using the bcc field

The answer to that is:

• Change All your Passwords ensuring they adhere to good policy
• Clear the Queue

Return-Path: <fahad.alsaeed198@gmail.com>
Received: (qmail 17332 invoked by uid 453); 6 Jul 2013 13:13:39 -0000
X-Virus-Checked: Checked by ClamAV on ber.local
Received: from localhost (HELO localhost) (127.0.0.1)
    by ber.local (qpsmtpd/0.83) with ESMTP; Sun, 07 Jul 2013 01:13:39 +1200
Received: from 120.141.193.8 ([120.141.193.8]) by 119.224.106.1 (Horde
 Framework) with HTTP; Sun,  7 Jul 2013 01:13:30 +1200
Message-ID: <20130707011330.17513dn4l9nvq22o@119.224.106.1>
Date: Sun,  7 Jul 2013 01:13:30 +1200
From: Fahad Al Saeed <fahad.alsaeed198@gmail.com>
Reply-to: fahad.alaeed23@yahoo.com
To: undisclosed-recipients:;
Subject: Re: Al Saeed Fahad From Libya ( Not A Junk Mail)
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) H3 (4.3.6)

[ber]

Hi Dave firstly thank you for the assessment.

Yes webmail (horde) is accessible and needed via the internet. I am in the processing of culling users that are no longer valid and hope that this along with updating passwords will help any further intrusions.

I was able to stop qmail and clear the queue. Once I did that it didnt fill up again- so maybe this would be a webmail only issue rather than a workstation?? Spam has stopped.

Can I ask why you suspect that the intrusion is via the webmail?
Also is there anything in the logs that would allow me to see which user account has been compromised? I have about two hundred users.

Kind Regards  John  (ber)

[Brave Dave]

Hi
The highlighted header says horde framework ... (aka webmail)

In horde you can specify another email - you have to confirm it, it looks like this reply email has been set to the gmail address - (s)he would have had to have access to that address to confirm it

you could search in mysql => horde for that alias - something like

select * from horde_prefs where pref_value like '%fahad%';

might show it up, but in the horde database you would be able to sleuth around and find it

[janet]

ber

Remote host said: 550-5.7.1 [119.224.106.1      12] Our system has detected that this message is
550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail,
550-5.7.1 this message has been blocked. Please visit
550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131 for
550 5.7.1 more information. so1si4664466pab.90 - gsmtp

--- Below this line is the original bounce.

Return-Path: <>
Received: (qmail 7451 invoked for bounce); 16 Jul 2013 21:38:00 -0000
Date: 16 Jul 2013 21:38:00 -0000
From: MAILER-DAEMON@ber.local

Also you should name your server with a resolvable domain name (ie not ber.local), so that external mail servers (spam filters & reverse lookup functions etc) can see that messages are coming from a "real" domain. If the source server is not resolvable (like ber.local), then gmail & others will refuse to accept the messages or smtp connection. Use whatever your main real domain is, refer these notes re PTR & SPF in the Manual Appendix section
http://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Appendix#PTR_Records

You should really update to sme8 asap as you are running a server that already has or will soon have security risks. eg anti virus updates are no longer happening & will not be updated
2013-07-19 11:15:04.237583500 WARNING: Your ClamAV installation is OUTDATED!
2013-07-19 11:15:04.237584500 WARNING: Local version: 0.97.7 Recommended version: 0.97.8

[ber]

Thanks all for the reply and assistance- happy to say that after culling a lot of the user accounts and clearing the qmail queue ive resolved the spam issue.
I'm now in a process to get google mail to allow my server to send them stuff or the ISP to renew my IP. 

Ive updated to SME8 and have taken up the advise from Janet- thanks again. 

Kind Regards

[janet]

ber

I'm now in a process to get google mail to allow my server to send them stuff or the ISP to renew my IP.

That can usually be resolved without any great effort, just by changing your server to use your ISP's smtp server for outgoing mail.
Look in the Email panel in server manager.
Mail will then go via your ISP's smtp server, & if they are any good, your ISP should have a good reputation & not be considered a spammer at all by gmail & other mail servers.