Topic: Too Many Connections, Sever Slows, Stops

[newhopenet]

I think I am being attacked with a Denial Of Service.  I'm very new to SME Server, and it is entirely possible that I'm wrong.  I'm willing to work and research to find answers, and I've read the forums, but I'm now stuck not knowing how to proceed.

Symptoms:

1) Server slows to a complete crawl, we stop getting all incoming mail.  Mail is returned to sender with a 'delivery delay' message.

2) Running netstat -an | grep :25.*EST results 70 - 80 connections.  Running it without the *.EST results in hundreds of connections in various states, from many different IP's.

3) qpsmtpd/current shows only 'Too many connections: 40 >=40. Waiting one second" logged over and over and over.

I've read the forums extensively, and this is as far as I can get based on the advice posted in the forums.  I now know the above things, but I have no idea what my next step is to resolve this.  What other information should I provide that would be relevant?

My SME server sits in a DMZ behind my firewall.  It processes incoming mail and then hands it off to our exchange server.  That is the only job of this SME server, no other functions are used.  It should not send mail.  Outgoing mail is sent directly from our exchange server.  The SME Server is a 550MHz machine with 1GB RAM.  Our network only has 10 users.

Is this is a security issue? and What should I do now?

I appreciate any guidance you can provide on what I should look at next.

Thanks!

[byte]

Is this is a security issue?

If you think this is a security issue never post to a public forum as per top of every new thread (before posting)

"Don't report security issues here - Contact security at contribs dot org"

Thanks.

[newhopenet]

Well, first I pretty much doubt myself on determining whether this is an actual security issue or not.  Secondly, the security email address rejects all mail from my gmail account, as they are apparently listed in an SBL.  So, I guess I'll start trying some other free email providers until I can get through.

[dickmorrell]

This is not a denial of service attack in the traditional sense.

Have you applied the greylisting mod on your 7.2 box ?

I have seen exactly the same issue three times this week and it seems to be BIND getting in a fix when faced with large amounts of inbound spam. As soon as I get time I'll file something in Bugzilla properly. Sorry been travelling a lot this week.

[dickmorrell]

Ok now the mirrors seem to have sync'd apply the latest updates to your box with yum upgrade from the command prompt as root. Problem should disappear. Seems to be linked to Perl issues which are solved with the updates to various Perl libraries getting their knickers in the proverbial twist.

If you still get this issue paste the log file or email it to me offline and I'll have a look through it for you.

[byte]

I have seen exactly the same issue three times this week and it seems to be BIND getting in a fix when faced with large amounts of inbound spam.

Where is the evidence of BIND being the issue? the OP h/w is not the most powerful when dealing with large amount of load that they are seeing.

[micropitt]

Did you actually see some of the incoming e-mails? Are they addressed to existing e-mail accounts? Are the e-mails coming from the same source? My first impression is, your domain is receiving large amount of Spam for some reason that you should investigate. If you receive a large amount of e-mails for non existing e-mail accounts it probably is a dictionary spam attack on your domain. One short term solution would be to shut down smtp for a day or two. If it doesn't stop, contact the hosting company for your domain and/or your ISP and they can set some blocks on there Routers.
 

[dickmorrell]

The evidence of BIND being tasked to the max (and it wasn't a dictionary attack) it seemed to be multiple massive amounts of inbound spam to the same real email address (an Ubuntu list recipient) and looking at the logfiles on the firewall (that I wrote..) was that the SME server was attempting huge amounts of dns lookups to RBLs and the port 53 logs in the firewall logs matched to the activity report on the SME server. Just box getting totally maxed out and almost all CPU taken. Box is a dual 1.8ghz CPU with 4GB of RAM and mirrored 32gb SCSI.

[mmccarn]

Too Many Connections, Sever Slows, Stops

I've been having exactly the same symptoms since updating to 7.2 on one of my servers.  I had this issue briefly after updating to 7.1, too.

Did you actually see some of the incoming e-mails?

On my systems there never is any email resulting from these connections.

In fact, I work around this issue by scanning /var/log/qpsmtpd/* for all connections that were denied by dnsbl or by check_earlytalker, and adding them to the firewall with a 'denylog' rule.  This always clears up the problem.  (I've written some really bad scripts that do the scanning and blocking for me...)

When this started again (about 3 weeks ago) I was blocking about 2000 - 4000 hosts.  Now I find I am blocking port 25 from 21000+ hosts that were denied connection to my server during the life of my log files.  This could either indicate an increase in "attack" behavior, or it could simply reflect the change I made to my qpsmtpd LogLevel after upgrading to 7.2 (with logterse).

yum upgrade from the command prompt as root. Problem should disappear

This would be fabulous news.  Do you have any specifics on why this would cause or fix the described behavior (except perhaps that qpsmtpd is written in perl)?

Don't report security issues here - Contact security at contribs dot org

The last time this happened to me (Nov '06) it seemed to be due to hardware that didn't meet SME's recommended requirements for spam and virus filtering -- which is largely why I haven't posted anything about it this time (I felt like an idiot last time...)

[dickmorrell]

Do the update - issue went away - was seeing same issue three or four times a day

Did update - problem went away

Also: Spam has dropped about 50% (of what was getting through) with new SpamAssassin update too

Thanks to all the hardworking package maintainers you're doing good stuff - much appreciated.

[newhopenet]

Thanks so much for all of your replies, I'm working as fast as I can to learn as much as I can, this is all very new to me.  I appreciate your time here.

Have you applied the greylisting mod on your 7.2 box ?

I have not.  I don't know anything about that mod, but I will do a search and figure it out.  Also, I should note that my box hasn't been updated (YUM) in a very long time (stupid, I know).  When I try to run yum update either through the web interface or through the command line, I get this in the log:

--> Processing Dependency: perl(Mail::DKIM) >= 0.20 for package: spamassassin
--> Processing Dependency: perl(HTTP::GHTTP) for package: perl-libwww-perl
--> Processing Dependency: pam_abl forftp://ftp.planetmirror.com/pub/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 4] IOError: [Errno ftp error] 550 7: No such file or directory
Trying other mirror.
http://distro.ibiblio.org/pub/linux/distributions/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://ftp.nluug.nl/os/Linux/distr/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Error: failure: CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm from smeos: [Errno 256] No more mirrors to try.
 package: e-smith-base
--> Processing Dependency: mod_auth_tkt for package: e-smith-manager
--> Processing Dependency: perl(IO::Socket::SSL) for package: spamassassin
--> Processing Dependency: smeserver-locale-sv for package: smeserver-s

No new RPM's are installed.  I can see that those addresses are returning 404, but I don't have any idea what to do about it, or how to get the correct addresses.  Note that these results are the same running from a command prompt as root, or running through the web interface.

Did you actually see some of the incoming e-mails? Are they addressed to existing e-mail accounts? Are the e-mails coming from the same source?

We unfortunately always have about 100 - 200 emails per day addressed to random, non-existent email addresses.  This has always been the case, and seems to be continuing.  However, when I get hit with the hundreds of connections I'm not really seeing a major increase in the number of emails logged leading up to the 40 connection limit message.  So, I don't think that these connections are actual SPAM messages, however they could be.  I'm not sure exactly how to verify that.  My understanding is that the SME server should drop any connection that is requesting a user who our exchange box would reject. (Please correct me if I'm wrong there)  We only have 10 users, and very low legit email traffic.


At the moment, the massive number of connections has stopped and the server seems to be operating normally except that I cannot successfully run yum update.  I think I should proceed by getting the box updated asap, but I'm not sure how to get yum update to work.

[mmccarn]

Do some searching on 'yum' here (in the forum), in the wiki, and in bugzilla.

There are lots of potential issues, and LOTS of stuff to update - you may want to download the 7.2 ISO and update from that.

[dickmorrell]

Thats totally right there are a lot of updates and the very impressive thing is that ALL spam hitting my users inboxes has ceased. Not one false positive and trailing the logs greylisting is working very efficiently too. I was getting a lot of stuff missing the filters and greylisting wasn't effective. Now since the update it's been perfect - absolutely perfect. Truly imrpessed and as an SME user since Mitel days this latest incarnation has to be applauded. It soundly beats even the enterprise version of ClarkConnect into a cocked hat - by a royal mile.

All I can think is why you are 404'ing is your mirror list needs updating. Mine 404'd and also dependency failed for a few days until I finally got it working this morning when the mirrors had sync'd. Problem hasn't happened since and it was happening every 2 hrs for last four or five days before. Also the spam benefit is obvious with the new perl libraries and SpamAssassin updates combined with the Greylisting mod.

Given my parents named me after a piece of genitalia and my email address was harvested from 1997 onwards I don't much stand a chance. The current 7.2 beats Cloudmark/AmavisD combinations and Barracuda and Proofpoint soundly.

One thing though:

I keep getting the following repeated in my qpsmtpd log file - any clues ?

Use of uninitialized value in pattern match (m//) at /usr/share/qpsmtpd/plugins/greylisting line 209.

Thanks

Richard

[newhopenet]

Ultimately, this thread:  http://forums.contribs.org/index.php?topic=37970.0 got me to where I could follow the instructions here: http://wiki.contribs.org/Updating_to_SME_7.2

After following those instructions, I was able to run yum update.

Now, I believe all software is up-to-date.  I have not yet seen the "many connections" problem appear again.  I will just hope it doesn't return for now.

[CharlieBrady]

I have seen exactly the same issue three times this week and it seems to be BIND getting in a fix ...

SME server does not have BIND installed (or running, obviously).

[newhopenet]

Just wanted to post this in case there is any other newbie out there, like me, that had not run YUM UPDATE in a long time. 

Run It! 

Since I got my box updated to 7.2 and any additional packages beyond that as well, my "too many connections problem" has disappeared, and this server is operating far more efficiently than before, AND Spam filtering is remarkable.  Lesson learned -- keep the box updated!

To those experts who maintain SME Server -- WOW.  Thank you.

[newhopenet]

Arrrrrgh!  Problem has returned, even with all the updates.

Server stops receiving mail, senders get delivery delay messages. 

Box is fully updated to 7.2 +any other updates found by the yum update command.

Our SME Server sits in a DMZ behind our firewall, and hands off incoming mail to our exchange server.  Outgoing mail is handled by the exchange server only, the SME box should not be sending mail.  This 'mail proxying' is the only thing our SME server is used for, no other functions are used.  It's on a 550Mhz Pentium box with 1GB of ram.  We only have 10 users (mail recipients) on our whole network.

I've pasted some sections of log files below.  I'm really new at this, and have no idea what is "normal" or "abnormal" in these log files.  I'm just hoping someone can spot something and point me in the right direction towards troubleshooting this problem.


This was the clamd/current log just as the problem reappeared:

Code: [Select]

2007-08-26 19:54:30.250558500 Database correctly reloaded (149167 signatures)
2007-08-26 21:02:41.625353500 SelfCheck: Database status OK.
2007-08-26 21:59:18.671036500 SelfCheck: Database status OK.
2007-08-26 22:40:05.587130500 SelfCheck: Database status OK.
2007-08-26 23:32:00.666187500 SelfCheck: Database status OK.
2007-08-27 00:13:16.561827500 SelfCheck: Database status OK.
2007-08-27 00:48:38.815830500 SelfCheck: Database status OK.
2007-08-27 02:15:57.896391500 SelfCheck: Database status OK.
2007-08-27 03:54:53.900529500 SelfCheck: Database status OK.
2007-08-27 05:51:47.990194500 SelfCheck: Database status OK.
2007-08-27 06:46:47.243520500 SelfCheck: Database status OK.
2007-08-27 07:01:05.028816500 Reading databases from /var/clamav
2007-08-27 07:02:38.883306500 Database correctly reloaded (149173 signatures)
2007-08-27 07:21:28.546203500 SelfCheck: Database status OK.
2007-08-27 07:51:54.220319500 SelfCheck: Database status OK.
2007-08-27 08:54:26.731284500 SelfCheck: Database modification detected. Forcing reload.
2007-08-27 08:55:18.230453500 Reading databases from /var/clamav
2007-08-27 08:55:34.168043500 Database correctly reloaded (149179 signatures)
2007-08-27 09:54:27.693210500 SelfCheck: Database modification detected. Forcing reload.
2007-08-27 09:54:27.693218500 Reading databases from /var/clamav
2007-08-27 09:54:40.696665500 Database correctly reloaded (149190 signatures)
2007-08-27 13:54:30.727351500 SelfCheck: Database modification detected. Forcing reload.
2007-08-27 13:54:30.727360500 Reading databases from /var/clamav
2007-08-27 13:54:45.642734500 Database correctly reloaded (149272 signatures)


This is qpsmtp/current:

Code: [Select]

2007-08-27 08:10:46.594736500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:10:46.660797500 15310 Accepted connection 39/40 from 208.70.185.49 / givestrength.com
2007-08-27 08:10:46.662597500 15310 Connection from givestrength.com [208.70.185.49]
2007-08-27 08:10:46.770094500 15310 check_smtp_forward plugin: newhope: 192.168.111.2
2007-08-27 08:10:46.771537500 15310 check_smtp_forward plugin: newhopefellowship.com: 192.168.111.2
2007-08-27 08:10:47.598473500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:10:47.809345500 15310 check_earlytalker plugin: remote host said nothing spontaneous, proceeding
2007-08-27 08:10:47.918993500 15310 220 david.newhopefellowship.com ESMTP
2007-08-27 08:10:47.957301500 15310 dispatching EHLO givestrength.com
2007-08-27 08:10:47.991191500 15310 250-newhopefellowship.com Hi givestrength.com [208.70.185.49]
2007-08-27 08:10:47.992526500 15310 250-PIPELINING
2007-08-27 08:10:47.993839500 15310 250-8BITMIME
2007-08-27 08:10:47.995177500 15310 250 SIZE 15000000
2007-08-27 08:10:48.030933500 15310 dispatching MAIL FROM:<phoenixuni@floppyshoes.com> BODY=8BITMIME
2007-08-27 08:10:48.033118500 15310 full from_parameter: FROM:<phoenixuni@floppyshoes.com> BODY=8BITMIME
2007-08-27 08:10:48.035715500 15310 from email address : [<phoenixuni@floppyshoes.com>]
2007-08-27 08:10:48.129683500 15310 getting mail from <phoenixuni@floppyshoes.com>
2007-08-27 08:10:48.131172500 15310 250 <phoenixuni@floppyshoes.com>, sender OK - how exciting to get mail from you!
2007-08-27 08:10:48.133021500 15310 dispatching RCPT TO:<becky@newhopefellowship.com>
2007-08-27 08:10:48.135763500 15310 to email address : [<becky@newhopefellowship.com>]
2007-08-27 08:10:48.173780500 15310 check_smtp_forward plugin: Checking <becky@newhopefellowship.com> on 192.168.111.2:25
2007-08-27 08:10:48.197471500 15310 check_smtp_forward plugin: 192.168.111.2 would accept message to <becky@newhopefellowship.com>
2007-08-27 08:10:48.207775500 15310 250 <becky@newhopefellowship.com>, recipient ok
2007-08-27 08:10:48.209640500 15310 dispatching DATA
2007-08-27 08:10:48.211823500 15310 354 go ahead
2007-08-27 08:10:48.410242500 15310 spooling message to disk
2007-08-27 08:10:48.606347500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:10:49.614189500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:10:50.618069500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:10:59.644782500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:11:00.647755500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:11:01.650546500 4796 Too many connections: 40 >= 40.  Waiting one second.
2007-08-27 08:11:02.653527500 4796 Too many connections: 40 >= 40.  Waiting one second.
(...this continues indefinitely)

This is qmail/current, leading up to the time the problem started:

Code: [Select]

2007-08-27 07:45:14.741031500 new msg 963295
2007-08-27 07:45:14.741043500 info msg 963295: bytes 31995 from <ESC1101779874249_1101412530530_2366@in.constantcontact.com> qp 14946 uid 453
2007-08-27 07:45:17.033575500 starting delivery 72: msg 963295 to remote josh@newhopefellowship.com
2007-08-27 07:45:17.033585500 status: local 0/10 remote 1/20
2007-08-27 07:45:18.420194500 new msg 963297
2007-08-27 07:45:18.672107500 info msg 963297: bytes 6762 from <WewCQfYx2EoAe0VAWuIoLwgNuTA3tolksy8HPYciq@525311.reply.touchhair.com> qp 14947 uid 453
2007-08-27 07:45:25.240238500 starting delivery 73: msg 963297 to remote cliff@newhopefellowship.com
2007-08-27 07:45:25.240247500 status: local 0/10 remote 2/20
2007-08-27 07:45:28.989887500 delivery 72: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0__<1101779874249.1101412530530.2366.0.2908002E@scheduler>_Queued_mail_for_delivery/
2007-08-27 07:45:32.826697500 status: local 0/10 remote 1/20
2007-08-27 07:45:32.826705500 delivery 73: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0__<qIuU2boX82iWM2kSI94890tu7HWn6YySvN2qEe60V@6ogWK9cbScYciec2ag0VnyIZgp6ZzU1aIyiR9sWCu.touchhair.com>_Queued_mail_for_delivery/
2007-08-27 07:45:32.826731500 status: local 0/10 remote 0/20
2007-08-27 07:45:33.216880500 end msg 963295
2007-08-27 07:45:34.387504500 end msg 963297
2007-08-27 07:45:38.830533500 new msg 963301
2007-08-27 07:45:38.830539500 info msg 963301: bytes 5149 from <DebtExperts@sunvessels.com> qp 14950 uid 453
2007-08-27 07:45:42.480491500 starting delivery 74: msg 963301 to remote ann@newhopefellowship.com
2007-08-27 07:45:42.480499500 status: local 0/10 remote 1/20
2007-08-27 07:45:52.630314500 delivery 74: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0_<GOLIATHpQq5pneeFUtL00000156@goliath.NEWHOPE>_Queued_mail_for_delivery/
2007-08-27 07:45:53.337599500 status: local 0/10 remote 0/20
2007-08-27 07:45:53.337606500 end msg 963301
2007-08-27 07:46:24.529286500 new msg 963295
2007-08-27 07:46:24.529295500 info msg 963295: bytes 32006 from <ESC1101779874249_1101412530530_2026@in.constantcontact.com> qp 14955 uid 453
2007-08-27 07:46:28.428117500 starting delivery 75: msg 963295 to remote becky@newhopefellowship.com
2007-08-27 07:46:28.428127500 status: local 0/10 remote 1/20
2007-08-27 07:46:39.269011500 delivery 75: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0__<1101779874249.1101412530530.2026.0.2908002E@scheduler>_Queued_mail_for_delivery/
2007-08-27 07:46:40.186992500 status: local 0/10 remote 0/20
2007-08-27 07:46:40.426944500 end msg 963295
2007-08-27 07:55:23.960865500 new msg 963346
2007-08-27 07:55:24.049762500 info msg 963346: bytes 32150 from <ESC1101779874249_1101412530530_2026@in.constantcontact.com> qp 15155 uid 453
2007-08-27 07:55:24.680350500 starting delivery 76: msg 963346 to remote becky@newhopefellowship.com
2007-08-27 07:55:24.680358500 status: local 0/10 remote 1/20
2007-08-27 07:55:26.896372500 delivery 76: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0__<1101779874249.1101412530530.2026.0.2908002E@scheduler>_Queued_mail_for_delivery/
2007-08-27 07:55:26.896385500 status: local 0/10 remote 0/20
2007-08-27 07:55:26.957183500 end msg 963346
2007-08-27 07:55:36.363116500 new msg 963346
2007-08-27 07:55:36.363123500 info msg 963346: bytes 9845 from <linensthings@helpfuleccentric.com> qp 15157 uid 453
2007-08-27 07:55:36.558074500 starting delivery 77: msg 963346 to remote ann@newhopefellowship.com
2007-08-27 07:55:36.558082500 status: local 0/10 remote 1/20
2007-08-27 07:55:36.671400500 delivery 77: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0__<8369f7a5e01b$914309748$387382101@helpfuleccentric.com>_Queued_mail_for_delivery/
2007-08-27 07:55:36.671414500 status: local 0/10 remote 0/20
2007-08-27 07:55:36.671417500 end msg 963346
2007-08-27 07:55:36.970489500 new msg 963347
2007-08-27 07:55:36.970494500 info msg 963347: bytes 2455 from <CostaDevelopers@gearfiber.net> qp 15158 uid 453
2007-08-27 07:55:37.287563500 starting delivery 78: msg 963347 to remote ann@newhopefellowship.com
2007-08-27 07:55:37.287571500 status: local 0/10 remote 1/20
2007-08-27 07:55:37.382627500 delivery 78: success: 192.168.111.2_accepted_message./Remote_host_said:_250_2.6.0__<200708270815.l7R8FYkj008212@rharb190.firemanadvise.net>_Queued_mail_for_delivery/
2007-08-27 07:55:37.383098500 status: local 0/10 remote 0/20
2007-08-27 07:55:37.383103500 end msg 963347

I appreciate any advice anyone can offer on what I should check next.  Thank you.

[raem]

newhopenet

Did you enable RBL rejection ?
http://wiki.contribs.org/Email

Did you customise the spam filter (by selecting Custom) to reject messages if the spam score is higher than the score you nominate ?
see server manager Email panel

Did you install the LearnAsSpam contrib that adds Bayesian filtering using the sonoraccom Howto ?
http://wiki.contribs.org/Email


You can also reduce the number of concurrent connections that qmail will handle to reduce the load on your server, although I suspect if you configure RBL & spammassassin correctly then the 40 connections setting will probably be OK.
Given that you are running a lower powered server, I'd probably reduce that to 20 or even 10.
See ConcurrencyRemote in
config show qmail

config setprop qmail ConcurrencyRemote 10
signal-event email-update

[newhopenet]

Thanks for your reply, and for taking the time to help me here.  This problem came up very suddenly two days ago, prior to that our system had been operating very smoothly for over a year.


Did you enable RBL rejection ?
Yes, it is enabled.  We have used this RBL / SBL setup for a very long time with great success, almost no SPAM and I can't think of a single false positive.

I ran this command to confirm that they are enabled:

Code: [Select]

[root@david ~]# config show qpsmtpd
qpsmtpd=service
    Bcc=disabled
    BccMode=cc
    BccUser=maillog
    DNSBL=enabled
    LogLevel=6
    MaxScannerSize=25000000
    RBLList=zen.spamhaus.org
    RHSBL=enabled
    RequireResolvableFromHost=no
    SBLList=dsn.rfc-ignorant.org
    access=public
    status=enabled

Did you customise the spam filter (by selecting Custom) to reject messages if the spam score is higher than the score you nominate ?
Yes.  It is enabled, set to custom, and should reject mail with a score higher than 5.  People tell me this score is low, however we have used it for over a year with no problems at all.

Did you install the LearnAsSpam contrib that adds Bayesian filtering using the sonoraccom Howto ?
Yes, I've used Bayesian filtering for a while and our users are able to 'train' it by moving uncaught SPAM to a public folder on the exchange server, which I then "learn" on a weekly basis.  We have a few thousand of both ham and spam in our Bayes database.

You can also reduce the number of connections per IP setting to reduce the load on your server, although I suspect if you configure RBL & spammassassin correctly then the 40 connections setting will probably be OK.
I will investigate doing that as you suggest.  However, I'm concerned that will only make the problem worse.  If I'm getting many connections, and I reduce the maximum number the server will deal with, won't that just cause additional rejection messages and additional delays?

One other forum post, where a similar problem was discussed, it was suggested that user run "netstat -an" When I do so, it reveals hundreds of connections, a few I've copied below:

Code: [Select]

tcp        1      0 192.168.222.2:25            206.162.204.150:63923       CLOSE_WAIT
tcp     5473      0 127.0.0.1:783               127.0.0.1:39634             CLOSE_WAIT
tcp        0      0 192.168.222.2:25            61.109.102.53:1275          ESTABLISHED
tcp    50621      0 127.0.0.1:783               127.0.0.1:39618             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            62.118.56.62:65397          CLOSE_WAIT
tcp      970      0 127.0.0.1:783               127.0.0.1:39650             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            85.180.169.121:4913         CLOSE_WAIT
tcp        1      0 192.168.222.2:25            202.78.162.223:1945         CLOSE_WAIT
tcp        1      0 192.168.222.2:25            65.12.104.160:63534         CLOSE_WAIT
tcp        1      0 192.168.222.2:25            125.74.163.234:3066         CLOSE_WAIT
tcp     7256      0 127.0.0.1:783               127.0.0.1:39585             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            211.252.104.90:1832         CLOSE_WAIT
tcp        1      0 192.168.222.2:25            200.127.121.24:1976         CLOSE_WAIT
tcp        1      0 192.168.222.2:25            89.208.155.146:58400        CLOSE_WAIT
tcp        1      0 192.168.222.2:25            210.213.84.162:4443         CLOSE_WAIT
tcp     5375      0 127.0.0.1:783               127.0.0.1:39552             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            70.42.193.103:40155         CLOSE_WAIT
tcp        1      0 192.168.222.2:25            58.141.205.54:4273          CLOSE_WAIT
tcp     2475      0 127.0.0.1:783               127.0.0.1:39632             CLOSE_WAIT
tcp     2357      0 127.0.0.1:783               127.0.0.1:39616             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            62.118.56.62:65351          CLOSE_WAIT
tcp        1      0 192.168.222.2:25            125.137.196.238:3595        CLOSE_WAIT
tcp     5226      0 127.0.0.1:783               127.0.0.1:39648             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            196.201.93.75:4824          CLOSE_WAIT
tcp        1      0 192.168.222.2:25            89.111.97.6:2111            CLOSE_WAIT
tcp        1      0 192.168.222.2:25            89.208.155.228:6567         CLOSE_WAIT
tcp     7242      0 127.0.0.1:783               127.0.0.1:39559             CLOSE_WAIT
tcp     5376      0 127.0.0.1:783               127.0.0.1:39607             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            125.137.196.238:4428        CLOSE_WAIT
tcp        1      0 192.168.222.2:25            200.161.167.135:2270        CLOSE_WAIT
tcp     2221      0 127.0.0.1:783               127.0.0.1:39623             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            123.22.12.209:34788         CLOSE_WAIT
tcp        0      0 127.0.0.1:783               127.0.0.1:39543             CLOSE_WAIT
tcp        1      0 192.168.222.2:25            66.218.67.71:23182          CLOSE_WAIT
tcp        1      0 192.168.222.2:25            89.49.86.135:2334           CLOSE_WAIT
tcp        0      0 127.0.0.1:39583             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39581             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39578             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39579             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39577             127.0.0.1:783               FIN_WAIT2
tcp     9933      0 127.0.0.1:783               127.0.0.1:39558             CLOSE_WAIT
tcp        0      0 127.0.0.1:39558             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39559             127.0.0.1:783               FIN_WAIT2
tcp        1      0 192.168.222.2:25            70.42.193.103:59805         CLOSE_WAIT
tcp        0      0 127.0.0.1:39557             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39552             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39564             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39563             127.0.0.1:783               FIN_WAIT2
tcp     5853      0 127.0.0.1:783               127.0.0.1:39606             CLOSE_WAIT
tcp        0      0 127.0.0.1:39606             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39607             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39612             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39610             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39608             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39585             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39598             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39599             127.0.0.1:783               FIN_WAIT2
tcp     9944      0 127.0.0.1:783               127.0.0.1:39638             CLOSE_WAIT
tcp        0      0 127.0.0.1:39638             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39636             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39634             127.0.0.1:783               FIN_WAIT2
tcp        0      0 127.0.0.1:39632             127.0.0.1:783               FIN_WAIT2
(and so on .....)

Any further ideas?

[raem]

newhopenet

  RBLList=zen.spamhaus.org

You could add more RBL lists, see the current suggestions for sme7.2
http://wiki.contribs.org/Updating_to_SME_7.2#DNSBL_Servers

.... reject mail with a score higher than 5.

Yes that's very low and you are probably rejecting real messages, I have seen lot's of legitimate messages get a spam score of 5. A rejection score of 10 or 12 would be more realistic.
What does this show ?
config show spamassassin

If I'm getting many connections, and I reduce the maximum number the server will deal with, won't that just cause additional rejection messages and additional delays?

You are reducing the number that the server will deal with at the same time. Your lower powered processor is trying to deal with too many connections, it can't handle them all, thus causing errors and delays.
I'd try changing the setting to 10, you only have a few users so you (I assume/guess) don't have hundreds of messages a minute coming in.

[raem]

newhopenet

Re number of connections, I meant to say
You can also reduce the number of concurrent connections that qmail will handle to reduce the load on your server, although I suspect if you configure RBL & spammassassin correctly then the 40 connections setting will probably be OK.
Given that you are running a lower powered server, I'd probably reduce that to 20 or even 10.
See ConcurrencyRemote in
config show qmail

config setprop qmail ConcurrencyRemote 10
signal-event email-update

RHSBL=enabled
    SBLList=dsn.rfc-ignorant.org

I'd also try disabling RHSBL as that can cause slowdown

[newhopenet]

You could add more RBL lists, see the current suggestions for sme7.2

OK -- I've added a couple.  A few of those did, in fact, cause a lot of false positives for us.  More than once we had mail from legit Yahoo Mail users rejected by some of those lists.  So, I'm cautious about them.

Code: [Select]

[root@david ~]# config show qpsmtpd
qpsmtpd=service
    Bcc=disabled
    BccMode=cc
    BccUser=maillog
    DNSBL=enabled
    LogLevel=6
    MaxScannerSize=25000000
    RBLList=multihop.dsbl.org:dnsbl-1.uceprotect.net:zen.spamhaus.org
    RHSBL=enabled
    RequireResolvableFromHost=no
    SBLList=dsn.rfc-ignorant.org
    access=public
    status=enabled
[root@david ~]#


that's very low and you are probably rejecting real messages

Yes, everyone tells me that, but I really don't think we are rejecting any real messages.  I watch it carefully -- and I check headers of emails regularly to see where they are scoring.  But, maybe something has changed recently...I don't know.  If you think I should try raising it, I will.

What does this show ?
config show spamassassin

Code: [Select]

[root@david ~]# config show spamassassin
spamassassin=service
    BayesAutoLearnThresholdNonspam=0.10
    BayesAutoLearnThresholdSpam=7.00
    DNSAvailable=yes
    MessageRetentionTime=90
    OkLanguages=all
    OkLocales=all
    RejectLevel=5
    ReportSafe=0
    Sensitivity=custom
    SkipRBLChecks=0
    SortSpam=disabled
    Subject=[SPAM]
    SubjectTag=disabled
    TagLevel=3
    UseBayes=1
    status=enabled
[root@david ~]#


I'd try changing the setting to 10, you only have a few users so you (I assume/guess) don't have hundreds of messages a minute coming in.

OK -- reading the wiki, it talks about that limit dealing with IMAP.  Correct me if I'm wrong here, but the command I saw was "db configuration setprop imap variable value".  This seems to deal with the number of mail clients who can connect to check their mail.  For me however, this could be zero.  No clients check their mail on my SME server.  The SME server is just a proxy for mail -- it passes all mail to my exchange box where all the user accounts reside.

Nevertheless, I did go ahead and reduce that value to 10.  Did I do the right thing?


-------  oops, Ok -- Saw your new post -----

Re number of connections, I meant to say
You can also reduce the number of concurrent connections that qmail will handle to reduce the load on your server, although I suspect if you configure RBL & spammassassin correctly then the 40 connections setting will probably be OK.
Given that you are running a lower powered server, I'd probably reduce that to 20 or even 10.
See ConcurrencyRemote in
config show qmail

config setprop qmail ConcurrencyRemote 10
signal-event email-update

OK.  Did that.  Down to 10.


At the moment, the mass number of connections has subsided.  But I feel a false sense of security, as this problem comes and goes randomly.

[newhopenet]

When it comes to us using a lower powered system, I should point out that we don't use many SME features.

No remote access, no FTP, not a part of a workgroup or domain, no printers or print server functions, no users (except admin), no groups, no used I-Bays, no one accessing files, no POP3, no IMAP, no webmail, no outgoing mail.

Since the server is literally only handing incoming mail and nothing else, shouldn't 550MHz with 1GB of RAM be able to handle this task?  It has done well up until 3 days ago???

[raem]

newhopenet

...I really don't think we are rejecting any real messages.  I watch it carefully -- and I check headers of emails regularly to see where they are scoring.

Your tag level is 3 which is the score level that messages would get moved to the junkmail folder if you used that function (you are using exchange instead so I don't know what you do with tagged messages), but your reject level is 4.
Reject means that the messages get rejected at smtp level, you never receive them so how can you read them !

At the moment, the mass number of connections has subsided.  But I feel a false sense of security, as this problem comes and goes randomly.

Well that's the nature of spam & spammers & hackers & crackers, there one minute gone the next. Cyclic behaviour is OK and to be expected. Remember that RBL lists will pick up new spammers within a few hours or so, so it takes a finite time for new spammers messages to start getting rejected.
In the meantime though the reduced number of connections will keep your server from overloading, qmail/qpsmtpd/clamav/spamassassin will use all memory & processor power too if the messages are laden with viruses or spam content.
The box is OK for your usage pattern, but the  lower setting suggested is required to prevent lockups.
I have one sme7.2 server that is Celeron 500MHz with 256Mb RAM, serving mail to 7 users in a busy small office, plus printing & file serving & a few small web sites and it runs perfectly OK, tweaked appropriately of course.

[newhopenet]

Thanks so much for your advice and your time. 

Obviously, you're right that I can't be reading messages that hit the reject level.  Messages that hit our tag level are sorted (by exchange) into the user's junk mail folder in Outlook.  Those, I review.  As I do any uncaught SPAM.  It's just that we've never had any complaints about people's mail not getting through (until recent days).  I'll raise the level a bit.

I think I'm also having spamassassin problems.  All of today's mail has had no tests performed on it, and I'm getting SPAM in my inbox today as well.  I'm going to do a search on the forum since this may be an unrelated problem.  Let me know if you think it is related.

Header Sample is below, this sender is in my white list.  On mail they sent yesterday, it scored a -100, today it scores a 0.0

X-Virus-Checked: Checked by ClamAV on newhopefellowship.com
X-Spam-Status: No, hits=0.0 required=3.0
   tests=
X-Spam-Check-By: newhopefellowship.com

from spamd/current (not really sure if this looks like normal operation or not...

Code: [Select]

2007-08-27 20:13:30.488485500 [7027] info: prefork: child states: II
2007-08-27 20:18:22.742706500 [7044] info: spamd: connection from localhost [127.0.0.1] at port 33472
2007-08-27 20:18:22.774011500 [7044] info: spamd: checking message <-WAYAWzToXA1zb5DJzx5sQ@xendep.com> for qpsmtpd:1005
2007-08-27 20:18:23.167034500 [7044] info: spamd: clean message (0.0/3.0) for qpsmtpd:1005 in 0.4 seconds, 2706 bytes.
2007-08-27 20:18:23.167896500 [7044] info: spamd: result: . 0 - scantime=0.4,size=2706,user=qpsmtpd,uid=1005,required_score=3.0,rhost=localhost,raddr=127.0.0.1,rport=33472,mid=<-WAYAWzToXA1zb5DJzx5sQ@xendep.com>,autolearn=failed
2007-08-27 20:18:23.689652500 [7027] info: prefork: child states: II
2007-08-27 20:20:09.691137500 [7044] info: spamd: connection from localhost [127.0.0.1] at port 33475
2007-08-27 20:20:10.106439500 [7044] info: spamd: checking message <050401c7e911$931744c0$6401a8c0@CRAPPER> for qpsmtpd:1005
2007-08-27 20:20:13.913168500 [7044] info: spamd: clean message (1.5/3.0) for qpsmtpd:1005 in 4.2 seconds, 36387 bytes.
2007-08-27 20:20:13.914040500 [7044] info: spamd: result: . 1 - MY_CID_AND_ARIAL2 scantime=4.2,size=36387,user=qpsmtpd,uid=1005,required_score=3.0,rhost=localhost,raddr=127.0.0.1,rport=33475,mid=<050401c7e911$931744c0$6401a8c0@CRAPPER>,autolearn=no
2007-08-27 20:20:14.737335500 [7027] info: prefork: child states: II
2007-08-27 20:27:30.016748500 [7044] info: spamd: connection from localhost [127.0.0.1] at port 33479
2007-08-27 20:27:30.055784500 [7044] info: spamd: checking message <000501c62de1$c7159610$88fc087b@zhang> for qpsmtpd:1005
2007-08-27 20:27:32.391804500 [7044] info: spamd: clean message (2.1/3.0) for qpsmtpd:1005 in 2.4 seconds, 1916 bytes.
2007-08-27 20:27:32.392696500 [7044] info: spamd: result: . 2 - MIME_QP_LONG_LINE,SARE_SXLIFE scantime=2.4,size=1916,user=qpsmtpd,uid=1005,required_score=3.0,rhost=localhost,raddr=127.0.0.1,rport=33479,mid=<000501c62de1$c7159610$88fc087b@zhang>,autolearn=no
2007-08-27 20:27:32.860821500 [7027] info: prefork: child states: II
2007-08-27 20:28:56.759171500 [7044] info: spamd: connection from localhost [127.0.0.1] at port 33482
2007-08-27 20:28:56.820361500 [7044] info: spamd: checking message <rd1808_101-33382-julienewhopefellowship.com@smtp1.rapiddeliveryserver.com> for qpsmtpd:1005
2007-08-27 20:28:58.131037500 [7044] info: spamd: clean message (0.0/3.0) for qpsmtpd:1005 in 1.4 seconds, 8648 bytes.
2007-08-27 20:28:58.131048500 [7044] info: spamd: result: . 0 - scantime=1.4,size=8648,user=qpsmtpd,uid=1005,required_score=3.0,rhost=localhost,raddr=127.0.0.1,rport=33482,mid=<rd1808_101-33382-julienewhopefellowship.com@smtp1.rapiddeliveryserver.com>,autolearn=failed
2007-08-27 20:28:58.735779500 [7027] info: prefork: child states: II

[raem]

All of today's mail has had no tests performed on it, and I'm getting SPAM in my inbox today as well.

I can only suggest to check the following:
Check correct repositories are enabled see wiki (re upgrading) for details
Run
yum clean all
run
yum list updates
to see if any more updates are available
Then if required
yum update


Then check ALL your email related settings in server manager VERY CAREFULLY, redo and save them in case settings have been corrupted.

You say:
"This problem came up very suddenly two days ago, prior to that our system had been operating very smoothly for over a year."

What did you do to the system prior to that ?


>  I'll raise the level a bit.

To about 12 if you want to be sure of receiving all legitimate email.


Look in other log files for clues

[kruhm]

is this really THE SmoothWall dickmorrell?

[raem]

newhopenet

I think I'm also having spamassassin problems.  All of today's mail has had no tests performed on it, and I'm getting SPAM in my inbox today as well.

There is another thread that refers to updating again. There was a new version of spamassassin released that fixes some problems.
I thought this was released a few days/week ago, when there was two or three spamassassin updates in a row each day, but maybe some people got the first update and not the later ones.

[mmccarn]

At the moment, the mass number of connections has subsided.  But I feel a false sense of security, as this problem comes and goes randomly.

You seem to be seeing behavior similar to what I have seen on 4 separate SME 7 servers since last November.

In every case the problem would pop up, give me headaches for a few days to a couple weeks, then go away.

I fought with this off and on last November on a couple servers -- then it went away.

I fought with this in January on one server, then it went away until about 3 weeks ago.

I fought with it again in May or June on a different server -- then the client chose to bypass the SME and have email delivered directly to the Exchange server.

I can *always* control the situation by creating iptables block rules for destination =port 25 on my server and sources = all hosts that have been blocked by either dnsbl or check_earlytalker from /var/log/qpsmtpd/* and restarting qpsmtpd.

The hardware in question ranges from underpowered (pIII / 933MHz / 192MB RAM) to virtual (running on dual xeon 2.8GHz hardware) to 'should-be-ok' (xeon 2.8GHz, 4GB RAM...). 

All of my systems are gateways for internal mail servers - as yours seems to be...

I've tried adjusting the various settings, enabling & disabling various modules, all with no concrete results.

[CharlieBrady]

is this really THE SmoothWall dickmorrell?

Yes, same Dick. Sometimes he knows what he is talking about, others not.

[CharlieBrady]

3) qpsmtpd/current shows only 'Too many connections: 40 >=40. Waiting one second" logged over and over and over.

That only indicates that SMTP connections are arriving faster than your box can finish handling the ones it is processing. The logs files will show whether that is because there are more connections, or because your system is taking longer to process the connections it is handling. If it is taking longer, we need to determine why it is taking longer, and what can be done to change that.

Please take this issue to the Bug Tracker, and provide details from your log files there.