Topic: Too Many Connections, Sever Slows, Stops

[newhopenet]

I think I am being attacked with a Denial Of Service.  I'm very new to SME Server, and it is entirely possible that I'm wrong.  I'm willing to work and research to find answers, and I've read the forums, but I'm now stuck not knowing how to proceed.

Symptoms:

1) Server slows to a complete crawl, we stop getting all incoming mail.  Mail is returned to sender with a 'delivery delay' message.

2) Running netstat -an | grep :25.*EST results 70 - 80 connections.  Running it without the *.EST results in hundreds of connections in various states, from many different IP's.

3) qpsmtpd/current shows only 'Too many connections: 40 >=40. Waiting one second" logged over and over and over.

I've read the forums extensively, and this is as far as I can get based on the advice posted in the forums.  I now know the above things, but I have no idea what my next step is to resolve this.  What other information should I provide that would be relevant?

My SME server sits in a DMZ behind my firewall.  It processes incoming mail and then hands it off to our exchange server.  That is the only job of this SME server, no other functions are used.  It should not send mail.  Outgoing mail is sent directly from our exchange server.  The SME Server is a 550MHz machine with 1GB RAM.  Our network only has 10 users.

Is this is a security issue? and What should I do now?

I appreciate any guidance you can provide on what I should look at next.

Thanks!

[byte]

Is this is a security issue?

If you think this is a security issue never post to a public forum as per top of every new thread (before posting)

"Don't report security issues here - Contact security at contribs dot org"

Thanks.

[newhopenet]

Well, first I pretty much doubt myself on determining whether this is an actual security issue or not.  Secondly, the security email address rejects all mail from my gmail account, as they are apparently listed in an SBL.  So, I guess I'll start trying some other free email providers until I can get through.

[dickmorrell]

This is not a denial of service attack in the traditional sense.

Have you applied the greylisting mod on your 7.2 box ?

I have seen exactly the same issue three times this week and it seems to be BIND getting in a fix when faced with large amounts of inbound spam. As soon as I get time I'll file something in Bugzilla properly. Sorry been travelling a lot this week.

[dickmorrell]

Ok now the mirrors seem to have sync'd apply the latest updates to your box with yum upgrade from the command prompt as root. Problem should disappear. Seems to be linked to Perl issues which are solved with the updates to various Perl libraries getting their knickers in the proverbial twist.

If you still get this issue paste the log file or email it to me offline and I'll have a look through it for you.

[byte]

I have seen exactly the same issue three times this week and it seems to be BIND getting in a fix when faced with large amounts of inbound spam.

Where is the evidence of BIND being the issue? the OP h/w is not the most powerful when dealing with large amount of load that they are seeing.

[micropitt]

Did you actually see some of the incoming e-mails? Are they addressed to existing e-mail accounts? Are the e-mails coming from the same source? My first impression is, your domain is receiving large amount of Spam for some reason that you should investigate. If you receive a large amount of e-mails for non existing e-mail accounts it probably is a dictionary spam attack on your domain. One short term solution would be to shut down smtp for a day or two. If it doesn't stop, contact the hosting company for your domain and/or your ISP and they can set some blocks on there Routers.
 

[dickmorrell]

The evidence of BIND being tasked to the max (and it wasn't a dictionary attack) it seemed to be multiple massive amounts of inbound spam to the same real email address (an Ubuntu list recipient) and looking at the logfiles on the firewall (that I wrote..) was that the SME server was attempting huge amounts of dns lookups to RBLs and the port 53 logs in the firewall logs matched to the activity report on the SME server. Just box getting totally maxed out and almost all CPU taken. Box is a dual 1.8ghz CPU with 4GB of RAM and mirrored 32gb SCSI.

[mmccarn]

Too Many Connections, Sever Slows, Stops

I've been having exactly the same symptoms since updating to 7.2 on one of my servers.  I had this issue briefly after updating to 7.1, too.

Did you actually see some of the incoming e-mails?

On my systems there never is any email resulting from these connections.

In fact, I work around this issue by scanning /var/log/qpsmtpd/* for all connections that were denied by dnsbl or by check_earlytalker, and adding them to the firewall with a 'denylog' rule.  This always clears up the problem.  (I've written some really bad scripts that do the scanning and blocking for me...)

When this started again (about 3 weeks ago) I was blocking about 2000 - 4000 hosts.  Now I find I am blocking port 25 from 21000+ hosts that were denied connection to my server during the life of my log files.  This could either indicate an increase in "attack" behavior, or it could simply reflect the change I made to my qpsmtpd LogLevel after upgrading to 7.2 (with logterse).

yum upgrade from the command prompt as root. Problem should disappear

This would be fabulous news.  Do you have any specifics on why this would cause or fix the described behavior (except perhaps that qpsmtpd is written in perl)?

Don't report security issues here - Contact security at contribs dot org

The last time this happened to me (Nov '06) it seemed to be due to hardware that didn't meet SME's recommended requirements for spam and virus filtering -- which is largely why I haven't posted anything about it this time (I felt like an idiot last time...)

[dickmorrell]

Do the update - issue went away - was seeing same issue three or four times a day

Did update - problem went away

Also: Spam has dropped about 50% (of what was getting through) with new SpamAssassin update too

Thanks to all the hardworking package maintainers you're doing good stuff - much appreciated.

[newhopenet]

Thanks so much for all of your replies, I'm working as fast as I can to learn as much as I can, this is all very new to me.  I appreciate your time here.

Have you applied the greylisting mod on your 7.2 box ?

I have not.  I don't know anything about that mod, but I will do a search and figure it out.  Also, I should note that my box hasn't been updated (YUM) in a very long time (stupid, I know).  When I try to run yum update either through the web interface or through the command line, I get this in the log:

--> Processing Dependency: perl(Mail::DKIM) >= 0.20 for package: spamassassin
--> Processing Dependency: perl(HTTP::GHTTP) for package: perl-libwww-perl
--> Processing Dependency: pam_abl forftp://ftp.planetmirror.com/pub/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 4] IOError: [Errno ftp error] 550 7: No such file or directory
Trying other mirror.
http://distro.ibiblio.org/pub/linux/distributions/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://ftp.nluug.nl/os/Linux/distr/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://ftp.surfnet.nl/ftp/pub/os/Linux/distr/smeserver/releases/7/smeos/i386/CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Error: failure: CentOS/RPMS/perl-Compress-Zlib-1.42-1.el4.rf.i386.rpm from smeos: [Errno 256] No more mirrors to try.
 package: e-smith-base
--> Processing Dependency: mod_auth_tkt for package: e-smith-manager
--> Processing Dependency: perl(IO::Socket::SSL) for package: spamassassin
--> Processing Dependency: smeserver-locale-sv for package: smeserver-s

No new RPM's are installed.  I can see that those addresses are returning 404, but I don't have any idea what to do about it, or how to get the correct addresses.  Note that these results are the same running from a command prompt as root, or running through the web interface.

Did you actually see some of the incoming e-mails? Are they addressed to existing e-mail accounts? Are the e-mails coming from the same source?

We unfortunately always have about 100 - 200 emails per day addressed to random, non-existent email addresses.  This has always been the case, and seems to be continuing.  However, when I get hit with the hundreds of connections I'm not really seeing a major increase in the number of emails logged leading up to the 40 connection limit message.  So, I don't think that these connections are actual SPAM messages, however they could be.  I'm not sure exactly how to verify that.  My understanding is that the SME server should drop any connection that is requesting a user who our exchange box would reject. (Please correct me if I'm wrong there)  We only have 10 users, and very low legit email traffic.


At the moment, the massive number of connections has stopped and the server seems to be operating normally except that I cannot successfully run yum update.  I think I should proceed by getting the box updated asap, but I'm not sure how to get yum update to work.

[mmccarn]

Do some searching on 'yum' here (in the forum), in the wiki, and in bugzilla.

There are lots of potential issues, and LOTS of stuff to update - you may want to download the 7.2 ISO and update from that.

[dickmorrell]

Thats totally right there are a lot of updates and the very impressive thing is that ALL spam hitting my users inboxes has ceased. Not one false positive and trailing the logs greylisting is working very efficiently too. I was getting a lot of stuff missing the filters and greylisting wasn't effective. Now since the update it's been perfect - absolutely perfect. Truly imrpessed and as an SME user since Mitel days this latest incarnation has to be applauded. It soundly beats even the enterprise version of ClarkConnect into a cocked hat - by a royal mile.

All I can think is why you are 404'ing is your mirror list needs updating. Mine 404'd and also dependency failed for a few days until I finally got it working this morning when the mirrors had sync'd. Problem hasn't happened since and it was happening every 2 hrs for last four or five days before. Also the spam benefit is obvious with the new perl libraries and SpamAssassin updates combined with the Greylisting mod.

Given my parents named me after a piece of genitalia and my email address was harvested from 1997 onwards I don't much stand a chance. The current 7.2 beats Cloudmark/AmavisD combinations and Barracuda and Proofpoint soundly.

One thing though:

I keep getting the following repeated in my qpsmtpd log file - any clues ?

Use of uninitialized value in pattern match (m//) at /usr/share/qpsmtpd/plugins/greylisting line 209.

Thanks

Richard

[newhopenet]

Ultimately, this thread:  http://forums.contribs.org/index.php?topic=37970.0 got me to where I could follow the instructions here: http://wiki.contribs.org/Updating_to_SME_7.2

After following those instructions, I was able to run yum update.

Now, I believe all software is up-to-date.  I have not yet seen the "many connections" problem appear again.  I will just hope it doesn't return for now.

[CharlieBrady]

I have seen exactly the same issue three times this week and it seems to be BIND getting in a fix ...

SME server does not have BIND installed (or running, obviously).