Topic: SMTP mail overwhelming server

[fosdyke]

Having huge problems with a SME 7.x server which has been running ok up until last Friday, when it slowed to a crawl and would not send or receive mail.

I have wiped it clean in case it had been compromised, but still no joy: as soon as I open port 25 on the router the server goes like a tortoise through treacle!

I am suspecting a dictionary spam attack or something similar, but would appreciate any help or ideas as to what it is and how to solve the problem - company has now been effectively without mail for 3 days..............help!

 

[CharlieBrady]

.....help!

Nobody is likely to be able to help unless you provide more detail. The fact that you have wiped the system clean is also bad - you will have erased evidence as to what the problem was.

[smeghead]

I presume it's on good well spec'd hardware with plenty of RAM & processor time to work with.

Use iptraf to look at activity on eth1 (assuming that is your wan nic) to see what traffic is hitting the system.

Previously when I have seen something similar I have tracked one or more abusive IP's & then added them into my router rules to block connections, problem solved (at least for the moment).

Charlie is right that wiping the server eliminated logs that would have been useful in diagnosing the prob; best to take system off line and investigate or at the very least backup the logs before wiping.

Do you have Spamassassin & Clamav turned on?

Most of the spam SMTP traffic is dropped if you have the system setup correctly.

Also check out Master Sleepy for the Snort contrib (& others) for an adaptive solution that's not perfect but may be useful.

HTH

[raem]

fosdyke

Here are some common reasons for what you are experiencing.

I'd suspect a virus infection on a local workstation that is sending lots of spam emails out.
Look in qpsmtpd & qmail log files for sources of email messages.
Disconnect all workstations to see if messages stop flowing, although you may have a lot of email messages still in the outgoing queue on the server. Install qmhandle to easily look at these.
http://wiki.contribs.org/Qmhandle_mail_queue_manager
Do a virus scan on all workstations.

Another possibility is lots of incoming virus laden & spam emails, which are overloading your system.
Make sure you enable spam filtering, antivirus scanning (of messages) and executable content blocking (all in the Email panel in server manager) and also enable RBL lists (see link below). All of these will combine to reduce the processing load on your CPU, by rejecting a lot of messages that would otherwise be processed and slow your system down to a crawl, see
http://wiki.contribs.org/SME_Server:Documentation:FAQ#Real-time_Blackhole_List_.28RBL.29

There may be other reasons for your problem, but the above are some good starting points to help tame your problem.

You should really look at the log files to try and identify where/what the problem is, as without correctly identifying the problem we are only guessing at what to do to fix it.

[fosdyke]

Thanks for all the replies.

Sorry I had to wipe the box, as I wasn't sure that it hadn't been compromised in some way, and wouldn't do any Yum updates, however as the the problem still exists the current logs may help.

Apologies here but I am not a complete teccy this is what I can say about the problem:

1. It is definitely not a lan virus - this is a network of Macs NO PC's !!
2. Its an inbound mail problem - if I shut down port 25 for incoming traffic - server comes back to life.
3. I do have Spamassassin and Clam AV on - and always have had - as is spam, virus scanning and RBL.
4. From the router logs most of the port 25 traffic comes through the ISP's server - I can't add it to a 'blocked' list otherwise I won't receive ANY mail!
5. The Mail server is not a great spec - 700mhz and 256mb ram but has been OK up until now - but I am increasingly coming to the conclusion I will have to 'upgrade' in the hope that this will sort out the problem.

Which logs should I be looking at? I tend to find the mailogs unhelpful in tracking down this sort of problem, but I am probably reading them wrong.

Any more thoughts????

 

[smeghead]

Firstly, unless the ISP is backup MX or your using multipop then ISP mail server can be blocked; the assunption here is that your system is authoritative for your domain for mail.

Secondly, please provide info on how the system is setup, partucularly on the WAN side.

Thirdly, make sure the ISP mail server is NOT added to the whitelists otherwise the system will accept email from it without testing it for spam.

Fourthly, take a look at the qpsmtpd logs for mail that is passed through that should be rejected, post up samples; make sure you post up all lines relating to a email specific thread in the log.

[raem]

fosdyke

In server manager View log files
Look in qpsmtpd & qmail log files for sources of email messages.

[pfloor]

5. The Mail server is not a great spec - 700mhz and 256mb ram but has been OK up until now - but I am increasingly coming to the conclusion I will have to 'upgrade' in the hope that this will sort out the problem.

Your conclusion is most likely correct.  Those specs will work on a lightly loaded server but if there is more that a few concurrent SMTP connections, it will bog down your modest hardware.  Your machine meets the "minimum" hardware requirements but that will only meet the demands for a file server and gateway.  To utilize the spam/AV you need at least 1.5G processor and 512M ram (see: http://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Chapter4#4.1._Minimum_Hardware_Requirements)

Time to upgrade

[raem]

fosdyke

Those specs will work on a lightly loaded server but if there is more that a few concurrent SMTP connections, it will bog down your modest hardware.

You can adjust a couple of settings that will slow down mail processing to tolerable levels, and allow your lower spec system to keep up with the demand, without users noticing any additional/significant mail delays.

To see current settings (if at all set, as defaults will probably apply)

config show qpsmtpd
config show qmail

You can experiment with the numbers shown below, but these should be appropriate for your hardware.

config setprop qpsmtpd Instances 5
config setprop qmail ConcurrencyRemote 5
config setprop qmail ConcurrencyLocal 10
signal-event email-update


Keep in mind you may still have a lot of messages in queues, so allow time for these messages to be sent, before you assess average performance/improvement due to these setting changes.
Use qmhandle to check queues.


To see what processes are going on when your machine runs slow do
top -i
and
htop

[fosdyke]

Once again thanks for all the thoughts.

Have bitten the bullet and upgraded the server - hey presto problem has now gone away!!

Will take on board comments about ISP mailservers - don't think they are on the whitelists but will check. I have also read something about a 'server-only' setup behind a NAT firewall may have problems identifying spam  - any tips or suggestions on this would be helpful.

Thanks again.

 

[raem]

fosdyke

Have bitten the bullet and upgraded the server

Just interested,
what did you upgrade to ?

Could you show output of
config show qpsmtpd


I run a 500MHz server with 256Mb RAM, with modified mail settings, and it works fine in a small office with light email load.

[fosdyke]

Upgrade to 1.7ghz AMD XP2000 with 512mb ram - will post the qsmtp log later.