Topic: Email Server Overwhelmed...please help

[wingman]

Running SME 6.0.1 with Spamassassin, ClamAV, and mailfront mailblocking. This server is configured in server only mode, it's only job is to filter email and pass it on to an internal Exchange server for delivery.
Problem: twice within the past 2 months this server's remote mail queue has gotten clogged with tens of thousands of outgoing emails addressed to the sayclub.com domain. There are so many there (over 20000) that SME's email server comes to a screeching halt. If I pull SME out of the network and send all incoming mail directly to the exchange server, everything works again. Plus, the exchange server's outgoing logs show no traffic going to the sayclub.com domain. So it appears that this traffic is either originating with the SME server or is somehow being relayed through it.

The last time this happened I formatted the drives and reinstalled SME totally. It has now been running for only 2 weeks and again has this problem. I am looking for a way to find (probably from the command line) all messages in the remote queues that are addressed to the sayclub.com domain and then delete them. Does anyone know how to do this?

Also, does anyone have a clue as to what is going on with this server? I could give someone remote access to it if you want to poke around. And, if you can spot the problem and get it fixed I will gladly pay for your expertise.

Joe

[jfarschman]

Joe,

  Are you using SMTP Authentication?  If you are it's really difficult for a spammer to inject email through your SMTP server.  If you are not... then I'd consider turning it on.  Most email clients can be configured to authenticate before sending messages through your server.

  So... if they aren't coming in on the SMTP ports, then how are they getting in.  Some spammers are actually taking advantage of Apache vulnerabilities and then placing .php files on the server.  While I'm not sure this has happened, or is even possible on the SME servers, I have seen it twice before.  The little .php file is then used to inject emails into the system.  This would be fairly easy to find.  You'd just look in the ibays for .php files that you didn't put there.

  I hope this helps?

[wingman]

Jay,
Thanks so much for your reply. I searched the entire server (using MC) for php files. Didn't find any in any of the ibays. In fact, most that I found were in the httpd folder or one of it's subfolders. Didn't see anything suspicious.

I don't really understand what you mean by having users authenticate. Again, all this server is doing is scanning email for viruses and spam and then sending it on through to an Exchange server. Users don't really attach to the SME box at all. Or am I misunderstanding what you are referring to?

Thanks again, I appreciate your help.

Joe

[jfarschman]

Hmmm,

  So it just accepts email for your domain(s) and forwards all of it to the Exchange Server.  I'm having a hard time seeing where the problem could be if no one is  using the smtp service from the outside.

  I'm not sure I see what it could be.  I could probably spend a few minutes in the server and find the problem.

[wingman]

Jay,

That would be great. Please send an email to me at (wingriders at gmail dot com) and let me know how you want to get in...through ssh or remote server manager?

Joe

[policymap]

wingman did you find a solution?

I've had the exact same setup, and also experienced the same problem.

VJ

[CharlieBrady]

Running SME 6.0.1 with Spamassassin, ClamAV, and mailfront mailblocking. This server is configured in server only mode, it's only job is to filter email and pass it on to an internal Exchange server for delivery.

Then you should have it in server-gateway mode. That will ensure that there is no direct path to the Exchange server.

Problem: twice within the past 2 months this server's remote mail queue has gotten clogged with tens of thousands of outgoing emails addressed to the sayclub.com domain. There are so many there (over 20000) that SME's email server comes to a screeching halt.

You need to make sure that those messages don't get into the queue.

 If I pull SME out of the network and send all incoming mail directly to the exchange server, everything works again. Plus, the exchange server's outgoing logs show no traffic going to the sayclub.com domain. So it appears that this traffic is either originating with the SME server or is somehow being relayed through it.

The log files will tell you exactly where those messages came from.

The last time this happened I formatted the drives and reinstalled SME totally.

Well, there go those log files.

It has now been running for only 2 weeks and again has this problem. I am looking for a way to find (probably from the command line) all messages in the remote queues that are addressed to the sayclub.com domain and then delete them. Does anyone know how to do this?

There's no point in doing that until you do something to stop it from happening again. You already have sound evidence that it *will* happen again if you do nothing.

Search here for qmHandle and you'll find hints on how to clear those messages.

[wingman]

Charlie,

I did find and install qmhandle. It is insufficient for deleting 20000+ messages in the queue since it's only option (that I could find) is to open each email individually and then delete it. That would take forever. By analyzing the current logs (no I did NOT delete those...) I did find where they were coming from. All from the same IP address.

What I did that seems to have solved the problem: I installed the email blocking contrib from Dungog and used it to disallow all email from or to the sayclub.com domain. Then used MC to mass delete all emails in the queue....at this point in time any good emails are so old that they are no longer relevant.

It seems to be working now.

Charlie, evidently I am not a very good communicator. When you say that I need to make sure that the server doesn't have a direct path to the exchange server, I don't really understand how that would help correct this issue. Maybe I'm just dense, but the messages are never getting there anyway since they cause the SME box to stop sending the messages through to Exchange. And, since exchange seems not to even GET these messages if the SME server is taken off of the network and emails are received directly by exchange, I don't see how this would be a problem anyway. Can you enlighten me?

Thanks
Joe

[CharlieBrady]

I did find and install qmhandle. It is insufficient for deleting 20000+ messages in the queue since it's only option (that I could find) is to open each email individually and then delete it. That would take forever. By analyzing the current logs (no I did NOT delete those...) I did find where they were coming from. All from the same IP address.

Was that IP address on the local network? If not, then you need to work out why they were accepted for relaying. Something about your configuration was wrong - perhaps a (very dangerous) bug in one of the contribs.

If they did come from a local network IP address, you need to work out why that IP was generating (or relaying) all those emails.

What I did that seems to have solved the problem: I installed the email blocking contrib from Dungog and used it to disallow all email from or to the sayclub.com domain.

That won't help if another mass mailing uses a different domain.

When you say that I need to make sure that the server doesn't have a direct path to the exchange server, I don't really understand how that would help correct this issue.

I must be the one not communicating well. I meant to say that if you are using the SME server to protect an Exchange server, then you must block direct access between the Internet and the Exchange server. That requires server-gateway mode. The "direct path to the Exchange server" I referred to was "from the Internet".

How is the server (in server-only mode) connected to the Internet?

[wingman]

Charlie,
Thanks for your continued help. The IP address was an external one. I did a tracert on it, and it appears to originate in Japan or Korea, couldn't tell for sure. And, you're right about it still having the vulnerability if the originating domain changes. I had thought of that, but hadn't figured out how to address it yet.

The SME server is connected to the internet through a Cyberguard firewall. The only ports forwarded to it by the firewall is port 25 and port 21 for a public (password protected) ftp folder.

Hope this helps explain my setup a little better. Funny thing is, I am running the exact same configuration at 2 other sites with no problems. The only difference seems to be that the mailblocking contrib on the problem box is newer than the contrib used on the other two systems.
Hmmmm kind makes one suspicious of that contrib.

Joe

[wingman]

policymap,

Did you find a solution? If not, contact me directly by email (see earlier post for address) and I'll let you know what my issue was.

[policymap]

yes, I think my problem is gone. I tryed to keep an eye on the mail queue, and then deleted the double-bounce messages (howto on sme.swerts-knudsen.dk)

this helped a lot, and havn't had trubles since  

kind regards
VJ

[judgej]

...This server is configured in server only mode...

Joe,

You say it is configured in server-only mode, and later you say that it is connected to the internet with port 25 relayed to it. If I understand correctly, what you have set up there is an open relay, and is highly likely to be exploited very quickly.

It should either be configured in server/gateway mode, with only the 'external' NIC connected to your Internet router, or should be accessible only to your local network.

-- JJ

PS I just found this mail rely tester, which seems pretty thorough at what it does. Try it out:

http://www.abuse.net/relay.html

[kruhm]

If it's server only and you have a firewall in front of your sme:

-don't forget to open up the ports for Razor2 & echo on your external firewall or SA won't work correctly.

-increase the number of concurrent emails handled to 40local & 20remote.

-make sure your mailfront is:
mailfront-0.91-8es &
e-smith-mailfront-1.5.0-13gr07

FYI -qmHandle can be managed from the command line.
qmHandle -D will delete all messages in the queue.
man qmHandle for details.

[CharlieBrady]

If it's server only:

-don't forget to open up the ports for Razor2 & echo or mailfront won't work correctly.

I can't think why that would be the case. Firstly, the firewall is not active in server only mode. Secondly, there is no need to open ports for outgoing queries - the netfilter connection tracking system allows return traffic automatically.