Koozali.org: home of the SME Server

Legacy Forums => Experienced User Forum => Topic started by: wingman on October 18, 2005, 05:39:34 PM

Title: Email Server Overwhelmed...please help
Post by: wingman on October 18, 2005, 05:39:34 PM
Running SME 6.0.1 with Spamassassin, ClamAV, and mailfront mailblocking. This server is configured in server only mode, it's only job is to filter email and pass it on to an internal Exchange server for delivery.
Problem: twice within the past 2 months this server's remote mail queue has gotten clogged with tens of thousands of outgoing emails addressed to the sayclub.com domain. There are so many there (over 20000) that SME's email server comes to a screeching halt. If I pull SME out of the network and send all incoming mail directly to the exchange server, everything works again. Plus, the exchange server's outgoing logs show no traffic going to the sayclub.com domain. So it appears that this traffic is either originating with the SME server or is somehow being relayed through it.

The last time this happened I formatted the drives and reinstalled SME totally. It has now been running for only 2 weeks and again has this problem. I am looking for a way to find (probably from the command line) all messages in the remote queues that are addressed to the sayclub.com domain and then delete them. Does anyone know how to do this?

Also, does anyone have a clue as to what is going on with this server? I could give someone remote access to it if you want to poke around. And, if you can spot the problem and get it fixed I will gladly pay for your expertise.

Joe
Title: Email Server Overwhelmed...please help
Post by: jfarschman on October 18, 2005, 08:23:47 PM
Joe,

  Are you using SMTP Authentication?  If you are it's really difficult for a spammer to inject email through your SMTP server.  If you are not... then I'd consider turning it on.  Most email clients can be configured to authenticate before sending messages through your server.

  So... if they aren't coming in on the SMTP ports, then how are they getting in.  Some spammers are actually taking advantage of Apache vulnerabilities and then placing .php files on the server.  While I'm not sure this has happened, or is even possible on the SME servers, I have seen it twice before.  The little .php file is then used to inject emails into the system.  This would be fairly easy to find.  You'd just look in the ibays for .php files that you didn't put there.

  I hope this helps?
Title: Email Server Overwhelmed...please help
Post by: wingman on October 18, 2005, 09:02:56 PM
Jay,
Thanks so much for your reply. I searched the entire server (using MC) for php files. Didn't find any in any of the ibays. In fact, most that I found were in the httpd folder or one of it's subfolders. Didn't see anything suspicious.

I don't really understand what you mean by having users authenticate. Again, all this server is doing is scanning email for viruses and spam and then sending it on through to an Exchange server. Users don't really attach to the SME box at all. Or am I misunderstanding what you are referring to?

Thanks again, I appreciate your help.

Joe
Title: Email Server Overwhelmed...please help
Post by: jfarschman on October 18, 2005, 09:21:10 PM
Hmmm,

  So it just accepts email for your domain(s) and forwards all of it to the Exchange Server.  I'm having a hard time seeing where the problem could be if no one is  using the smtp service from the outside.

  I'm not sure I see what it could be.  I could probably spend a few minutes in the server and find the problem.
Title: Email Server Overwhelmed...please help
Post by: wingman on October 18, 2005, 09:46:07 PM
Jay,

That would be great. Please send an email to me at (wingriders at gmail dot com) and let me know how you want to get in...through ssh or remote server manager?

Joe
Title: Email Server Overwhelmed...please help
Post by: policymap on October 23, 2005, 05:05:12 PM
wingman did you find a solution?

I've had the exact same setup, and also experienced the same problem.

VJ
Title: Re: Email Server Overwhelmed...please help
Post by: CharlieBrady on October 23, 2005, 06:30:59 PM
Quote from: "wingman"
Running SME 6.0.1 with Spamassassin, ClamAV, and mailfront mailblocking. This server is configured in server only mode, it's only job is to filter email and pass it on to an internal Exchange server for delivery.


Then you should have it in server-gateway mode. That will ensure that there is no direct path to the Exchange server.

Quote

Problem: twice within the past 2 months this server's remote mail queue has gotten clogged with tens of thousands of outgoing emails addressed to the sayclub.com domain. There are so many there (over 20000) that SME's email server comes to a screeching halt.


You need to make sure that those messages don't get into the queue.

Quote

 If I pull SME out of the network and send all incoming mail directly to the exchange server, everything works again. Plus, the exchange server's outgoing logs show no traffic going to the sayclub.com domain. So it appears that this traffic is either originating with the SME server or is somehow being relayed through it.


The log files will tell you exactly where those messages came from.

Quote

The last time this happened I formatted the drives and reinstalled SME totally.


Well, there go those log files.

Quote

It has now been running for only 2 weeks and again has this problem. I am looking for a way to find (probably from the command line) all messages in the remote queues that are addressed to the sayclub.com domain and then delete them. Does anyone know how to do this?


There's no point in doing that until you do something to stop it from happening again. You already have sound evidence that it *will* happen again if you do nothing.

Search here for qmHandle and you'll find hints on how to clear those messages.
Title: Email Server Overwhelmed...please help
Post by: wingman on October 23, 2005, 06:56:43 PM
Charlie,

I did find and install qmhandle. It is insufficient for deleting 20000+ messages in the queue since it's only option (that I could find) is to open each email individually and then delete it. That would take forever. By analyzing the current logs (no I did NOT delete those...) I did find where they were coming from. All from the same IP address.

What I did that seems to have solved the problem: I installed the email blocking contrib from Dungog and used it to disallow all email from or to the sayclub.com domain. Then used MC to mass delete all emails in the queue....at this point in time any good emails are so old that they are no longer relevant.

It seems to be working now.

Charlie, evidently I am not a very good communicator. When you say that I need to make sure that the server doesn't have a direct path to the exchange server, I don't really understand how that would help correct this issue. Maybe I'm just dense, but the messages are never getting there anyway since they cause the SME box to stop sending the messages through to Exchange. And, since exchange seems not to even GET these messages if the SME server is taken off of the network and emails are received directly by exchange, I don't see how this would be a problem anyway. Can you enlighten me?

Thanks
Joe
Title: Email Server Overwhelmed...please help
Post by: CharlieBrady on October 23, 2005, 10:24:43 PM
Quote from: "wingman"

I did find and install qmhandle. It is insufficient for deleting 20000+ messages in the queue since it's only option (that I could find) is to open each email individually and then delete it. That would take forever. By analyzing the current logs (no I did NOT delete those...) I did find where they were coming from. All from the same IP address.


Was that IP address on the local network? If not, then you need to work out why they were accepted for relaying. Something about your configuration was wrong - perhaps a (very dangerous) bug in one of the contribs.

If they did come from a local network IP address, you need to work out why that IP was generating (or relaying) all those emails.

Quote

What I did that seems to have solved the problem: I installed the email blocking contrib from Dungog and used it to disallow all email from or to the sayclub.com domain.


That won't help if another mass mailing uses a different domain.

Quote

When you say that I need to make sure that the server doesn't have a direct path to the exchange server, I don't really understand how that would help correct this issue.


I must be the one not communicating well. I meant to say that if you are using the SME server to protect an Exchange server, then you must block direct access between the Internet and the Exchange server. That requires server-gateway mode. The "direct path to the Exchange server" I referred to was "from the Internet".

How is the server (in server-only mode) connected to the Internet?
Title: Email Server Overwhelmed...please help
Post by: wingman on October 24, 2005, 03:47:04 AM
Charlie,
Thanks for your continued help. The IP address was an external one. I did a tracert on it, and it appears to originate in Japan or Korea, couldn't tell for sure. And, you're right about it still having the vulnerability if the originating domain changes. I had thought of that, but hadn't figured out how to address it yet.

The SME server is connected to the internet through a Cyberguard firewall. The only ports forwarded to it by the firewall is port 25 and port 21 for a public (password protected) ftp folder.

Hope this helps explain my setup a little better. Funny thing is, I am running the exact same configuration at 2 other sites with no problems. The only difference seems to be that the mailblocking contrib on the problem box is newer than the contrib used on the other two systems.
Hmmmm kind makes one suspicious of that contrib.

Joe
Title: Email Server Overwhelmed...please help
Post by: wingman on October 26, 2005, 07:43:16 PM
policymap,

Did you find a solution? If not, contact me directly by email (see earlier post for address) and I'll let you know what my issue was.
Title: Email Server Overwhelmed...please help
Post by: policymap on October 28, 2005, 02:35:30 PM
yes, I think my problem is gone. I tryed to keep an eye on the mail queue, and then deleted the double-bounce messages (howto on sme.swerts-knudsen.dk)

this helped a lot, and havn't had trubles since  :-)

kind regards
VJ
Title: Re: Email Server Overwhelmed...please help
Post by: judgej on October 29, 2005, 03:12:14 AM
Quote from: "wingman"
...This server is configured in server only mode...


Joe,

You say it is configured in server-only mode, and later you say that it is connected to the internet with port 25 relayed to it. If I understand correctly, what you have set up there is an open relay, and is highly likely to be exploited very quickly.

It should either be configured in server/gateway mode, with only the 'external' NIC connected to your Internet router, or should be accessible only to your local network.

-- JJ

PS I just found this mail rely tester, which seems pretty thorough at what it does. Try it out:

http://www.abuse.net/relay.html
Title: Email Server Overwhelmed...please help
Post by: kruhm on November 06, 2005, 06:23:25 PM
If it's server only and you have a firewall in front of your sme:

-don't forget to open up the ports for Razor2 & echo on your external firewall or SA won't work correctly.

-increase the number of concurrent emails handled to 40local & 20remote.

-make sure your mailfront is:
mailfront-0.91-8es &
e-smith-mailfront-1.5.0-13gr07

FYI -qmHandle can be managed from the command line.
qmHandle -D will delete all messages in the queue.
man qmHandle for details.
Title: Email Server Overwhelmed...please help
Post by: CharlieBrady on November 22, 2005, 01:29:13 AM
Quote from: "kruhm"
If it's server only:

-don't forget to open up the ports for Razor2 & echo or mailfront won't work correctly.


I can't think why that would be the case. Firstly, the firewall is not active in server only mode. Secondly, there is no need to open ports for outgoing queries - the netfilter connection tracking system allows return traffic automatically.
Title: Email Server Overwhelmed...please help
Post by: kruhm on November 22, 2005, 01:11:18 PM
wingman stated that the sme is in server-only mode. so obviously there's some type of firewall in front of it. we have no idea of the type or caliber of the firewall and we don't know if it's causing issues or blocking outgoing ports.

it was a suggestion to double-check everything.
Title: Email Server Overwhelmed...please help
Post by: CharlieBrady on November 22, 2005, 11:56:06 PM
Quote from: "kruhm"
wingman stated that the sme is in server-only mode. so obviously there's some type of firewall in front of it. we have no idea of the type or caliber of the firewall and we don't know if it's causing issues or blocking outgoing ports.


You didn't suggest anywhere that he should be opening ports on an *external* firewall.

If you search this board, you will find that many, many people have been wasting time trying to open ports on the SME server unnecessarily.
Title: Email Server Overwhelmed...please help
Post by: kruhm on November 23, 2005, 12:42:50 AM
i can see your concern

alone the comments may cause confusion. in the context of the post, they make sense.

i edited the comments for documentation purposes.

-sme server-only was stated
-assumed external firewall

regards