Topic: Double-buffered SMTP server?

[rickjones]

I'll explain the problem. The sheer volume of mal-email hitting my server is beginning to look like a DOS attack. The spam comes in waves, but when it does it ties up smtpfront as each incoming is scanned (I do both virus and spam scanning at this point via Amavis). This clearly causes timeouts to further incoming, which then diverts via my backup channel (my hosting provider).

This causes further problems, as it has to wait until fetchmail picks it up, and multidrop fetch is not ideal, with some inevitable mis-delivery etc.

What I'd like is for incoming messages to be dropped direct to disk, forming a queue which is then picked up, scanned, and delivered at whatever speed the system can cope with. It's not a powerful server, but it doesn't need to be, and I'm damned if I'm going to buy a bigger server just so I can throwaway spam faster!

Any ideas on how to configure this? Never mind templates for the moment, what would the core elements of the setup have to be?

TIA for any ideas.

[smeghead]

G'Day Rick

I have had several clients hammered by spam/virus attacks and found the best solution was to install the mailfront contrib to kill off most viruses before they get into the server (as virus scanning is the most intensive process).

I also up the RAM in the server to give it a few more resources to cope with the load.

Lastly I modify SMTP to process fewer concurrent sessions so the server load is reduced:

/sbin/e-smith/db configuration setprop qmail ConcurrencyRemote 10
/sbin/e-smith/db configuration setprop qmail ConcurrencyLocal 10
/sbin/e-smith/signal-event email-update
/etc/init.d/qmail restart

HTH

[CharlieBrady]

This clearly causes timeouts to further incoming, which then diverts via my backup channel (my hosting provider).

This causes further problems, as it has to wait until fetchmail picks it up, and multidrop fetch is not ideal, with some inevitable mis-delivery etc.

You've got a very strange MX backup arrangement if you are collecting via fetchmail. Why aren't they delivering via SMTP (as is standard behaviour for MX backup hosts)?

As someone else has said, dropping your SMTP concurrency setting will reduce your server load. But it's not the qmail setting you need to tweak (that's for local and remote deliveries *from* qmail, not into qmail) - you need to change the number of concurrent smtpfront-qmail processes. That's not currently controlled by a database setting, but you can tweak it by creating a custom template fragment

/etc/e-smith/template-custom\
/var/service/smtpfront-qmail/runenv/Concurrency

containing:

CONCURRENCYREMOTE=4

then do:

/sbin/e-smith/signal-event email-update

[raem]

Oh Charlie, Thank you.

I think I have a similar problem ie too much email slows down the server.

Do you mean:

mkdir -p /etc/e-smith/templates-custom/var/service/smtpfront-qmail/runenv

cd /etc/e-smith/templates-custom/var/service/smtpfront-qmail/runenv

pico Concurrency

add following details

CONCURRENCYREMOTE=4

Alt C to save, alt X to exit

/sbin/e-smith/signal-event email-update


Can you explain this a little more please:

"But it's not the qmail setting you need to tweak (that's for local and remote deliveries *from* qmail, not into qmail)"
particularly the "into qmail" part.

Thanks

[hordeusr]

Using ASSP on my server fixed this problem.  It rejects SPAM at the SMTP level and blocks all executable attachements.  Anti-Spam-STMP-Proxy, so obviously it's a proxy to the real SMTP server(in my case it's on the same server)  The real SMTP server doesn't see the SPAM.  This made a huge improvement in server load for me.  ASSP does take a few days to "learn" (depending on how many e-mail you get), and some effort on your part in the initial learning.  Once it's going it works great.

[rickjones]

This clearly causes timeouts to further incoming, which then diverts via my backup channel (my hosting provider).

This causes further problems, as it has to wait until fetchmail picks it up, and multidrop fetch is not ideal, with some inevitable mis-delivery etc.

You've got a very strange MX backup arrangement if you are collecting via fetchmail. Why aren't they delivering via SMTP (as is standard behaviour for MX backup hosts)?

Because it's cheap! I'm just using the standard mailbox facility of my Web hosting provider - true backup with SMTP forwarding would be an extra service with a cost, I haven't considered that necessary in relation to the amount of genuine email that comes in. I'm now getting something like 100 mal-mail messages for each genuine one, and it's a real pain. I hate the idea of being forced by Internet scum to spend more on hardware and services.

I'm not convinced that changing concurrency will help . Spam attacks seem to come in waves, and the server has to scan each message on arrival. If, say, 50 messages arrive in one minute, then the server has to scan 50 messages, and it will take about the same total time whether it does them serial of parallel. If the total scanning time is 5 mins, it's not going to keep up however the concurrency is configured. It's only going to have time to accept one message in 5.

This is why I was thinking of a caching system. Incoming messages could just be dropped to disk while a separate service picks them up, scans and locally delivers. Incoming SMTP would never be blocked, while the scan & deliver service could catch up in its own time. I was wondering about a second SMTP service on a different port, which would handle the internal scan and LDA.

It's a bit like the ASSP solution that's been mentioned, but I want to scan at the second level, and have the two run asynchronously.

Any thoughts?

[KelvinLee]

Hi Rick,

You have not specified you SME version. With SME 5.6 you could add a contrib from Charlie (I forget the name) which adds RBL lookups to incoming SMTP mails. This is built into SME 6.0.1-1 and just needs to be enabled.

While RBL lookups is not perfect, it does drop a large percentage of the spam mails before the mails are accepted for processing by qmail.

Kelvin

[rickjones]

Hi Kelvin

You have not specified you SME version. With SME 5.6 you could add a contrib from Charlie (I forget the name) which adds RBL lookups to incoming SMTP mails. This is built into SME 6.0.1-1 and just needs to be enabled.

Thanks for the tip, I'm running 6.0.1, and I hadn't realised that feature was available! I've turned it on, I'll try it with conservative settings and see how it goes. Hopefully it will have enough effect to avoid the problem.

Cheers