Topic: qmail limitations - workarounds?

[daytron]

Hello,

I've recently been investigating network congestion problems on a VSAT (high latency, low outgoing bandwidth (64kbps)) connected SME7.x server. The ultimate culprit is qmail. By design it unbundles message recipients and delivers each as an individual transaction - even if all the recipients are on the same domain (no multiple RCPT TO).

Amazingly, it also does this when configured to use an SMTP smarthost.

This behavior can of course very quickly saturate an outgoing link. Traffic shaping outgoing bandwidth is not an option as VSAT systems utilise shared bandwidth in a 10:1 or 20:1 configuration.

As an aside djbdns also generates (relatively) huge amounts of traffic but I'm able to bypass that by using upstream resolvers.

It seems that patching qmail to behave is prohibitively complex. Since qmail is so embedded in SME it's not easy to replace it with something more flexible such as postfix.

Has anyone encountered this problem and created a workaround? I'm considering an SMTP proxy running on a custom port to which I direct my clients - and have that proxy just forward the mail to my upstream smarthost.

[raem]

daytron

config setprop qmail ConcurrencyRemote 1
signal-event email-update

This will really slow down outgoing mail, so choose a number that suits you

[daytron]

Concurrency remote limits the concurrent connections.

The problem is that in a smarthost configuration qmail turns a 2MB email addressed to 10 people into a 20MB transaction when it shouldn't.

reducing the concurrency limit doesn't help

[raem]

daytron

Concurrency remote limits the concurrent connections.
The problem is that in a smarthost configuration qmail turns a 2MB email addressed to 10 people into a 20MB transaction when it shouldn't.
reducing the concurrency limit doesn't help

You just said
"By design it unbundles message recipients and delivers each as an individual transaction"

so won't qmail send 10 messages of 2Mb each, and if ConcurrencyRemote is set to 1, then only 1 message will be sent a a time, rather than all 10 (or more) at a time.

Have you tried changing it ?

[byte]

Its not really a "qmail limitation" as you put it - more your own limitation.  How about changing maximum message size you can send ?

[daytron]

With other MTA's, if I configure it to use  a smarthost then it passes each message off to the smarthost in a similar manner to an email client. One transaction for each message. Qmail unbundles that one message into separate messages for each recipient. N recipients, ~N times the traffic.

Naturally I have limited the maximum message size - but lets be pragmatic, normal users simply break one message into several to keep each one under the limit.

Yes it's part of qmail's design and in the days before email attachments the overhead was limited. The issue is that in a low bandwidth environment it performs poorly. I'm not interested in defense of qmail (there are plenty of qmail fanboy sites out there devoted to that), I'm interested in any solutions to work around this problem.

No qmail configuration tweaks will help what is a fundamental design problem. I'm just interested if anyone has worked around it with qpsmtpd modifications or adjacent smtpproxy installations - or something else I haven't thought of.

[byte]

Yes it's part of qmail's design and in the days before email attachments the overhead was limited. The issue is that in a low bandwidth environment it performs poorly. I'm not interested in defense of qmail (there are plenty of qmail fanboy sites out there devoted to that), I'm interested in any solutions to work around this problem.

No qmail configuration tweaks will help what is a fundamental design problem. I'm just interested if anyone has worked around it with qpsmtpd modifications or adjacent smtpproxy installations - or something else I haven't thought of.

Then your question will be better off asked in a qmail/qpsmtpd forum/mailing list.

[CharlieBrady]

I'm considering an SMTP proxy running on a custom port to which I direct my clients - and have that proxy just forward the mail to my upstream smarthost.

Yes, you could do that (probably via a custom smtp forward plugin in qpsmtpd) but you'd need to be aware that you then lose any mail queuing if your  link goes down. Or rather, the queuing would be pushed back into your user's mail clients and their Unsent folders.

Note also that if you do this, you'd also be sending all your internal email via your ISP link. You can't forward SMTP to two places at once, so if you are going to SMTP inject via your ISP, that's where your internal email will go, and they will then send it back. You'd better make sure that you have differing qmstpd configurations for internal and external connections to your server.

Or, if you are really clever with qpsmtp plugins, you could write two queue plugins, one to smtp forward the message to all external recipients, and one to qmail inject to all internal recipients.

Have fun!