Topic: E-mail return to sender option is misnamed (bug 2291)

[piran]

Good to see dialogue, I'm sure the community will benefit;~)

gordonr---- I read your response to mej's 'wibble' concern...
http://bugs.contribs.org/show_bug.cgi?id=1325
...but it's not fully appropriate.
FWIW I too have been uncomfortable with all such rubbish.
Is there a more straightforward and laconic mode of response?
My site shows...
220 myservername.mydomain
...but the wibble is the inane...
250 <admin@contribs.org>, sender OK - how exciting to get mail from you!
...stuff. I looked through the remaining files in...
/var/service/qpsmtpd/config
...but none contained the ' -how exciting to get mail from you!' drivel.
Is there somewhere else I must look to cull the inappropriateness?

mej---- Just a small point... my site finds the response lag quite useful.
In SME6 days the RBL lookup frequently beat the latter's ability to post
new culprits, my log post mortems clearly demonstrated this slightly ironic
situation. With a brace of dozen or so RBLs slowing things down further
I find that SME7's spam control procedures extremely effective overall.
Yes, I grant that my site doesn't bear the sheer load that you quoted.

FWIW the MX was a spam nightmare, I've switched it off. 100% of any
delivered spam came from that route. Indeed most of the log post
mortems showed that the secondary MX was deliberately used.

[gordonr]

I do not. I see no point whatsoever in returning mail to someone who did not send it (1). I do not want to be part of an NDN attack (2). Nor do I want to accept badly-addressed mail in the first place! I thought SME7 was supposed to reject it by default?

It does and this explains the behaviour on your system. The wording on the panel is now inaccurate with the changes in SME7 (which has previously been raised, but has not yet been corrected). It should say "rejected" rather than "returned". And I have ensured that a bug is raised for that wording change:

http://bugs.contribs.org/show_bug.cgi?id=2291

Mail for unknown users in SME7 is rejected by default during the SMTP transaction. SME6 returned the mail (qmail backscatter). This change was a specific design goal of SME7 and works as advertised and is in line with the recommendations in your links. It was one of my primary goals for the SME7 mail system changes.

However, you have asked it to not reject mail for unknown users, so you get everything addressed to local domains. That mail defaults to delivery to admin, but you could, for example, send it to a 'deadletter' user which you can check from time to time (if you like wading through spam for misdirected mail).

If you ask for recipient filtering, you get it (and that is the default). If you want everything for a domain, you can have that too (the option you have chosen).

Some people find this useful, particularly in new installs. Most people switch back to the dedault of full recipient filtering/rejection once they've checked that all looks o.k.

In Debian I just have to do an apt-get to change mail server.

And then write the configuration. Installing the new MTA is the easy part. The hard part is the configuration changes which need to be made to support all of the features of the SME Server (e.g. pseudonyms, group aliases, mailing list support addons, etc). We do a lot of qmail configuration, and would have to do a similar amount of configuration for any other MTA. Templates actually ease that work, but it still has to be done, and has to be tested.

As I stated in the threads I linked to from devinfo, most of the work has been done for Postfix support, but there are some difficult upgrade issues and a lot of testing time required. That "last 10%" is the expensive part of most development tasks.

My estimate is that there is at least two weeks fulltime for a senior developer to complete the conversion and deal with the upgrade issues. That's not a trivial amount of work, which is why we haven't done it. And I expect a number of contribs to break after the conversion.

Oh. You did not pickup on the speed issue. Sometimes the response is so slow that the next mail goes to the backup MX - which spoils the whole idea and lets anything in. This was less of a prob with 6.

Then raise it in the bug tracker so we can deal with it.

Similarly with the response messages - raise them in the bug tracker so that we can deal with them.

[gordonr]

Is there somewhere else I must look to cull the inappropriateness?

No, unfortunately. The strings come from the qpsmtpd code and should be configurable. I put in patches for this some time back, but they have not been applied.

Raise it in the bug tracker, please.

[gordonr]

I've taken the liberty of renaming the thread as there's a lot of information in here, but the fundamental issue is captured in this bug report:

http://bugs.contribs.org/show_bug.cgi?id=2291

[mej]

I do not. I see no point whatsoever in returning mail to someone who did not send it (1). I do not want to be part of an NDN attack (2). Nor do I want to accept badly-addressed mail in the first place! I thought SME7 was supposed to reject it by default?

It does and this explains the behaviour on your system. The wording on the panel is now inaccurate with the changes in SME7 (which has previously been raised, but has not yet been corrected). It should say "rejected" rather than "returned". And I have ensured that a bug is raised for that wording change:

http://bugs.contribs.org/show_bug.cgi?id=2291

OK, you have convinced me. I will raise any future points that may be even slightly doubtful in the Bug Tracker.

But just as you asked for courtesy, I'd ask that it be applied there too. I feel that as soon as I raise a bug, there is an implied confrontational situation (= "you got to be an idjit"). That is why I avoid the bug tracker and have done ever since my first experiences with previous versions of it.

I do understand that most issues are in fact PEBCAK/EBSAC (1)- I have some hundreds of users at different companies to remind me of this - and this is why I raised the issue here instead. Because I was correct in my supposition - I had misconfigured.  As far as:

> omg, it took forever to find his problem
> i'll draft a patch

is concerned - this could have been established immediately after my first message.

But there wasn't a bug in the system software - it was an inconsistency in the interface - it lied to me about what the effect of a setting was.

Incidentally: should previously-raised unresolved interface issues not be added to the release notes?

FWIW, I immediately changed the settings and another test shows it operating as stated:

Code: [Select]


01/13/07 12:35:37 SMTP Verify trash@client.co.uk, at mail.client.co.uk
Contacting 217.154.x.x
220 mailserver1.client.co.uk ESMTP

HELO stabilys.com
250 client.co.uk Hi 85-210-x-x.dsl.x.com [85.210.x.x]; I am so happy to meet you.

VRFY trash@client.co.uk
252 Just try sending a mail and we'll see how it turns out ...

EXPN trash@client.co.uk
500 Unrecognized command

Doesn't want to talk to us
RSET
250 OK

MAIL FROM:<me@stabilys.com>
250 <me@stabilys.com>, sender OK - how exciting to get mail from you!

RCPT TO:<trash@client.co.uk>
550 invalid recipient trash@client.co.uk

Doesn't want to talk to us
RSET
250 OK

QUIT
221 client.co.uk closing connection. Have a wonderful day.



Thank you. Just the wibble to go. YES I will raise a bug.

In Debian I just have to do an apt-get to change mail server.

And then write the configuration. Installing the new MTA is the easy part. The hard part is the configuration changes which need to be made to support all of the features of the SME Server (e.g. pseudonyms, group aliases, mailing list support addons, etc). We do a lot of qmail configuration, and would have to do a similar amount of configuration for any other MTA. Templates actually ease that work, but it still has to be done, and has to be tested.

As I stated in the threads I linked to from devinfo, most of the work has been done for Postfix support, but there are some difficult upgrade issues and a lot of testing time required. That "last 10%" is the expensive part of most development tasks.

My estimate is that there is at least two weeks fulltime for a senior developer to complete the conversion and deal with the upgrade issues. That's not a trivial amount of work, which is why we haven't done it. And I expect a number of contribs to break after the conversion.

I know all this (* though note Debian does much of the re-configuration too). There has been loadsa discussion elsewhere about what priorities to adopt in SME Server. Personally I'd prefer to have it based on Debian Not holding my breath.

I don't know what the most pressing issues are for changes to the system.

What has troubled me most has been difficulties in migration which YES I will raise again as a new bug.

If you want sponsorship for a change to Postfix then some could be arranged. I'll PM you (or you me).

JR

(1) http://www.johnrostron.co.uk/fun/nonsense/pebcak.htm

[piran]

Is there somewhere else I must look to cull the inappropriateness?

Raise it in the bug tracker, please.

http://bugs.contribs.org/show_bug.cgi?id=2293

[mej]

FWIW the MX was a spam nightmare, I've switched it off. 100% of any
delivered spam came from that route. Indeed most of the log post
mortems showed that the secondary MX was deliberately used.

Yeah too right

However it is not my site I am talking about. Clients are IME prepared to accept loadsa spam rather than lose an email. When an email can represent 50K to them I can understand this

Since they are generally on ADSL/SDSL sometimes it goes down or has maximum contention so the backup MX is needed.

I'm looking into other ways of solving the problem optimally (probably involving us sticking a box in a rack somewhere to host the backup MX with filtering ourselves...)

MeJR

[piran]

What a busy chap, would that any of my emails came carrying 50K;~)
FWIW have filed a New Feature Request against the inane text strings
(aka wibble) but it looks highly probable we're all stuck with the stuff.

[mej]

would that any of my emails came carrying 50K;~)

Me2

Many of our clients are solicitors (lawyers). + We have a major corporate headhunter. Sadly our fees are at a different level...  

There is a *lot* of competition around.

80% of our clients are on Windows Server 2003 and more and more on Citrix (with yet another Windows 2003 server) but I still have 4 on SME-Server. Trying to keep it alive...

MeJ

[CharlieBrady]

Since they are generally on ADSL/SDSL sometimes it goes down or has maximum contention so the backup MX is needed.

No, it is not needed. If there is no backup MX, and the server or connection is down, the sending mail server will keep the mail and retry later.

[mej]

Since they are generally on ADSL/SDSL sometimes it goes down or has maximum contention so the backup MX is needed.

No, it is not needed. If there is no backup MX, and the server or connection is down, the sending mail server will keep the mail and retry later.

I absolutely agree that this is RFC and how it is supposed to be.

Unfortunately no-one has told the sysops of a large number of mailservers - which return mail as undeliverable if not delivered first attempt. Mostly they are not running Free or Open mailservers.



J

[gordonr]

Unfortunately no-one has told the sysops of a large number of mailservers - which return mail as undeliverable if not delivered first attempt. Mostly they are not running Free or Open mailservers.

I have not seen such behaviour for many years. Your experiences may vary, but I think there's a bit of net folklore here. Any mailer which doesn't handle retries isn't speaking SMTP - it's not a matter of strict compliance, it's a basic part of SMTP.

IMO the best thing to do is to force compliance in this case. If a mailer doesn't retry, that sender will lose mail and papering over the cracks doesn't help anyone in the longer term.

What I have seen is broken mailers which treat 4xx temporary error codes as 5xx permanent failure codes. A backup MX probably won't help all that much in those cases as there are plenty of reasons why you might need to return a 4xx code from time to time. Those mailers are just as broken and we (the recipient) can't fix them. The kindest thing to do is to make it plain that the sender's mailer is broken and needs to be fixed.

I understand how valuable these mails might be, but you fundamentally can't make any sort of guarantee if the client doesn't retry, or ignores temporary errors. A backup might reduce the chance of losing the mail, but probably not by much.

Gordon

[mej]

...mailservers - which return mail as undeliverable if not delivered first attempt

I have not seen such behaviour for many years. Your experiences may vary, but I think there's a bit of net folklore here. Any mailer which doesn't handle retries isn't speaking SMTP - it's not a matter of strict compliance, it's a basic part of SMTP.

IMO the best thing to do is to force compliance in this case. If a mailer doesn't retry, that sender will lose mail and papering over the cracks doesn't help anyone in the longer term.

What I have seen is broken mailers which treat 4xx temporary error codes as 5xx permanent failure codes. A backup MX probably won't help all that much in those cases as there are plenty of reasons why you might need to return a 4xx code from time to time. Those mailers are just as broken and we (the recipient) can't fix them. The kindest thing to do is to make it plain that the sender's mailer is broken and needs to be fixed.

I understand how valuable these mails might be, but you fundamentally can't make any sort of guarantee if the client doesn't retry, or ignores temporary errors. A backup might reduce the chance of losing the mail, but probably not by much.

Gordon

It's a tricky call. We cannot afford to lose email for clients. We give them the option - they prefer to have the backup. It's probably because "backup" sounds good to them.

However I have seen mail go missing in exactly this way - correction, I have received *reports* from clients whose contacts are complaining that mail is undeliverable. I've seen the complaints too, but can't read them - they're in Chinese!

It was the experience of large numbers of such reports that caused us to arrange the backup MX.

As far as our own system is concerned, I only want mail from compliant sources.

The 4xxx/5xxx issue you mention also breaks greylisting

We've tried.

Is it a coincidence that many of these broken mailers seem to be in the Far East?

JR

[gordonr]

It's a tricky call. We cannot afford to lose email for clients. We give them the option - they prefer to have the backup. It's probably because "backup" sounds good to them.

The tricky part with any backup MX is keeping the config in sync with the master (if you want to do complete recipient filtering on the backup, which I pretty much bet that you do). I've seen lots of misconfigured backup MX servers which are out of sync with the master. This means that they bounce mail precisely when you need them

An rsync of /home/e-smith/db/ (at least domains and accounts) to the backup MX, followed by "signal-event email-update" would be very close to what you want.

Note that the backup would want to be in "Delegate Mail Server" mode, queuing for the primary, or an internal mail server, so the sketch above isn't quite enough.

Also, in delegate mail server mode we do an SMTP connection to the internal mail server and return 4xx if we can't connect to it.