Topic: E-mail return to sender option is misnamed (bug 2291)

[mej]

Scenario: New install of SME 7 on HP/Compaq server (was previously SME 6.00). Patched up to date. Data migrated from old system, all contribs removed in process.

This company deals with China and the far east extensively, has been doing so for 15 years, and so has (predictably) a large body of spam arriving, and a huge quantity of forged undeliverable plus mail to random users (a few thousand a day when at its worst).

Under 6.00 I used the contrib from Dungog to block mail to unknown recipients. Together with double bounce deletion that cleared things up - or kept it manageable.

SME 7 is supposed to block email to unknown recipients by default. However this does not seem to be working - see test script following (sanitised) - this is repeatable:

Code: [Select]

01/11/07 10:26:45 SMTP Verify tripe@client.co.uk, at mail.client.co.uk
Contacting 217.154.x.x
220 targetserver1.client.co.uk ESMTP

HELO example.com
250 client.co.uk Hi mail.testingbox.co.uk [213.254.x.x]; I am so happy to meet you.

VRFY tripe@client.co.uk
252 Just try sending a mail and we'll see how it turns out ...

EXPN tripe@client.co.uk
500 Unrecognized command

Doesn't want to talk to us
RSET
250 OK

MAIL FROM:<spade@example.com>
250 <spade@example.com>, sender OK - how exciting to get mail from you!

RCPT TO:<tripe@client.co.uk>
250 <tripe@client.co.uk>, recipient ok

RCPT TO:<bogus11205@client.co.uk>
250 <bogus11205@client.co.uk>, recipient ok

RSET
250 OK

QUIT
221 client.co.uk closing connection. Have a wonderful day.

This is troubling and the server responses do not look at all like the examples shown here by others.

First, where did all the wibble ('Nice to meet you' etc) come from? Something has turned it on - can I turn it off again?

But this is not important - what is important is the acceptance of email to tripe@ and bogus-n-@ (which I assure you do not exist).

So - have I missed a switch somewhere in the interface? (duh moment?). Can't see one.

And why the wibble? Have I done something else stupid? Any suggestions welcome...

JR

[gordonr]

SME 7 is supposed to block email to unknown recipients by default.

It does.

And why the wibble? Have I done something else stupid? Any suggestions welcome...

All issues should be reported in the bug tracker, and only there. Thanks.

And when you do, please do not obfuscate logs or domain names. Doing so makes it nearly impossible for us to actually diagnose the issue. We need to be able to do things like look at the DNS entries and we can't do that if we don't know which domain is being talked about.

Please note that Bugzilla now allows you to specify that a bug should be limited to the 'security team' (it's a checkbox when you report the bug). That can be useful in these cases. However, we remove that restriction once patches are made available (if required).

If your request is more sensitive, please contact security at lists dot contribs dot org as this is only read by the security team.

[dmay]

Make sure you look at the logs in /var/log/qpsmtpd.

Darrell

[mej]

mej wrote:

SME 7 is supposed to block email to unknown recipients by default.

gordonr:
It does.

mej wrote:

And why the wibble? Have I done something else stupid? Any suggestions welcome...

gordonr:
All issues should be reported in the bug tracker, and only there. Thanks.

And when you do, please do not obfuscate logs or domain names. Doing so makes it nearly impossible for us to actually diagnose the issue.

GordonR, thanks very much for all the work you put in on the SME Server.

That said, no thanks at all for that extremely unhelpful response.

I have not posted it as a bug because I am asking *IS* this a bug? Or is it me? Until I know which - I don't know if it is a (putative) bug. So I am posting this transcript to find out if it is a recognised consequence of mis-configuration on my part. Perhaps I did not make that sufficiently clear - I hope I have now.

You have not addressed any of my questions. That log is a genuine log taken straight from the SMTP conversation on the machine as stated. Any comments? Can I have misconfigured so as to cause this?

Now as to data obfuscation: this is a public board. In my country we have a thing called the Data Protection Act. I will NOT post details that will identify a client to a public board in possible contravention of the Data Protection Act, and I feel no need to apologise for not doing so, as a Registered Data Controller.

*If* this is identified as a (putative) bug then I will post it to the bug tracker unobfuscated for diagnosis - but that point has not yet been reached.

Some help to find out if this *is* a putative bug would be useful - thank you Darrell, I will check that log tomorrow.

JR[/quote]

[slords]

That said, no thanks at all for that extremely unhelpful response.

I have not posted it as a bug because I am asking *IS* this a bug? Or is it me? Until I know which - I don't know if it is a (putative) bug. So I am posting this transcript to find out if it is a recognised consequence of mis-configuration on my part. Perhaps I did not make that sufficiently clear - I hope I have now.

You can't expect a helpful answer unless you post in the right place.  You (everyone) have been told that ANY issue with the system should be posted to the bugtracker.  We don't mind closing bugs.  If it isn't a bug then we will tell you so.  But asking here isn't going to get the attention of the people that can fix it if it is a real bug.

I hope I'm making myself (and the other developers) clear that if you are having problems first search the bugtracker and if you don't find a bug describing your issue then post one.  Can I be any clearer??

[mej]

Can I be any clearer??

Yes. You could answer my question - Is this something I could have caused through mis-configuration?

I suppose from all this that the answer is "No" and that is why you are suggesting I post it to the bug tracker, so I will do that.

Why can't I get a straight answer to a simple question? Because it's Linux

J

[bpivk]

Ok here is the answer from linux/windows user.....
NO... Please post your problem to the bugtracker. Would this be ok?
You're asking for help and thay say to post your problem to the bugtracker.

[gordonr]

GordonR, thanks very much for all the work you put in on the SME Server.

That said, no thanks at all for that extremely unhelpful response.

I'm not quite sure what you want me to say. My answer was complete, and correct. I answered your question (SME Server 7 does block mail to unknown recipients). I did not go on to say that there is an option to tweak, because there isn't one. I then  told you how to report the issue, in detail, so that we could help you to solve the problem. I asked it to be reported in the bug tracker and gave you two options to maintain privacy of potentially sensitive information.

I have spent hundreds (more likely thousands) of hours on the SME Server and have now managed a large number of releases (4.0, 4.1, 5, 5.5, 5.6, 6, 7) and sub-releases. My time is valuable, and it would be polite of you to appreciate that. I also think that as the elected Development Manager I have every right to set the policy of how issues are reported.

I say in every announcement and to every post I take time to respond to 'Please report issues to the bug tracker, and only there'. I do this as it is the most efficient way to fix any issues with the SME Server. We (the dev team) ask again and again for bugs, and potential bugs, to be raised in the bug tracker. We do this for a reason.

If we answer the question here, we answer it again in the bug tracker. Or, more likely, someone's individual problem gets fixed in the forums, and that's all that happens. But the issues don't get reported in the bug tracker and so they don't get fixed for us all to benefit. Or they get reported with a reference to "see forums" without the logs or other details we need to diagnose the problem. This means the dev team has to wander through the forums to collect the information and we don't have the time or resources to do that.

The place to report issues, and potential issues, is the bug tracker. Period.

[piran]

no thanks at all for that extremely unhelpful response.

Actually mej---- it's you that is the one that's being extremely unhelpful.
The UK Data Protection Act has got absolutely NOTHING to do with it.
It's YOUR job (fine), the people here (trying) to help you do it for free.
Perhaps you'd do well to remember that in your next response.

The Developers don't know what you might of done (config/plugins/install).
They have asked you for what they need from you.

IMHO it seems probable that you're too impatient to have spotted the
obvious in the qpsmtpd/current log. Namely that a later part of the
response algorithm drops emails for which there are no recipients.
Just like what the nice (extremely knowledgeable) people said earlier.

The bit you've apparently taken amiss is likely to be a part of the process
checking that the email address vaguely 'looks' like an email ie got the
rough bits enclosed (beginning, middle and end) NOT a specific email
address existence check. As for all the 'wibble'... well there's some here
who are probably asking themselves the very same thing about you;~)

I'm not an expert and I rely on the helpful responses of all those here.
Perhaps you might consider offering them a sincere apology?

[mej]

GordonR, thanks very much for all the work you put in on the SME Server.

That said, no thanks at all for that extremely unhelpful response.

I'm not quite sure what you want me to say. My answer was complete, and correct. I answered your question (SME Server 7 does block mail to unknown recipients). I did not go on to say that there is an option to tweak, because there isn't one. I then  told you how to report the issue, in detail, so that we could help you to solve the problem. I asked it to be reported in the bug tracker and gave you two options to maintain privacy of potentially sensitive information.

I have spent hundreds (more likely thousands) of hours on the SME Server and have now managed a large number of releases (4.0, 4.1, 5, 5.5, 5.6, 6, 7) and sub-releases. My time is valuable, and it would be polite of you to appreciate that. I also think that as the elected Development Manager I have every right to set the policy of how issues are reported.

First, I do respect the work you put in and appreciate it - I thanked you for it and it was not weasel words.

Next, my experience of reporting issues to the bug tracker has been unfortunate. This is almost certainly my fault, but I have not resolved any of the ongoing problems that I have seen with the 6->7.xx upgrade by reporting them to the bug tracker. That failure did not make the problems go away. I was completely unable to upgrade from 6 to 7 using tape because tapes written with 6 (on various specific hardware that I am supporting) were not readable on 7 (on the same hardware). There is nothing wrong with the hardware - backup and restore works both before and after the upgrade. Yes, I know, report it to the bug tracker. Well I did and got nowhere - probably because I had misidentified the problem.

I resolved the upgrade data issue using the restore from disk script due, I think, to Charlie Brady (thank you), and a secondary IDE disk copy of the hardware RAID arrays. Yes I know this is not the place to report that and I am not reporting it.  Yes I know that I should have raised the new issue with the bug tracker and worked through that - but the clients needed their servers back and I had no time to do so.

However, given that prior experience I simply asked if what I was seeing could be a result of my own mis-configuration. The answer to that is a single word, "no" and I don't think it would take that long to type it

So I am sorry if I have rattled any cages, or ruffled any feathers, I *do* know how much work goes into this and how much you all get paid for it and again, I say thanks. But would it really have been so hard to just say, no, looks like you should go to the bug tracker?

I say in every announcement and to every post I take time to respond to 'Please report issues to the bug tracker, and only there'. I do this as it is the most efficient way to fix any issues with the SME Server. We (the dev team) ask again and again for bugs, and potential bugs, to be raised in the bug tracker. We do this for a reason.

If we answer the question here, we answer it again in the bug tracker. Or, more likely, someone's individual problem gets fixed in the forums, and that's all that happens. But the issues don't get reported in the bug tracker and so they don't get fixed for us all to benefit. Or they get reported with a reference to "see forums" without the logs or other details we need to diagnose the problem. This means the dev team has to wander through the forums to collect the information and we don't have the time or resources to do that.

The place to report issues, and potential issues, is the bug tracker. Period.

I accept the logic. But I was merely asking - is this a real issue? That is all. I don't understand why this is so hard to answer. The reason I started looking at the SMTP response was simply this: I was receiving up to 500 emails for unknown recipients per day in the admin mailbox. If that is 'dropping mail for unknown recipients' then its workings are a mystery to me. On 6 with the Dungog addins they were blocked at the front door.

Anyway, as you have requested and made clear, I will cease posting here and move to the bug tracker. With trepidation: I think it should have a warning: 'Abandon hope all ye who enter here'

With genuine great respect,

apologies if necessary,

and a polite, indeed obsequious bow,

(have I demeaned myself sufficiently yet?)



James Roberts

[piran]

Noted;~)
You have my sympathy.
I too find it daunting in 'there' (bugzilla) - VERY technical & unforgiving.
But that's the sharp end and its where the guts of an issue get fixed.
They'll still want your log files, so brace yourself;~)
Curious symptoms... have you been 'Joe Jobbed' do you think
ie your site been 'quoted' as the source of a spam wave?

[mej]

Noted;~)
You have my sympathy.
I too find it daunting in 'there' (bugzilla) - VERY technical & unforgiving.
But that's the sharp end and its where the guts of an issue get fixed.
They'll still want your log files, so brace yourself;~)
Curious symptoms... have you been 'Joe Jobbed' do you think
ie your site been 'quoted' as the source of a spam wave?

No its not a joejob, just a client with mostly East Asian contacts. I have had a joejob - that's worse

Thanks for the sympathy And I did not intend to offend anyone. It's been a difficult Friday, admittedly, with BT pulling out 3 of 5 T1s to our backbone providers

 

I have been using e-smith since early version 4. I know it pretty well,- for a user. I've been using Linux on and off since 1993 and qualified as a programmer/analyst in 1979 (IBM assembler:) so I am an old f**t. I claim no expertise whatever in the SME Server except as a user. I know Linux and BSD reasonably well but am still most definitely an amateur.

I have been using and supporting email on bulletin board systems since before the Internet became generally available and have had a (commercial) Internet email account since 1987.

My point in saying so is only, I have some slight experience of email and how it is supposed to work.

Large parts of the issues with e-smith/SME Server email handling are because the developers have chosen to stick with QMail. They have very good and cogent reasons for doing so (I am inclined to disagree with them - but I respect their choice), but because of its design it is an issue for it to drop mail to unknown senders, and this issue requires various workarounds (basically a front end).

I stated that I was receiving up to 1000 emails a day for unknown users to admin, and therefore checked the SMTP conversation. I specified the condition - clean install with no contribs. I asked if I had done something stupid (wouldn't be the first, or last time). This was to filter out a spurious bug report from myself. It does not seem unreasonable to me.

The bit you've apparently taken amiss is likely to be a part of the process
checking that the email address vaguely 'looks' like an email ie got the
rough bits enclosed (beginning, middle and end) NOT a specific email
address existence check. As for all the 'wibble'... well there's some here
who are probably asking themselves the very same thing about you ;~)

Quite so

I'm not an expert and I rely on the helpful responses of all those here.
Perhaps you might consider offering them a sincere apology?

I have bowed and scraped, hope that will suit. I have no pride when faced with a problem I can't solve, I will ask for help anywhere I can get and take it gratefully when offered

Now as to the UK Data Protection Act. That is my responsibility and I repeat I will not post identifying information in a public forum. We have to negotiate various legislation, including the UK DPA, RIPA, EHCR and others, and I am not going to take any risks with legislation that can see you banged up (even if it is very unlikely). I have to show due diligence and I offer no apologies for doing so

However let me demonstrate a couple of SMTP conversations (client data sanitised).

Here is one with a Barracuda in front of an e-smith:

Code: [Select]


01/12/07 17:20:50 SMTP Verify trash@example.co.uk, at mail.example.co.uk
Contacting 217.x.x.210
220 barracuda.example.co.uk ESMTP (13dd60aee979e619b83ae9387882d649)

HELO stabilys.com
250 barracuda.example.co.uk

VRFY trash@example.co.uk
502 VRFY command is disabled

Doesn't want to talk to us
RSET
250 Ok

EXPN trash@example.co.uk
502 Error: command not implemented

Doesn't want to talk to us
RSET
250 Ok

MAIL FROM:<me@stabilys.com>
250 Ok

RCPT TO:<trash@example.co.uk>
553 <trash@example.co.uk>: Recipient address rejected: Sorry, that is an invalid e-mail address

421 Error: too many errors

Doesn't want to talk to us
RSET
QUIT

The Barracuda is there because the client unwittingly bought a domain name that had been used by a spammer. We were getting tens of thousands of bounced and forged emails a day at first

*That* is rejecting mail to an unknown recipient. And note - no wibble.

Here is the dialogue for our own mail server running exim:

Code: [Select]


01/12/07 17:57:55 SMTP Verify trash@stabilys.com, at mx1.balanced.thingy.mail.example.com
Contacting 208.97.132.51
220 thingymail-mx2.example.com ESMTP

HELO stabilys.com
250 thingymail-mx2.example.com

VRFY trash@stabilys.com
502 VRFY command is disabled

Doesn't want to talk to us
RSET
250 Ok

EXPN trash@stabilys.com
502 Error: command not implemented

Doesn't want to talk to us
RSET
250 Ok

MAIL FROM:<me@stabilys.com>
250 Ok

RCPT TO:<trash@stabilys.com>
550 <trash@stabilys.com>: Recipient address rejected: User unknown in virtual alias table

Doesn't want to talk to us
RSET
250 Ok

QUIT
221 Bye


Note unknown recipient rejected. And no wibble.

Now the same thing for SME 6 without the Dungog addins:

Code: [Select]


01/12/07 17:29:03 SMTP Verify trash@client.example.com, at client.example.com
Contacting client.example.com
220 serv1.example.com mailfront ESMTP

HELO stabilys.com
250 serv1.example.com

VRFY trash@client.example.com
252 Send some mail, I'll try my best.

EXPN trash@client.example.com
500 Not implemented.

Doesn't want to talk to us
RSET
250 OK

MAIL FROM:<spade@stabilys.com>
250 Sender accepted.

RCPT TO:<trash@client.example.com>
250 Recipient accepted.

RCPT TO:<bogus22943@client.example.com>
250 Recipient accepted.

RSET
250 OK

QUIT
221 Good bye.


Note that the junk addresses are accepted. (But no wibble).

Here is the SME 7 dialog posted earlier:

Code: [Select]


01/11/07 10:26:45 SMTP Verify tripe@client.co.uk, at mail.client.co.uk
Contacting 217.154.x.x
220 targetserver1.client.co.uk ESMTP
HELO example.com
250 client.co.uk Hi mail.testingbox.co.uk [213.254.x.x]; I am so happy to meet you.
VRFY tripe@client.co.uk
252 Just try sending a mail and we'll see how it turns out ...
EXPN tripe@client.co.uk
500 Unrecognized command
Doesn't want to talk to us
RSET
250 OK
MAIL FROM:<spade@example.com>
250 <spade@example.com>, sender OK - how exciting to get mail from you!
RCPT TO:<tripe@client.co.uk>
250 <tripe@client.co.uk>, recipient ok
RCPT TO:<bogus11205@client.co.uk>
250 <bogus11205@client.co.uk>, recipient ok
RSET
250 OK
QUIT
221 client.co.uk closing connection. Have a wonderful day.

Note that the junk addresses are accepted. And loadsa wibble. I can't demonstrate the result due to the Dungog addins, as the machine is upgraded now to 7 - but it was like the first one above.

This is not rejecting email addresses for unknown recipients. It is ACCEPTING email for unknown recipients. If they are dropped later then that is something else, but it is not rejecting. This means that client bandwidth is used up - though probably not much. Rejecting it uses no bandwidth. BLOCK =/= ACCEPT and DROP.

As far as the wibble is concerned, Sendmail used to provide similar chatty responses but they do not look very professional and are only funny the first few times, so they were disabled years ago; I'm wondering how they crept back in here! I know the messages are coming from qpsmtpd - but I don't want them. I want to turn them *off*, and want to know if there is a canonical e-smith way to do so. Ialso find qpsmtpd occasionally slow - there are only 86400 secs in 24 hours and if it takes 2 secs to receive an email that limits us to (much less than) 43200 transactions a day: some of my clients may push that (because of the spam load).

For one (of thousands) of pages listing reasons for not using QMail see: http://www.pgregg.com/forums/viewtopic.php?id=24&qid=155

Anyway I think that is more than enough on this and I have moved the issue to the bug tracker. As requested

Have a nice weekend, all. And more apologies to anyone who I have inadvertently contrived to offend...

MeaJamjaR spinning quietly...  

[piran]

...awaiting the explanation of, and any resolution to,
the apparent issue you've raised - with much interest;~)

[gordonr]

Large parts of the issues with e-smith/SME Server email handling are because the developers have chosen to stick with QMail.

http://lists.contribs.org/mailman/public/devinfo/msg08093.html

However let me demonstrate a couple of SMTP conversations (client data sanitised).

This is precisely the information we need in the bug tracker if we're to resolve your issue. Why write it twice?

Let me be clear - if a standard SME7 box is allowing mail to invalid recipients, I want to know about it. It would be a serious issue which would need to be fixed.

I presume you do have "E-mail to unknown users    Return to sender" configured on the E-mail panel?

Apologies, I misspoke before, there is an option to not block mail for invalid recipients. This can be useful for testing during conversion (which is the only reason we left it in), but should normally be disabled. The default is disabled/reject.

As far as the wibble is concerned,

http://bugs.contribs.org/show_bug.cgi?id=1325

For one (of thousands) of pages listing reasons for not using QMail see: http://www.pgregg.com/forums/viewtopic.php?id=24&qid=155

So, does it matter enough to you to fund the development and testing?

http://lists.contribs.org/mailman/public/devinfo/msg09015.html
http://lists.contribs.org/mailman/public/devinfo/msg09053.html
http://lists.contribs.org/mailman/public/devinfo/msg09058.html
http://lists.contribs.org/mailman/public/devinfo/msg09039.html

Anyway I think that is more than enough on this and I have moved the issue to the bug tracker. As requested

What is the bug reference?

[mej]

However let me demonstrate a couple of SMTP conversations (client data sanitised).

This is precisely the information we need in the bug tracker if we're to resolve your issue. Why write it twice?

Because these were examples on other systems illustrating the rejection of email for piran. They would not constitute a valid bug report.

Let me be clear - if a standard SME7 box is allowing mail to invalid recipients, I want to know about it. It would be a serious issue which would need to be fixed.

I presume you do have "E-mail to unknown users    Return to sender" configured on the E-mail panel?

I do not. I see no point whatsoever in returning mail to someone who did not send it (1). I do not want to be part of an NDN attack (2). Nor do I want to accept badly-addressed mail in the first place! I thought SME7 was supposed to reject it by default? There is no option for this in the long list of accounts it can be delivered to. Are we saying that email for unknown recipients is still sent to admin (if that is selected) and not rejected then? "Return to sender" is not = rejected in my book.


This is what my original query was concerned with. (!!)

I will check further to see if anything else is wrong with the set up before I post a bug.

Apologies, I misspoke before, there is an option to not block mail for invalid recipients. This can be useful for testing during conversion (which is the only reason we left it in), but should normally be disabled. The default is disabled/reject.

Not a problem. I have been known to mispeak myself

For one (of thousands) of pages listing reasons for not using QMail see: http://www.pgregg.com/forums/viewtopic.php?id=24&qid=155

So, does it matter enough to you to fund the development and testing?

http://lists.contribs.org/mailman/public/devinfo/msg09015.html
http://lists.contribs.org/mailman/public/devinfo/msg09053.html
http://lists.contribs.org/mailman/public/devinfo/msg09058.html
http://lists.contribs.org/mailman/public/devinfo/msg09039.html

Perhaps (at least in part!). As I stated, I understand why you (all) have not changed. In Debian I just have to do an apt-get to change mail server. Because of the intricate clockwork (alright templates) holding e-smith together, it is no trivial task to change a component (which is both its merit and a pain in the a***), and qmail is pretty solid as a mailserver, it's the one big merit of its design. You may prefer sponsorship of other, more pressing, missing features. And so might I. Ideally, though, a mail server that simplified dealing with today's problems could be a good move.

Oh. You did not pickup on the speed issue. Sometimes the response is so slow that the next mail goes to the backup MX - which spoils the whole idea and lets anything in. This was less of a prob with 6.

What is the bug reference?

I was hoping for a day off (having worked right through Xmas and last weeked- aah)! But will post this tomorrow. I have just had a half bottle of wine and now is not a good moment

It's very late. G'Night.

(1) http://www.city-fan.org/mailwasher-bounce-bad.html (and many others)
(2) http://www.techzoom.net/paper-mailbomb-7.asp