Topic: An old chestnut: pop3 sorting

[Jason Judge]

I've scanned the forums, but this seems to be a problem that never gets answered completely. Any help would be appreciated.

I have one pop3 mailbox at the isp (blueyonder.co.uk) and that mailbox supports several e-mail aliases (say, fred@blueyonder.co.uk and jim@blueyonder.co.uk). When I pick up mail from that mailbox it all gets sent to admin, with the usual error in the header:

X-Fetchmail-Warning: recipient address fred@blueyonder.co.uk didn't match any local name

I would like the e-mail to be sorted into the appropriate local mailboxes. The usual answer is to set up a local virtual domain for blueyonder.co.uk, and that works to an extent. It falls down if I want to send anything to a.nother@blueyonder.co.uk, as the SME box thinks it should be handling that mail.

How do I tell the SME box to sort _incoming_ mail for anyone at blueyonder.co.uk, but to pass _outgoing_ mail for anyone other than fred and jim to mail.blueyonder.co.uk?

The only way I can see at the moment is not to use aliases on the blueyonder ISP - but to give fred and jim completely separate pop3 mailboxes. Multipop can then pass everything from each of these pop3 boxes to local users. However, I'm limited on the number of mailboxes I can use.

I would love to find the definitive answer to this, as it comes up time and time again on the forums, but no-one has ever questioned not being able to send e-mail to other users of the ISP.

I'm using SME 5.1.2 - and love it!

-- Jason

[Matt Quelch]

Just out of curiosity, does Blueyonder not give you a catchall type mailbox? So that any messages sent to anything@whatever.blueyonder.co.uk all come straight in to you? That's what I have through Easyspace...

Matt

[Jason Judge]

I don't know - but I'll investigate. I did try to create some personal webspace on the ISP, but the request was refused as I didn't have an alias on the main e-mail account. I wasn't sure what the link was between the personal webspace and the e-mail alias, but it is beginning to make sense if it is a 'catchall' mailbox.

I can see where you are heading: create a local virtual domain for myarea.blueyonder.co.uk, while still leaving blueyonder.co.uk as a remote domain.

I'll report back when I find out...

Thanks,

-- Jason

[Jason Judge]

Okay, the results from a bit of research:

- The personal webspace of the blueyonder is located at www..pwp.blueyonder.co.uk, where is any of the e-mail aliases that you have set up (so long as they don't have a dot in them). That's quite a handful to type, so even if it did work as a 'catchall' mailbox then it's not an e-mail I would like to send out to friends and family.

- E-mails sent to .pwp.blueyonder.co.uk or .blueyonder.co.uk seem to go into a black hole, so it does not look like it works anyway. I'm expecting a delivery failure to be returned in the next day or so, as Blueyonder seem to hold on to their delivery failures for some time.

- Blueyonder ISP allow up to five mailboxes and three aliases for each mailbox.

The best way it seems to use these is to provide one mailbox for each user. Each user can then have up to three aliases at blueyonder.co.uk. The multipop add-in is then used to download from each mailbox in turn and deliver everything in each mailbox to a specific user on the SME box.

This has its limitations: i.e. only five users can be supported, but then these are family-based mailboxes and so I don't need too many. I've got other domains I can use that offer more control and better mail sorting.

For example, on mailbox 1, I have set up aliases 'jason.judge', 'jason_judge' and 'judgej', so that gives me three e-mail addresses: jason.judge@blueyonder.co.uk, judgej@blueyonder.co.uk etc. Mail to these three aliases are all picked up in one go using multipop and delivered to the local user 'judgej'. Similarly, several more mailboxes with their own aliases are set up with their own multipop entries.

I'm just including those details in case they are useful for others using Blueyonder or some similar broadband ISP. At the end of the day, the absolute best way to set up the e-mail exactly how you want it is to get a new domain and point it at your box using a service such as zoneedit or dyndns. Any number of these domains can be used alongside multipop on a single SME box. Now, that really works a treat.

-- Jason

[Kelvin]

Hi Jason,

This is an old long standing problem. The specific requirement differs from other people but the basic reason is the same.

Out of the box, SME MUST have either its primary or at least a virtual domain setup for mails coming in via multidrop to be sorted by SME itself. As you have seen, this works but if that domain is used by people in multiple locations, you have a problem as any mail address to anyone in that domain sent from a workstation behind SME, it will not get sent out as SME regards it as local mail.

There was apparently a contrib by Daniel Van Raay (e-smith-ofmipd or something like that) which was supposed to help address this issue but unfortunately, as there was no documentation about what it should do or what we should expect to modify when it gets installed, etc., I did not geta round to pursuing the matter when it came up some time back.

Using multipop contribs (my preferred one is Vincent Filali's) helps to a degree but can get messy and is by no means elegant.

Kelvin

[Jason Judge]

Kelvin,

I think it's always going to be a hack. It's the equivalent of giving two big cities at opposite ends of a country the same name and postcode, then asking the postal service to use telephone directories to look up the name on the front of each envelope to determine which city to send each letter to. That would be a very messy thing to have to handle, and would always be sensitive to errors: which city takes the incorrectly addressed letters?

I think it needs to be made more clear in the documentation that if you are using POP3 to retrieve e-mail, then for practical purposes either (1) the domain has to be one that you do not share with anyone else (a sub-domain of the ISP allocated just to yourself would work); or (2) each local user has a separate POP3 mailbox and something akin to multipop is used.

In my setup, I have use both SMTP to get mail delivered to me directly via my own domain and multi-pop to retrieve e-mail from several mailboxes on the ISP domain and from several mailboxes on other ISPs. I use method (2) from above as there are no sub-domains I can use.

-- Jason

BTW Any idea if multipop is built into SME 5.6? From what I see, it is a simple feature to  implement, and absolutely vital in any _practical_ setup (I'm guessing most people retrieve their mail from more than one POP3 mailbox).

[Kelvin]

Hi Jason,

The scenarios I've worked with requiring this usually involves companies with multiple sites / offices that only own one domain. Being a company with many levels of users, it was always going to be simpler to just have everyone in the organisation have e-mails of the same domain (ie. someone@companydomain.com). The head office SME would be the only mail server having a complete set of users. The remote offices collect mails from the head office except for any mails deemed internal. Eg. Joe and Sally are in office1 while Mike is at the head office. If Joe sends Sally an email (sally@companymail.com), it gets delivered locally (without needing to go out to the net or the head office first). If Joe sends Mike an e-mail (mike@companymail.com) the SME server at office1 will find that Mike is not a local account but is in the same domain and so passes it on to the head office for processing. If the e-mails are not for someone from the same domain, it sends it out to the net instead of passing it to the head office. In this scenario, Mike might well be in office2 instead of the head office and the same process occurs.

Unfortunately, because SME does not go far enough with its server manager to expose more of qmail's more powerful features (and there are a great many), a lot of "hacks" are required (although, strictly speaking, I would not call them that - they are features available in qmail, just not easily accessible without needing to work at console level - hence a hack as far as newbies / non-technical administrators are concerned). SME must evolve to the stage where most of it's components' features are server-manager accessible before it can truly be an "easy to setup and administer" server. It achieves this already to an extent but still requires far too much console / command prompt access to configure the other "features" still.

Oh yes, multipop is NOT in SME 5.6.

Kelvin

[Jason Judge]

Totally agree with everything Kelvin!

I think over that last few years there have been  many contributions that solve real-world problems and have been shown to be extremely popular. It's a shame more of them are not making their way into the base SME install. I can see the argument for keeping it simple, as that is one of the main selling points, but some features are more simplistic than simple, and miss out on some really useful business tools.

Having an hierarchy of SME servers sounds great - thanks for explaining it - it looks really scalable to the largest of organisations (Mitel shouldn't handcuff themselves to the smaller end of the SMEs, IMO). Perhaps there is scope for someone to produce an mega-RPM that installs all these handy admin screens in one go (multipop, hierarchical mail servers, a fully working hylafax, etc.). That's just wishful thinking though - I wish I had the time to volunteer to help put something together

Being based on a standard Linux and largely standard RPMs is the best bit, of course. If you know the tools then you can get the box to do anything.

-- Jason

[Jason Judge]

Hehe. The mail delivery failure came back to me a few minutes ago (testing out the 'catchall' address mentioned in the thread above). It took 36 hours to return to me. Blueyonder must wait until off-peak time before sending out delivery failures in batches

[Mark]

I found this at http://www.arda.homeunix.net/mailsetup.shtml#fetchmail and thought that it might help.

Let me know what you think,
Mark

Retrieving Mail from my ISP

In addition to the mail accounts I have set up on my own mail server, I still have an email account with my ISP that I just haven’t been able to ween myself away from. Rather than have mail in two different locations, I decided it would be nice to be able to automatically download mail from this account into an account on my own mail server. Another reason to do this is because I get spammed on this account from time to time. Rather than pay my ISP to apply their anti-spam measures to my account, I’d rather download the messages to my server and use my own spam-stopping tools. This is what I’m using Fetchmail to accomplish.
Fetchmail really is a gem of a program. My simple setup doesn’t do justice to its capabilities. If anyone needs a program to take mail from one or more mail accounts and deliver it to accounts on one or more other mail servers, you should take a look at Fetchmail. You’ll be glad you did.
Since my ISP mail account doesn’t see a lot of volume, I’m running Fetchmail twice a day from a cron job to check for mail on my ISP’s mail server and download it to Callisto. Here is what the cron job in /etc/crontab looks like.
# check my ISP mail account twice a day
24      5,18    *       *       *       vmail   /usr/local/bin/fetchmail -f /home/vmail/.fetchmailrc
Fetchmail runs as vmail, the system account I use to control virtual mail accounts on Callisto. Fetchmail doesn’t have to run as vmail, but I didn’t see any reason why it should run as root and running it as vmail is convenient.
In my setup, Fetchmail gets most its runtime parameters from a configuration file, /home/vmail/.fetchmailrc. Here is what the file looks like.
# fetchmail configuration file

set postmaster postmaster@arda.homeunix.net
set syslog

poll pop1.ISP.domain.dom protocol pop3 authenticate password
        user ""
        password ""
        is "andrewISP"
        smtpaddress arda.homeunix.net
Since this file contains the password to my mail account on my ISP’s server, I’ve set the permissions on this file to 600; something that Fetchmail requires, in fact. It’s nice when a program does the Right Thing by default. Here is a listing of the configuration file.
-rw-------  1 vmail    vmail     230 Mar 26 23:38 /home/vmail/.fetchmailrc
This configuration file tells Fetchmail to connect to my ISP’s mail server using the POP3 protocol, log in with my user ID and password, and deliver any mail it finds there to andrewISP@arda.homeunix.net on my local mail server. The smtpaddress parameter is necessary because otherwise Fetchmail will try to deliver mail to andrewISP@localhost, something my qmail setup doesn’t like. By default, Fetchmail tries to connect to port 25 on localhost to deliver mail it has retrieved which is why I don’t need to tell Fetchmail where my mail server is.
Once Fetchmail was set up, I needed to create an entry in qmail’s user database and set up an account directory in /home/vmail/arda-homeunix-net just like I would with any other mail user account.
And that’s it. For very little effort, I am able to pull all of my mail delivered to my ISP’s mail server into my own mail server where I can manipulate it to my heart's content.