Topic: forwarding emails and spamfilter

[tiwang]

hi out there
I have several accounts on my sme-server 6.01 for which I forward their emails to external smtp servers. After I added the spam-filter (spamassasin 2.53) they don't get their emails deliverd any more ?? Any suggestion on how to track this ? Are there a way to enable debugging in qmail ? or maybe somebody out there have had similary problems ? If I reconfigure the mailaccounts to keep a local copy also the local email is delivered - but the forwarded mail to the external mail server never shows up ?

[tiwang]

extra note regarding my forwading problem: how is this forwarding facility implemented ??

regards /ti

[CharlieBrady]

hi out there
I have several accounts on my sme-server 6.01 for which I forward their emails to external smtp servers. After I added the spam-filter (spamassasin 2.53) they don't get their emails deliverd any more ??

No-one can answer your question until you are more specific about what additional software you have installed on your server. sme-server 6.01 does not do any spam filtering, even if you install a spamassassin rpm.

[tiwang]

hi again
ok ok - could have been more specific, but if you search this site for spamfilter you get a hit "how to install spamfilter for dummies" which is what I expect normally is used on the SME servers:
http://sme.swerts-knudsen.dk/index.html?frame=http%3A//sme.swerts-knudsen.dk/howtos/howto_29.htm
This package installs without any problems and is simple to configure afterwards. But - it seems to me that after I have installed this package I cannot have both emails delivered locally AND forwarded to an external server. If I make use of either local delivery or forwarding to an external it works - but not both - can somebody out there give me some tips about where to look ?
Regards /tiwang

[frederikbay]

Had the same problem, and disabled spam filter and removed the forward for the users in question and saved.

Then readded the forward and saved again, and then enabled spam filter, then it worked for me.


Know it's a rough workaround and if you have a lot for forwards it's a killer.

- Frederik

[tiwang]

thanks for the advice - this was also what I would try as next step - it is just annoying not to know why but I am already spending to much time with my PC so I don't worry any more..

[lancew]

Cool, I was finding the same problem, good to know how to fix it.

Lance

P.s. Has anyone actually checked if you need to reset every users settings? Or is just disabling and re-enabling the spam filter enough?

P.p.s. I also have this nagging feeling that doing the fix in this thread stops the spam filter from working properly. In that it then starts tagging email but still delivering the spam into your inbox.

[lancew]

As I mentioned above applying this fix seemed to stop Jespers spam filter contrib from moving spam emails to the junk mail folder. (the emails were being tagged as spam, but seemed to just sit in the inbox)

I have solved this by from the console typing:
lat-procmail -c "username|enabled|no|some|normal" for each of my in house users. (not those with email redirection)

Although this works, it is not ideal as although (as yet) I have no proof I assume that the redirected emails would receive the spam.

Any thoughts anyone?

[verti]

I try it the command, but it fails the result, After executing it the Spam does not move to the junkmail folder.

   

we have continue searching...  Thanks

[lancew]

I try it the command, but it fails the result, After executing it the Spam does not move to the junkmail folder.

   

we have continue searching...  Thanks

What happens when you type in:

Code: [Select]


lat-procmail -c "username|enabled|no|some|normal"

                 ^^^^^^^^  <-- Don't forget to change this to the login ID of your user.

What messages do you get?

Lance

[verti]

say:

Code: [Select]

Activating procmail for user 'verti'

Then the spam has ben moved to JunkMail folder but not fordward the mail.

Thanks

[kruhm]

i have the same problem.
SME 6.0.1-01+

users that are set on LOCAL & FORWARD get the message locally but the message never get forwarded.

i've tried several things (turn off AV scan of outing, restart spam filter, etc) but the only way i can get it to forward is to set it to FORWARD EMAIL TO ADDRESS BELOW.

i saw one post ( http://forums.contribs.org/index.php?topic=21324.msg87894#msg87894 ) that suggests customizing "e-smithForward20" but i can't find that file anywhere.

another post ( http://forums.contribs.org/index.php?topic=21324.msg87894#msg87894 ) sets a rule for each user, but then i couldn't use LAT easily

ADDITIONAL INFO ADDED:
since SA works with PROCMAIL for filtering to junkmail folder, if PROCMAIL is turned on then PROCMAIL handles the email processing (hence procmail), including forwarding of messages. PROCMAIL must be told to forward the message. This can be done in the USER-MANAGER. Too bad I can't have LAT easily add rules for each user.

Any other workarounds or fixes?

[kruhm]

well, it happened again. When SpamAssassin updated, it automatically turned on procmail with a:
Running event handler: /etc/e-smith/events/spamfilter-update/S10spamfilter-expand-procmailrc

Good for most because it will automatically sort out the JunkMail. But it also overtakes the handling of the mail and therefore turns off all the mail forwards setup.

Interestingly (and maybe I'm the last to find this out), a lat-dump -d shows "procmail" in the EMAILFORWARD field which isn't documented as valid input. Since "procmail" isn't valid, it never shows in the Users page and stays as "both deliver locally and forward" which is where the confusion for most is.

Is there a way to automatically create a .procmailrc template for all users to get the mail to forward?

[raem]

kruhm & others

> since SA works with PROCMAIL for filtering to
> junkmail folder, if PROCMAIL is turned on then
> PROCMAIL handles the email processing (hence
> procmail), including forwarding of messages.
> PROCMAIL must be told to forward the message.
> This can be done in the USER-MANAGER.

The crux of the issue is that as designed the system is working as it should, it's just that users expectations are somewhat different.

When Jesper Knudsens Spam Filter script runs, it installs the User Manager contrib and each user is automatically configured to use procmail to filter to the junkmail folder.
The settings in the standard server manager Collaboration/Users panel are overridden. Even though it says deliver "Both deliver locally & forward" that is incorrect as the settings in the actual User Manager contrib prevail (www.yourserver/user-manager) as procmail has been selected for all users by the spamssassin install. Unfortunately server manager does not have knowledge of the User Manager settings and incorrectly shows a value of  "Both deliver locally & forward"  when it should ideally show "procmail enabled" or something similar. That's the problem of integrating other contribs into the base system, not always perfect.

Using procmail implies that only local delivery of messages can occur (procmail does not work any other way except on locally delivered messages).
Therefore if junkmail messages filtering is enabled then procmail is enabled and you will have to manually setup procmail rules for each user that requires mail to be forwarded to external users.

I think that with LAT you can selectively add procmail rules for selected users (based on a list you create for LAT to use) but I have not tried it myself so please check the LAT documentation.
Other than that you would need to log in to each users User Manager (ie www.yourserver/user-manager) one by one and create the required forwarding rule within the user manager contrib panel using procmail. You will of course need to know the password for each user.

Sorry for the good/bad news.

[kruhm]

Other than that you would need to log in to each users User Manager (ie www.yourserver/user-manager) one by one and create the required forwarding rule within the user manager contrib panel using procmail.

That's exactly what I mean. I don't want to go one by one.

Isn't there a way to get:

my $ForwardAddress = db_get_prop(\%accounts, $USERNAME, "ForwardAddress");

automatically into the .procmailrc at /etc/e-smith/templates-user/.procmailrc/99default?

[kruhm]

this post was removed for documentation purposes. see solution in post below. thanks,

[Damian]

This is seriously bad news. I now have 4 sites screwed due to the change.

Knuddi - didn't you want to mention these changes so that we could choose to update or not ? Or did you and I missed it ? I mean, I trust your packages but now who knows what cleanup needs to be done after an upgrade. Lost faith there a little.

Also thanks to Ray for explaining the nuts-n-bolts. I tried modifying the user-manager page for the few users that I do know the password for and it still doesn't forward. These sites have no techy expertise and rely on external techies to fix these things for them.

In most cases they want a copy of all incoming mail to go to a "safe" inbox, untouched by human hand except in the case of legal action. This was broken by the upgrade.

This leaves our SME servers in an unknown state. I'm embarrased that I don't have an answer for the customer's who are now emailing with these issues.

I've recommended SME to sites because it's rock solid and then find it isn't. Not a happy day.

Damian

[Damian]

Are all the Spamassassin contribs thus mangled ? Is there any mileage in going a different route for spam ?
The last SA update breaks server-manager as it renders the Add User section's mail settings useless.

[kruhm]

It's just a consequence of using the contrib. It isn't Jesper's job to get it working in all scenerios (or even at all). Though some documentation would have been nice.

It would be nice if spam was removed before it hit the user with 'qmail-scanner' as it can "integrate with SpamAssassin to provide comprehensive anti-spam tagging for an entire site" (http://qmail-scanner.sourceforge.net/). This would leave procmail out of the solution. Or reroute the procmail to follow the delivery already in place (explained in last post). With a small team, or someone who knows a little more than I do, it could probably finished in a night's work- maybe less. Then released in a new contrib.

But don't hold your breathe, since I seem to be the only one offering solutions/interested in getting it to work. My work/contribs amounts to a small pile.

Before you go fixing things be aware that turning procmail off with a: lat-procmail -c "*|disabled|no|some|normal" will set all your users to deliver local only. It's better to do a

lat-dump -d
change the file quickly in EXCEL/spreadsheet
lat-users -a -i <path to file>

This actually updates the users' information already there (something else I didn't see in documentation). It's a pain but it's better than doing it by hand.

[raem]

Damian

> The last SA update breaks server-manager as it renders the Add User section's mail settings useless.
> I've recommended SME to sites because it's rock solid and then find it isn't. Not a happy day.

I think you are being unfair on sme & Jesper, and  blaming them for not taking it upon on yourself to fully analyse the consequences of installing a add on contrib.
That's the way the Spam Filter contrib works and as a result of that, it does what it does.
If you don't like the functionality then put your request (nicely) to the author of the contrib for the changes you think are necessary.

You don't have to use it !!

[Damian]

All,

I'm not blaming anyone. I appreciate the very great amount of voluntary work done by all of these contributors and I know that I don't have to use any contrib. But when the contribs are as well made and functionally useful as Jesper's and work time after time you get casual about checking. That's actually a complement.
Say I ran a business leasing cars to customers where the new cars are delivered straight to them. After a couple of years the manufacturer decided to ship all new models without glass in the windows. The first I knew is when the customers started to ring me. That's kind of where I'm coming from (although I don't sell cars ). I welcome the new auto-update feature in these contribs except that by using it you're definately out of the verification-before-insallation loop.
Anyhow, there's some useful info coming out of this thread and I'll definately keep watching it. If I wasn't too dumb to program I'd even contribute code
Damian

[raem]

Damian

> I welcome the new auto-update feature in these contribs except that by using it you're definately out of the verification-before-insallation loop.

That's not exactly correct, you can choose the option that suits your situation.
The spam filter panel (update field) has choices to Warn only, Download & warn, Download & install or Disable updates.

[d6hq]

I fell foul of this also - poor documentation of an otherwise excellent contrib. Am I correct in thinking that the "Download and warn" setting will allow me to perform a SA update at our discretion?

[kruhm]

no, no, no, all wrong! all wrong!

Here, this should work:

CREATE CUSTOM TEMPLATE I:
-mkdir -p /etc/e-smith/templates-user-custom/.procmailrc
-cp /etc/e-smith/templates-user/.procmailrc/99_spamfilter_default
/etc/e-smith/templates-user-custom/.procmailrc
-in /etc/e-smith/templates-user/.procmailrc/99_spamfilter_default you should have

Code: [Select]


{
 # check whether the User-Manager panel has placed the template files
 # before creating one.

 use esmith::config;
 use esmith::db;

 ###pulls account info
 my %accounts;
 tie %accounts, 'esmith::config', '/home/e-smith/accounts';

 my $ForwardAddress = db_get_prop(\%accounts, $USERNAME, "ForwardAddress");

 $OUT = '';

 unless ( -e '/etc/e-smith/templates-user/.procmailrc/99default' ) {

 # Default rule
 if ($ForwardAddress eq "")
 {
 $OUT .= "\n";
 $OUT .= "# --------------------------\n";
 $OUT .= "# all else goes to the inbox\n";
 $OUT .= "# --------------------------\n";
 $OUT .= ":0\n";
 $OUT .= "\$DEFAULT\n";
 }
 else
 {
 $OUT .= "\n";
 $OUT .= "# --------------------------\n";
 $OUT .= "# all else gets forwarded\n";
 $OUT .= "# --------------------------\n";
 $OUT .= ":0\n";
 $OUT .= "! $ForwardAddress\n";
 }
 } #end unless

}


This will forward all email to the FORWARDADDRESS field.

To capture the email and send a copy to the FORWARDADDRESS, put the following code underneath else (caution -this will fill up your HD space if users don't check regularly):

Code: [Select]


{
 $OUT .= "\n";
 $OUT .= "# --------------------------\n";
 $OUT .= "# all else goes to the inbox & to FORWARDADDRESS\n";
 $OUT .= "# --------------------------\n";
 $OUT .= ":0 c\n";
 $OUT .= "! $ForwardAddress\n";
 $OUT .= ":0\n";
 $OUT .= "\$DEFAULT\n";
 }


CREATE CUSTOM TEMPLATE II:
#create a new filter to stop email looping
vi /etc/e-smith/templates-user-custom/.procmailrc/45_procmail_bounce
-in /etc/e-smith/templates-user-custom/.procmailrc/45_procmail_bounce you should have

Code: [Select]


{

# If a forwarded mail gets bounced,
# it's delivered into the local INBOX to prevent looping
$OUT .= " ";
$OUT .= "# ---------------------------------- ";
$OUT .= "# all bounced mail goes to the inbox ";
$OUT .= "# ---------------------------------- ";
$OUT .= ":0 ";
$OUT .= "* ^FROM_DAEMON ";
$OUT .= "* ^FROM_MAILER ";
$OUT .= "$DEFAULT ";

}


CLEAN UP & CREATE .procmailrc's
Then delete or move the existing fragment: /etc/e-smith/templates-user/.procmailrc/99default to ensure that the recreation picks up the changes.

Then update everyone's individual .procmailrc file with a:
/sbin/e-smith/signal-event spamfilter-init-procmail

USERACCOUNTS PANEL:
::to change the useraccounts panel to show the 'procmail' option
-vi /etc/e-smith/web/functions/useraccounts
-the emailforward section should look like this:

Code: [Select]


        <field type="select" id="EmailForward" options="'local' =>
        'DELIVER_EMAIL_LOCALLY', 'forward' => 'FORWARD_EMAIL',
        'both' => 'DELIVER_AND_FORWARD', 'procmail' => 'Use procmail to
        process mail'" validation="nonblank" value='local'>
            <label>EMAIL_DELIVERY</label>
        </field>


FORCE PROCMAIL FOR ALL OPTIONS:
-vi /etc/e-smith/templates-user-custom/.qmail/e-smithForward20
-the emailforward section should look like this:

Code: [Select]


if ($EmailForward eq "local")
    {
        $OUT .= "\n";
        $OUT .= "| /usr/bin/procmail ~/.procmailrc \n";
    }
    elsif ($EmailForward eq 'forward')
    {
        $OUT .= "\n";
        $OUT .= "| /usr/bin/procmail ~/.procmailrc \n";
    }
    elsif ($EmailForward eq 'both')
    {
        $OUT .= "\n";
        $OUT .= "| /usr/bin/procmail ~/.procmailrc \n";
        $OUT .= "./Maildir/\n";
    }
    elsif ($EmailForward eq 'procmail')
    {
        $OUT .= "\n";
        $OUT .= "| /usr/bin/procmail ~/.procmailrc \n";
    }
    else
    {
        # this shouldn't happen - if it does, deliver through procmail
        $OUT .= "\n";
        $OUT .= "| /usr/bin/procmail ~/.procmailrc \n";
    }


USERPANEL-FORWARDING:
::to always include the procmail option
-vi /etc/e-smith/web/functions/userpanel-forwarding   
-the if/else statement for the procmail section should include 'procmail' in values and look like:

Code: [Select]


-values  => ['local', 'forward', 'both', 'procmail'],


ADDITIONAL NOTES:
-the individual .procmailrc files are recreated with a /sbin/e-smith/signal-event user-modprocmail <username>
-this automatically happens in userpanel-forwarding but not in the useraccounts panel
-the .procmailrc files are only recreated if the $emailforward value is "procmail"; if another value (forward, local, both) then the changes only take place with a /sbin/e-smith/signal-event spamfilter-init-procmail (i'm guessing user-modprocmail is hard-coded to react this way).

[filk]

I followed all of your instructions, however, .procmailrc is not picking up the template fragment.  Am I missing an "expand-template" step?  I don't think so, because the mod-procmail event is supposed to look at the user-custom templates.

I deleted all the .procmailrc files, but they are just being recreated exactly as they were before.

Any ideas why?

[kruhm]

there is no expand-template step for the .procmailrc

The individual files are created from the templates themselves (i think that makes sense). Whatever changes you make should be picked up when you recreate the .procmailrc's. [You can test this safely by deleting a .procmailrc in the users directory and changing some comments in the template. Then recreate the .procmailrc with  /sbin/e-smith/signal-event spamfilter-init-procmail ]

Be certain to delete/move the following file: /etc/e-smith/templates-user/.procmailrc/99default

Also, if you're deleting all of the .procmailrc's for the users, you shouldn't have to do a /sbin/e-smith/signal-event spamfilter-modprocmail (sorry about that)

you should only have to do a  /sbin/e-smith/signal-event spamfilter-init-procmail (i don't know if there's a way that works better than deleting and recreating)

of course, make sure you have a forwarding email address in for the users

[jbath]

I've been having similar problems.  What should the .procmailrc file look like after the template's been changed where I have a user with a forwarding email address?

I'm new to procmail and understand the concepts behind it, but the implementation is another thing entirely.

Thanks

[kruhm]

@What should the .procmailrc file look like after...
It will look exactly like the same as the default except the last section will include 2 lines to forward the email. The entire last section is below:

# --------------------------
# all else goes to the inbox
# --------------------------
:0 c
! <forwardaddress@willbehere>
:0
$DEFAULT

FYI -You can use midnight commander, mc, to easily delete all your .procmailrc files

@...but the implementation is another thing entirely
It took me awhile to figure it out myself. Procmail is install during the orginal sme install but not implemented until the sa contrib is applied. Each user gets an individual .procmail file derived from the templates.

[steever]

I am very sorry for hijacking this thread but I am trying to contact Damian.  Damian could you email me at towsons AT marist DOT ac DOT jp .  We need your help desperately regarding auto mounting home directories.

Once again, sorry!
 

[raem]

I'm not sure what happens when Auto update is turned on as I never allow anything to auto update my systems, but in the manually run install script there is clear warning displayed near the very beginning of the script that says:
"This script will enable procmail for all users and thereby filter spam to a junkmail folder. If you are forwarding emails then you will need to change the settings back after installation"

To quote:

clear
echo "========================================================"
echo "= SpamFilter Installation Script                       ="
echo "=                                                      ="
echo "= This script will install SpamAssassin, SpamFilter    ="
echo "= server-manager module as well as DCC and             ="
echo "= Razor2. It also ensures that you on an daily basis    ="
echo "= check for new SA updates. This script will only work ="
echo "= on SME 6.0.x                                         ="
echo "=                                                      ="
echo "= This script will enable procmail for all users and   ="
echo "= thereby filter spam to a junkmail folder. If you are ="
echo "= forwarding emails then you will need to change the   ="
echo "= settings back after installation                     ="
echo "=                                                      ="
echo "= No express or implied warranties are provided and its="
echo "= usage is at your own risk.                           ="
echo "=                                                      ="
echo "= If you feel confortable with the above then press    ="
echo "= enter if not press Ctrl+C to abort the installation  ="
echo "= script.                                              ="
echo "=                                                      ="
echo "========================================================"
read x
clear

[kruhm]

FYI -the above code was edited to include:

-bounce messages are captured in the local INBOX (this prevents email looping when procmail forwards)
-the visibility of PROCMAIL in the useraccounts panel
-forcing "procmail" to be enabled despite the selection (local, forward, both)
-the visibility of the USERPANEL-FORWARDING to always show for users without the USERPANEL-PROCMAIL panel showing (procmail confuses end-users)