Topic: Spam Filter

[sonoracomm]

Hi all,

Does anyone have any recommendations for configuring spam filtering on SME 7?  The configuration panel text, unlike most server-manager panels, doesn't really help me, though once I understand the options better, it will probably make sense.

For me, experimentation with settings has not been easy or helpful.  Changes I have experimented with don't seem to work as you would think, though my testing was on RC1.

Is there any documentation that anyone could point me to specific to SME 7?

Yes, I've spent the requisite two hours serching the forum, documentation and buglist for info.  Most of what you find is not for SME 7.  Although I found several posts by knowledgeable folks admonishing posters to "search the forum" for answers, I didn't find any basic configuration information.

Thanks in advance,

G

[idyll]

Hello.

I would assume you refer specifically to the page titled "Email settings" ? I also assume you are running the latest build? Assuming these are true...

A good starting point is do you have any experience with spamassassin? If you don't, then the spamassassin wiki is a good place to get a base education. If you do, the terminology should make more sense than your post implies.

The Spam Sensitivity is designed to get you started by determining the low and high thresholds for the system to use to sort or delete messages tagged with a specific numeric "rating" based upon characteristics associated with 'spam".

Thus the "low" number will be the number which is the "low " mark for determining if email is SPAM or not. The "high" number is the one to be careful with as email tagged higher than this integer is automatically deleted. Mail with ratings between these two numbers is sorted to the "junkmail" folder per user, optionally.

Does this 10,000 foot view help?

regards,

patrick

[sonoracomm]

Thanks much for your help!

As you surmised, I am not familiar with SA.

I was hoping for a configuration guide for SA as implemented in SME 7...so as not to reinvent the wheel, so to speak.  However, your info gave me more to work on, and I spent another couple of hours writing this howto:

http://www.sonoracomm.com/index.php?option=com_content&task=view&id=49&Itemid=32

I would appreciiate any feedback to make this howto more accurate.  Please send corrections/additions to gcooper(at)sonoracomm(dot)com.

Thanks again for pointing me in the right direction,

G

p.s.  Here's a ASSP on SME howto I wrote.  I recently updated it for SME 7, but it needs testing.

http://www.sonoracomm.com/index.php?option=com_content&task=view&id=48&Itemid=32

[raem]

sonoracomm

>... for older versions of SME, I still recommend ASSP.

I wouldn't recommend using anything older than 6.x as they are insecure.

ASSP & similar became pretty much deprecated by features incorporated into and added to sme 6.0.x eg RBL, pattern matching etc, of course in conjunction with knuddi's antivirus & spam filter contribs and the additional tweaks mentioned in this howto.

http://mirror.contribs.org/smeserver//contribs/rmitchell/smeserver/howto/Mail%20system%20tweaks%20HOWTO%20for%20sme%20server.htm

[raem]

sonoracomm

The howto looks good. It will help novices tryng to understand the concepts.

In my opinion RBL rejection is a fundamental part of spam & virus management.
I would strongly suggest adding the commands to your howto for enabling and configuring RBL on sme7. As this is not enabled by default, and users may be unsure which RBL 's to use, some information would be good. Search here on RBL to find the commands which I (& others) have posted previously.

[sonoracomm]

Hi Ray,

Thanks for your input.

Well, I spent another hour researching...and updated the howto.  I just added commands to enable the default blacklists as I didn't find any suggestions as to particular lists.

If you have any specific suggestions, they most welcome.

As for ASSP on SME 6.x, it is incorrect to write it off as being obsoleted by RBLs.  Believe me when I tell you that if SA was not already included in SME 7, I'd be installing ASSP.  And I have literally dozens of installations in the field (SME and Windows) to base my opinion on.  ASSP is small, fast, elegant, easy to configure and uses many advanced techniques to fight spam.  It is said that the technology in ASSP obsoleted RBLs...and the results are easy to see.

I look forward to gaining more experience with SA so that I'll have a more balanced perspective...  

Thanks again and more (specific) comments/changes/fixes to this howto are most welcome, from anyone.

G

[idyll]

Do a search and you'll find these two topics....

Specifically why RBL are NOT enabled by default on the server? And, can they slow down mail delivery? For perspective they are useful for troubleshooting and to understand why the developers opted not to enable them by default.  I use RBLS and they kick ass.

Bayes "learning" is also disabled by default. If enabled, then the SA Bayes filtering needs to "see" or learn 200 SPAM and 200 HAM before it will try to learn new spam signatures. This is when SA filtering becomes really, really granular and effective.  I find this feature to be glossed on 50% of the SA installations I've seen. No clue why.

Brian Read posted a script Knuddi and I devised to "teach" SA about email which should have been tagged as spam. It's a simple, cron-driven PERL script. You have the user create a LearnAsSpam folder and they can drop this not-tagged email into the folder. It is collected on a cron and taught to the Bayes process. Brian is creating the awesome SPAM statistics contrib and the LearnAsSpam PERL scrip is in his contrib repository under his name.

You should use both of the scripts if you administer multiple machines or domains, etc.

regards,

patrick

[sonoracomm]

Hi Patrick,

Thanks much for the input.  I'd like to bounce a few comments off of you and Ray for your thoughts.

When I first ventured out as a spam fighter, I used RBLs exclusively.  I found back then that they stopped about 57% of the junk. and that seems to match Ray's more contemporary statistics.  Not bad results for a truly minimal effort..  Then I started using ASSP which implemented RBLs...then it grew out of them after a while.  As long as the RBLs are up to date and well-maintained, they provide a great value...but I'm no longer familiar with the reliability of individual RBLs and I'd appreciate the recommendations of folks who are in order to improve the howto..

As for performance, I've noted much greater hits when spam was allowed in.  Also, as we are in the business, we routinely replace our customers' servers at three years of age (unless they are beefy, industrial SCSI RAID5 boxes, which we replace at 3-4 years of age).  This means that horsepower has not been/is not a real problem for me. I'll opt for the RBLs, unless there is a problem.

Bayesian "learning" filtering has always been part of my ASSP installations.  I always used the sample spam tar file to prepopulate the spam/notspam databases so ASSP took off running.  Maybe we should use the same technique with SA? Here's a URL to the sample database, if you're interested.

http://easynews.dl.sourceforge.net/sourceforge/assp/asspsmpl-0.1.tgz

I'm mainly looking for a way to add reasonably reliable spam filtering with as little maintenance overhead as possible.  I want to set it and forget it...unless the customer complains and is willing to pay for more of my time to fix the problem.  

Would simply turning on the Bayesian filtering work to that end?  I realize it won't be optimized for some time, but my customers seem OK with that concept...as long as the bulk of the junk gets knocked down immediately.   Should I change my recommendations in the howto to allow _less_ (or no) spam to be rejected in order to populate the Bayesian database?

As far as reporting spam/ham, I have found that it is the rare user who cares about this function.  They just delete the spam and move on.  Also, if the filter is working properly, it's a rare message that gets through as a false-negative (based on my experience with ASSP).  In this context, I would question the value of time spent setting up the LearnAsSpam functionality (and training users to use it).  Perhaps it's different for SA...

Thanks again to both you and Ray for your insights.

G

[raem]

sonoracomm & idyll

A contribs forum search on spamhaus found this recent post

http://forums.contribs.org/index.php?topic=32054.0

which lists the RBL's I use.

sbl-xbl.spamhaus.org
relays.ordb.org
dnsbl.njabl.org
whois.rfc-ignorant.org


Do either/both of you use pattern matching executable content rejection (a contrib for sme 6 & default for sme7) as part of your spam & virus defence system.

It's like saying RBL knocks the stuffing out of them and pattern matching finishes them off.
About the only thing getting through to my servers are html viruses and these are caught by clamav. A handful of spam messages are filtered to the junkmail folders and 99% or more of these are correctly identified. False positives are infrequent.

[raem]

sonoracomm

> I spent another hour researching...I didn't find any suggestions as to particular lists....

A 3 second search would have found what you needed.

A search on RHSBL found this and quite a few others that are useful

http://forums.contribs.org/index.php?topic=29803.0

http://forums.contribs.org/index.php?topic=30792.0

http://forums.contribs.org/index.php?topic=29799.0


My original howto details the various lists quite well.

http://mirror.contribs.org/smeserver//contribs/rmitchell/smeserver/howto/Spam%20blocking%20HOWTO%20using%20smtpfront-qmail%20for%20sme%20server.htm

& revised for sme7

http://mirror.contribs.org/smeserver//contribs/rmitchell/smeserver/howto/Spam%20blocking%20HOWTO%20using%20qpsmtpd%20&%20RBL%20for%20sme%20server.htm

[idyll]

I use the default RBLs. The only RBL I would warn about would be SPAMCOP. They are really aggressive and I just prefer slightly less arbitrary email nuking  

I have noted almost zero pattern matched rejections. I believe I have seen four. I have them all enabled except for ZIP2, which is useful to allow to pass. All of the others are enabled. I rearely see virus, maybe one a month.

Ray - as I am using Brian Read's contrib to gather statistics, do you know the command syntax to specifically read pattern-matched rejections? I may be rejecting more but I have no clue how to gather that data manually as I rely upon Brian's statistic contrib.

In my case, the need to train Bayes is important as the spammers are continually tweaking their email to slip by my filters. I can share examples of this. Sonoracomm - the database you allude to is not going to plug into SA. But thanks for the creative thought!

My SA settings are custom, and rather harsh. I consider 4 to be the low edge and twelve the high edge. I incur zero errant deletions, but I still see perhaps 10 or less SPAM get by per week overall. These I drop into the folder I mentioned, and in a very short time these are learned and not seen again. It works great for the newer, more sophisticated spam email.

Have you seen the new type which are generally drug or "enhancement" related, which have a link and then a series of nonsense words? These can get by, very often, unless the SA system is trained, if they actually luck onto your literal email name. If they don't add up enough tokens, there is nothing to stop them. So they must be learned.

At least, this has been my experience.

I also have virtually no false positives.

Overall, I am /extremely/ pleased with the SME7 filtering capabilities. I don't believe it can get any better in any practical sense. Maybe I just like fiddling, but the auto-learn aspect, to me, is a small tweak which pays a nice dividend as well.

I previously used ASSP and found it to be very difficult to manage with a user base of diverse recipients. That may well have been just my inability to grok the concept. I also had serious CPU redlining, runaway race conditions which the developer never seemed willing to assist troubleshooting. In fact, the lack of developer support was my biggest beef. I found the overall Spam Assassin community to be not only vast, but far more interested in helping. Again, this is just my experience and I am pleased you have made it work to your satisfaction.

How many times has a great product been less than successful because of issues other than the product, per say? Many times, he says.....

Thanks for assembling the material you have assembled, I am SURE this will be read by hundreds of new users, to the mutual benefit of all.

regards,

patrick

[idyll]

Here's my editorial take....

the whole issue of SPAM detection, etc. is rapidly approaching the levels associated with deep inspection on firewalls.

Our commercial installations (multinational) require almost universal "reject all, accept the following exceptions" in order to preclude denial or service and the raft of other automated attacks.

Thus in my world, aggressive filtering and equally aggressive whitelisting is a sane appoach to existing in the hostility of the net. It is not grossly intensive to maintain but it sure as heck is not set and forget.

regards,

patrick

[sonoracomm]

Thank you both again!

I made some changes to the howto based on both of your comments.

I also decided to keep this a simple (quickie) howto and point foks at other resources if they want to do more.  I decided to draw the line at enabling the Byesian filter as things went beyond 'quickie' at that point.  I'd include the Bayesian filtering if someone suggested some text for this document that made it look easy.  I made the cutoff when it appeared you have to enable console access for each user...  I know the RemoteUserAccess contrib does this easily, but...

Could you please check the howto one more time for accuracy?  I would sure appreciate it!

http://www.sonoracomm.com/index.php?option=com_content&task=view&id=49&Itemid=32

Thanks again for all your time and trouble,

G

p.s.  Updated howto 6/16/06 to include Bayesian filtering.

[azche24]

Hi Sonoracomm and Ray Mitchell,

i would like to say thanks:

Information on Spamfilters in SME 7 was pretty wide-spread and is now bundled nicely in your howtos.

I do not know the reason, why RBL is not ON in SME7 by default. But it should be possible even for linux newbies to implement that from your howtos.

[raem]

idyll

>..... do you know the command syntax to specifically read pattern-matched rejections?

Apply these to sme 7 server.

On a sme 6.0 server the (smtpfront-qmail/current) log file entry looks like this:

2006-05-28 18:12:41.622887500 smtpfront-qmail[2580]: 554 We don't accept email with executable content TVqQAAMAA - TVqQAAMAA (#5.3.4)


Here is an entry for the "badhelo" rejection in mailfront

2006-05-29 08:17:52.198154500 smtpfront-qmail[24711]: 553 Sorry, I don't believe that is who you are.


Here is an entry for the invalid recipient rejection from the dungog-mailblocking contrib, default behaviour now in sme 7.

2006-05-29 08:51:16.217505500 smtpfront-qmail[25639]: 553 Sorry, that is an invalid e-mail address

[bovnet]

I ve always been happy with assp on sme 6.

Assp whilst very effective and self tuning has recently undergone a leap in development with added features.

Has anyone put it onto sme 7 is it still possible with sme added built in spam features to disble them and just add assp ?

[sonoracomm]

I have not installed ASSP on SME 7, however, I'd be very interested in your results.

The only ASSP upgrade I have done recently was on a Windows server.  The upgrade was painless and I've had no trouble with it.  (I will note that on that Windows server, I did have to set a daily scheduled job to restart the ASSP service...it would occasionally die and not restart before...I haven't tested with the new version, I just let it restart every day...never had that problem on Linux)

ASSP works as a proxy to the smtp daemon...it effectively sits in front of the existing services.  Therefore, I doubt you will have any problems with it conflicting, but...  I would make sure the SA spam filter is disabled in the Server-Manager E-Mail panel.

Please keep us informed.  If you'll e-mail me any changes to the howto, I'll be happy to add them.

Thanks,

G

[m]

sonoracomm,

to make the Bayesian filter working, you first must create the bayes databases:

# sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
# chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
# chmod 750 /var/spool/spamd/.spamassassin/bayes_*


You may also want add a note to your Howto, that the Bayesian autolearn feature is active. With default thresholds 0.1 (ham) and 12 (spam).

[sonoracomm]

Hi Michael,

Thanks for your input.

I am a little confused by your post.  Before performing your recommended steps, I looked at the location:

Code: [Select]

[root@mail ~]# ll /var/spool/spamd/.spamassassin
total 300
-rw--w-rw-  1 spamd spamd      6 Jul 18 04:40 bayes.mutex
-rw-r-----  1 spamd spamd  24576 Jul 18 04:40 bayes_seen
-rw-r-----  1 spamd spamd 335872 Jul 18 04:40 bayes_toks

Does this indicate that the databases had already been created and are of the correct permissions?

Also, I don't quite understaand your last suggestion.  Are you suggesting that I inform the admins that once the autolearn feature is enabled that the default thresholds are 0.1 and 12?  Are these good settings?  Should I recommend changing them to something else?  How?

Also, after giving some thought to the Bayesian filtering, It seems to me this may not be a really good idea to implement without some easy, automated or semi-automated whitelisting method.  Without an easy method for users to deal with false-positives, they can be show-stoppers.  Does anyone have any suggestions for whitelisting?

Thanks again,

G

[m]

sonoracomm,

you find me confused, too.
These are the databases. The permissions/ownership is ok. (spamd can read/write)

Did you a fresh SME7 Final install?
I have updated from RC3 to final. On RC3 the databases were not created and on SME7 Final they were not re-created after I have deleted them. May be they were only created during fresh install.

Autolearn:
I feel, that score 12 for spam is much to high. I have run autolearn with bayes_auto_learn_threshold_spam=4 for years on serveral SME6 servers . To change the thresholds, you need a new template.
I just packaged an RPM that enables site-wide Bayesian Filter, enables blacklist testing and intodruces spamassassin properties BayesAutoLearnThresholdSpam and BayesAutoLearnThresholdNonspam.
If you want to have a look on it, you can find it here: http://mirror.contribs.org/smeserver/contribs/michaelw/sme7/smeserver-spamassassin-features-0.0.1-0.noarch.rpm

[azche24]

Hi,

I just packaged an RPM that enables site-wide Bayesian Filter, enables blacklist testing and intodruces spamassassin properties BayesAutoLearnThresholdSpam and BayesAutoLearnThresholdNonspam.
If you want to have a look on it, you can find it here: http://mirror.contribs.org/smeserver/contribs/michaelw/sme7/smeserver-spamassassin-features-0.0.1-0.noarch.rpm

great. Works!

No more fiddling with chown and db-locations!

[rich]

Autolearn:
I feel, that score 12 for spam is much to high. I have run autolearn with bayes_auto_learn_threshold_spam=4 for years on serveral SME6 servers . To change the thresholds, you need a new template.
I just packaged an RPM that enables site-wide Bayesian Filter, enables blacklist testing and intodruces spamassassin properties BayesAutoLearnThresholdSpam and BayesAutoLearnThresholdNonspam.
If you want to have a look on it, you can find it here: http://mirror.contribs.org/smeserver/contribs/michaelw/sme7/smeserver-spamassassin-features-0.0.1-0.noarch.rpm

I installed your RPM and got the following error:

[root@catpower ~]# rpm -Uvh smeserver-spamassassin-features-0.0.1-0.noarch.rpm
Preparing...                ########################################### [100%]
   1:smeserver-spamassassin-########################################### [100%]
ERROR: configuration specifies 'use_bayes 0', sa-learn disabled
chown: cannot access /var/spool/spamd/bayes_*': No such file or directory
chmod: cannot access /var/spool/spamd/bayes_*': No such file or directory
spamassassin properties:
BayesAutoLearnThresholdNonspam=0.10
BayesAutoLearnThresholdSpam=4.00
UseBayes=1
qpsmtpd properties:
DNSBL=enabled
RHSBL=enabled


In /var/log/spamd/current I see  ",autolearn=no" at the end of each entry.
Any suggestions?

[azche24]

ERROR: configuration specifies 'use_bayes 0', sa-learn disabled
chown: cannot access /var/spool/spamd/bayes_*': No such file or directory
chmod: cannot access /var/spool/spamd/bayes_*': No such file or directory

Go to shell and into some junkmail directory. Do

Code: [Select]

sa-learn --spam *

once and repeat the installation of the contrib.rpm. That should have created the bayes databases and the install should work.

Or you issue the commands from the shell stated here earlier. This will also create the bayes dbs.

hth

[m]

I installed your RPM and got the following error:

RPM fixed. Thanks for testing and reporting.
New RPM is on its way to the mirror:
http://mirror.contribs.org/smeserver/contribs/michaelw/sme7/smeserver-spamassassin-features-0.0.2-0.noarch.rpm

[rich]

That worked without errors.
I am watching the logs to see the effects . . .

[rich]

Okay, let me see if I get this . . .

spamd clearly spots this as spam and then says "autolearn=spam"
I assume that qmail won't even accept a mail from them next time.
Is that the case?
If not then what is it doing?


@4000000044bea977309dea4c [3377] info: spamd: connection from localhost [127.0.0.1] at port 32796
@4000000044bea9773180dbf4 [3377] info: spamd: checking message <OBVWEA$HZDCD8574774VEW65B866UK1R57O0488@connectusa.com> for qpsmtpd:1005
@4000000044bea97e1510344c [3377] info: spamd: identified spam (21.0/5.0) for qpsmtpd:1005 in 6.5 seconds, 2509 bytes.
@4000000044bea97e1510538c [3377] info: spamd: result: Y 21 - HTML_MESSAGE,MIME_HTML_ONLY,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E4_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_NJABL_DUL,UNPARSEABLE_RELAY,URIBL_AB_SURBL,URIBL_JP_SURBL,URIBL_OB_SURBL,URIBL_SBL,URIBL_SC_SURBL scantime=6.5,size=2509,user=qpsmtpd,uid=1005,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=32796,mid=<OBVWEA$HZDCD8574774VEW65B866UK1R57O0488@connectusa.com>,autolearn=spam

[m]

I assume that qmail won't even accept a mail from them next time.
Is that the case?

No.

If not then what is it doing?

In short: Spamassassin has learned something new about spam (tokens).
It will use this knowledge to score future mails, regardless of the sender.

See: http://wiki.apache.org/spamassassin/BayesInSpamAssassin

[mercyh]

Good Day,

I have attempted to set up Bayes with SpamAssassin using the LearnAsSpam.pl from here. http://www.sonoracomm.com/sonoracomm/pdf/howto_configure_sme7_spamfilter.pdf

When the nightly cron runs I get the following error:

bayes: expire_old_tokens: locker: safe_lock: cannot create lockfile /var/spool/spamd/.spamassassin/bayes.mutex: Permission denied

I have done the following

chmod 777 /var/spool/spamd/.spamassassin/
chown <user> /var/spool/spamd/.spamassassin/

the chown command worked one time with the following run logged in as the user:

-bash-3.00$ sa-learn --spam *

however when the nightly cron ran I was back to the old error message for that user. (this is the only user learning spam on my system.)

When I run: sa-learn --spam *  as Super User everything is happy.

Any Ideas or suggestions??

Thanks for sharing your work on the script and howto

Royce Holdeman

[brianr]

Something seems to have changed in sa-learn in version 3.1.3, i have tried the spamassassin mailing lists, but got no real help.

So, I have uploaded a newer version of the LearnAsSpam perl script to my contribs area, it now runs sa-learn as root, not as the user, please let me know if it works for you..it seems to work for me now.

Contribs area:

mirror.contribs.org/smeserver/contribs/bread/mailstats

look for LearnAsSpam.pl dated today (26thJuly), it might take a few hours to sync up.

Ideally you would have reported this as a bug through bugzilla.

[mercyh]

Brian,

Thanks for a quick answer. I was not sure which buglist to go post on as this is really a contrib that grew from a question on the main server forum. I have removed the old version of the script and the cron job associated with it but will not have time to reinstall for a couple of days.

If the new script runs as root, does each user still need shell access??


Thanks for your work  

Royce Holdeman

[brianr]

If the new script runs as root, does each user still need shell access??

I am not sure, but I presume not - perhaps you can try it?

[mercyh]

Brianr

>>If the new script runs as root, does each user still need shell access??

>>I am not sure, but I presume not - perhaps you can try it?

I got the time to install the script last night and set up a LearnAsSpam folder for a user without Shell access. The Cron job ran and worked great. The spam  for the user without shell was processed. This is a nice improvement for some of us that  need things simple.

Royce Holdeman

[mercyh]

Brianr

I got the time to install the script last night and set up a LearnAsSpam folder for a user without Shell access. The Cron job ran and worked great. The spam for the user without shell was processed. This is a nice improvement for some of us that need things simple.

I spoke too quickly. It looks like the script needs shell for each user to perform the LearnAsSpam folder delete function.


Royce Holdeman

[mdo]

I have just started to look into the script only and ran into that (deletion) problem as well. I changed it for root to do the delete command like this:

# Now delete the file after learning
#my $delete = su - $key -c "rm -f $filetolearn";
my $delete = su - root -c "rm -f $filetolearn";

Michael

[Curly]

The cron-job rus as root, so the delete-command could be:

my $delete = rm -f $filetolearn;

I like the feature !

[brianr]

ok, have editted in curley's suggestion, the script no longer uses sudo, so does not need shell access for the users.

Also uploaded "sa-update" which runs the new for SA 3.1.3 online rule update program.  Have been running it for about 10 days with no problems, Note that  it does not run under SME6 - there are clearly some perl libraries missing.

Any comments?

[JonB]

Thanks for the sa-update Brian.

I note that you actually need to run it twice. The first time it fails due to an invalid verification key but downloads the correct key.

second time it works like a charm.

Jon

[sonoracomm]

Hi All,

I updated my Spam Filter Configuration for SME 7 howto.  

Very little testing has been done, so please help me out by sending me any corrections, suggestions, errors, configuration ideas, etc. to gcooper(at)sonoracomm.com.

http://www.sonoracomm.com/index.php?option=com_content&task=view&id=49&Itemid=32

Thanks to everyone for all their help and contributions,

G

[mercyh]

Hello again,

Just a note for those just starting with this.

D_gerst has posted a contrib RPM that sets everything up and gets you started with Bayes learning. See the thread here:

http://forums.contribs.org/index.php?topic=33350.msg142252#msg142252

Good Luck!

Royce H.