Koozali.org: home of the SME Server

Obsolete Releases => SME Server 9.x => Topic started by: Jáder on July 17, 2016, 02:52:10 PM

Title: spamassassin vs groups
Post by: Jáder on July 17, 2016, 02:52:10 PM
Hi,

I've be loosing the fight against spam because company have created groups (like commercial, finnancial, technical) and make them public to send e-mails.
This way several people receive the e-mails... but spam appears to pass thru all config on spam.
Even bayes appears do not be effective when e-mail is directed to group e-mail.

Am I doing something wrong ?
Are there a way to enhance spamassassin to be efective on groups e-mails ?

Regards
Title: Re: spamassassin vs groups
Post by: mmccarn on July 18, 2016, 12:01:15 PM
I'm guessing here, but if there is something specific to spam filtering for groups, you might be able to get better performance (for eg 'groupA') like this:

1) Delete the existing groupA
2) Create a new group groupA-email
3) Create a user account groupA that forwards to groupA-email
Title: Re: spamassassin vs groups
Post by: Daniel B. on July 18, 2016, 12:12:09 PM
Spamassassin will work the same for a single user or for a group. There's no difference in how the email will be filtered. If you give the latest updates of qpsmtpd, smeserver-qpsmtpd and smeserver-spamassassin a shot (from smeupdates-testing) you might be able to have better result. Those updates brings several new filtering capabilities (uribl, karma, dkim, spf, dmarc), and provide more detailed spamassassin headers so you can analyse which spamassassin test is triggered (and the score for each)
Title: Re: spamassassin vs groups
Post by: Jean-Philippe Pialasse on July 18, 2016, 12:45:30 PM
Quote from: jader on July 17, 2016, 02:52:10 PM
Hi,

I've be loosing the fight against spam because company have created groups (like commercial, finnancial, technical) and make them public to send e-mails.
This way several people receive the e-mails... but spam appears to pass thru all config on spam.
Even bayes appears do not be effective when e-mail is directed to group e-mail.

Do you only activate the bayes or do you ask your users to reports spams ?

if you do not use it already I strongly suggest to install smeserver-learn https://wiki.koozali.org/Learn.
By asking a few of your user to put in the learnAsSpam folder you might enhance the effect of bayes. DO not hesitate to also pass a second time the emails already tagged [SPAM] in junkmail. Sometime there are more tags to learns.


Quote from: jader on July 17, 2016, 02:52:10 PM
Am I doing something wrong ?
no, but maybe there way to improve ;)

I strongly support what suggest Daniel, it might need a little work to tweak everything, but you should enhance your filtering capability.
Title: Re: spamassassin vs groups
Post by: Jáder on July 19, 2016, 11:12:09 PM
Quote from: Jean-Philippe Pialasse on July 18, 2016, 12:45:30 PM
Do you only activate the bayes or do you ask your users to reports spams ?

Yes, I´ve activated and installed Learn, most of users went moving spam to LearnAsSpam folder and I have a report script to verify how many each day, like this:

Code: [Select]
Tue Jul 19 17:59:13 BRT 2016
Usuario: administrativo2                LearnAsSpam: 160                        LearnAsHam: 0
Usuario: comercial3             LearnAsSpam: 34                         LearnAsHam: 0
Usuario: gcontratos             LearnAsSpam: 0                          LearnAsHam: 52
Usuario: michel                 LearnAsSpam: 70                         LearnAsHam: 0
Usuario: sec_dir                LearnAsSpam: 165                        LearnAsHam: 0
ls: cannot access supervisoradm/Maildir/.LearnAsSpam/cur: No such file or directory
---------------------------------------------------------------------------------------------
  Total:                        Spam: 429                                       Ham: 52

But when I verify the spam moved to LearnAsSpam, all of them (or 99.99%) is from e-mails delivered to a group (like sales@mydomain.tld) and several people received it and even if all of them move to LEarnAsSpam it continues to getting thru.

This is one segment of headers of one spam moved today to LEarnAsSpam by user sec_dir:
Code: [Select]
Received: (qmail 19070 invoked by uid 453); 19 Jul 2016 20:13:33 -0000
X-Spam-Level: *
X-Spam-Status: No, hits=1.0 required=4.0
        tests=HTML_IMAGE_ONLY_16,HTML_MESSAGE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS

Subject: =?ISO-8859-1?Q?Campanha_Rel=E2mpago_-_AutoCAD_LT_+_Revit_LT_+_Treinamento?=

Message-ID: <4bf778ddb69689e2798a5e0332f570b6@localhost.localdomain>
List-Unsubscribe: <http://grapho.campaignsender.com.br/admin/sair.php?id=274835|142|0&uid=142227342313080&acao=gravar>
X-List-Unsubscribe: <http://grapho.campaignsender.com.br/admin/sair.php?id=274835|142|0&uid=142227342313080&acao=gravar>
X-Unsubscribe-Web: <http://grapho.campaignsender.com.br/admin/sair.php?id=274835|142|0&uid=142227342313080&acao=gravar>
X-MessageID: 274835
X-ListMember: group@mydomain.tld
Precedence: bulk
X-LocaWeb-COR: locaweb_2009_x-mail
x-locaweb-id: 1RvfJ7/ecbQExJxUjicY0iel7v05/CrzN0nlvFses2GLdlm+uo6eNVEdCnMriVQNixGw1M+OqxWeACKmcw9AaHecZDOG13OJgvFCRtsIFiqqF6L85/NChB99OeOy90Fmafayg1ttKgQ6i/gw09iv7UIQZgm86I+Tgjypcx9PKr99I9qlslBxgDERJvQmwcuqEsUWXRdBZtJ1ielcPVIbkEU4xvxr8BsEgUfwVdu1sUk=
x-locaweb-id2: NmQ2MTcyNmI2NTc0Njk2ZTY3NDA2NzcyNjE3MDY4NmYyZTYzNmY2ZDJlNjI3Mg==
MIME-Version: 1.0

This user move all spam each day to folder LearnAsSpam... and this e-mail was target to group account!
Please note the subject is about a quick campaign to sell AutoCAD... :o
The source is a great ISP (in size not feature or user care) at Brasil.
And they include a X-ListMember and X-List-Unsubscribe, X-Unsubscrive-Web headers !!
It´s clear it´s a spam but is not being tagged as SPAM... not even close (score 1 of 4!)

Any tips ?

Jáder
PS: I can install lattest updates from testing, but not now, because I´m out of town in training and without reliable internet connection or access during daytime! After Friday/saturday I can change to testing version of SA.
Title: Re: spamassassin vs groups
Post by: mmccarn on July 20, 2016, 12:03:23 PM
Your example includes a 3 rules that probably have negative scores:
RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_PASS

I have one server that is constantly receiving snowshoe spam - I found that RHSBL using dbl.spamhaus.org helped (https://forums.contribs.org/index.php/topic,52105.msg266528.html#msg266528)

You might see improved behavior if you reset your bayes database as described here:
https://forums.contribs.org/index.php/topic,50712.msg258844.html#msg258844

(however, your sample email header doesn't include any bayes rules, so maybe not...)


Title: Re: spamassassin vs groups
Post by: Jáder on July 28, 2016, 03:07:59 AM
Hi you all.

Could you provide me with statistics as from "Count messages denied by DNSBL Block Lists" of https://wiki.contribs.org/Email_Statistics for some of your install ? (maybe later I'll update wiki with data - anonymous mode on - )

I'd like to compare numbers.
I heard about 90% received messages are SPAM but my server is identifying only 10% as SPAM.

Thank you
Title: Re: spamassassin vs groups
Post by: Knuddi on July 28, 2016, 03:03:12 PM
Normally the spam rate is around 80+% as can be seen on this graph which is for all domains on ScanMailX.

(https://www.scanmailx.com/domainstats.php?type=spamhourall&domain=ALL)

These numbers are not DNSBL but all methods used - I do not have numbers for the amount blocked by DNSBL, but I would assume that only 10% of this.
Title: Re: spamassassin vs groups
Post by: Jean-Philippe Pialasse on July 28, 2016, 03:15:08 PM
It highly depends on a lot of factors:
- your users habits
- number of users
- your volume of email
- the efficiency of your system in detecting the spams
- your domain name and its previous use
- your practice in divulging your emails


in all server I control, the rate vary from 97% spam to 10%.

On high volumes I would say it will tends to the numbers shown by Knuddi
Title: Re: spamassassin vs groups
Post by: Knuddi on July 29, 2016, 09:45:54 AM
I do not know how SME "handy" you are but I have just added NFR for SQLGrey which is very efficient for BOT type of senders.

https://bugs.contribs.org/show_bug.cgi?id=9707

Try also to add these extra DNBDL in a new file called /etc/mail/spamassassin/Blacklists.cf and restart SpamAssassin (sv t spamd). Then check whether any of these gets triggered.

Code: [Select]
## MailBlacklist.com Spam sources

header __RCVD_IN_MAILBLCOM_B eval:check_rbl('mailblcom-lastexternal', 'service.mailblacklist.com.')
tflags __RCVD_IN_MAILBLCOM_B net

##### MailBlacklist.com Definitions - Bad senders
# Definitions - Bad senders
#################################################

header          __RCVD_IN_MAILBLCOM_S eval:check_rbl_sub('mailblcom-lastexternal', '127.0.0.2')
describe        __RCVD_IN_MAILBLCOM_S Listed at MailBlacklist.com, Generic Blacklist Listing (-2)
tflags          __RCVD_IN_MAILBLCOM_S net

header          RCVD_IN_MAILBLCOM_B5 eval:check_rbl_sub('mailblcom-lastexternal', '127.0.0.6')
describe        RCVD_IN_MAILBLCOM_B5 Listed at MailBlacklist.com, Very Bad Reputation Sender (-5)
tflags          RCVD_IN_MAILBLCOM_B5 net

header          RCVD_IN_MAILBLCOM_B4 eval:check_rbl_sub('mailblcom-lastexternal', '127.0.0.7')
describe        RCVD_IN_MAILBLCOM_B4 Listed at MailBlacklist.com, Bad Reputation Sender (-4)
tflags          RCVD_IN_MAILBLCOM_B4 net

header          RCVD_IN_MAILBLCOM_B3 eval:check_rbl_sub('mailblcom-lastexternal', '127.0.0.8')
describe        RCVD_IN_MAILBLCOM_B3 Listed at MailBlacklist.com, Low Reputation Sender (-3)
tflags          RCVD_IN_MAILBLCOM_B3 net

header          RCVD_IN_MAILBLCOM_B2 eval:check_rbl_sub('mailblcom-lastexternal', '127.0.0.9')
describe        RCVD_IN_MAILBLCOM_B2 Listed at MailBlacklist.com, Suspicious Sender (-2)
tflags          RCVD_IN_MAILBLCOM_B2 net

# MailBlacklist.com Bad
meta            RCVD_IN_MAILBLCOM_BL RCVD_IN_MAILBLCOM_B5 || RCVD_IN_MAILBLCOM_B4 || RCVD_IN_MAILBLCOM_B3 || __RCVD_IN_MAILBLCOM_S
describe        RCVD_IN_MAILBLCOM_BL MailBlacklist.com Bad Senders
tflags          RCVD_IN_MAILBLCOM_BL net
score           RCVD_IN_MAILBLCOM_BL 1.0

##########################################################################################
## http://mailspike.org/usage.html
##########################################################################################

header __RCVD_IN_MSPIKE          eval:check_rbl('mspike-lastexternal', 'bl.mailspike.net.')
tflags __RCVD_IN_MSPIKE          net

##### Reputation compensations
# Definitions
header __RCVD_IN_MSPIKE_Z     eval:check_rbl_sub('mspike-lastexternal', '^127\.0\.0\.2$')
describe __RCVD_IN_MSPIKE_Z   Spam wave participant
tflags __RCVD_IN_MSPIKE_Z     net
header RCVD_IN_MSPIKE_L5     eval:check_rbl_sub('mspike-lastexternal', '^127\.0\.0\.10$')
describe RCVD_IN_MSPIKE_L5   Very bad reputation (-5)
tflags RCVD_IN_MSPIKE_L5     net
header RCVD_IN_MSPIKE_L4     eval:check_rbl_sub('mspike-lastexternal', '^127\.0\.0\.11$')
describe RCVD_IN_MSPIKE_L4   Bad reputation (-4)
tflags RCVD_IN_MSPIKE_L4     net
header RCVD_IN_MSPIKE_L3     eval:check_rbl_sub('mspike-lastexternal', '^127\.0\.0\.12$')
describe RCVD_IN_MSPIKE_L3   Low reputation (-3)
tflags RCVD_IN_MSPIKE_L3     net

# *_L and *_Z may overlap each other, so account for that
meta __RCVD_IN_MSPIKE_LOW RCVD_IN_MSPIKE_L5 || RCVD_IN_MSPIKE_L4 || RCVD_IN_MSPIKE_L3
meta RCVD_IN_MSPIKE_ZBI __RCVD_IN_MSPIKE_Z && !__RCVD_IN_MSPIKE_LOW

# Scores
score RCVD_IN_MSPIKE_ZBI     4.1
score RCVD_IN_MSPIKE_L5      4.1
score RCVD_IN_MSPIKE_L4      3.5
score RCVD_IN_MSPIKE_L3      2.9

# BarracudaCental.org RBL
header          RCVD_IN_BRBL            eval:check_rbl('brbl-lastexternal','b.barracudacentral.org')
describe        RCVD_IN_BRBL            Received via a relay in Barracuda BRBL
tflags          RCVD_IN_BRBL            net
score           RCVD_IN_BRBL            3.0

# Spam Eating Monkey
# SEM-BACKSCATTER
header          RCVD_IN_SEMBACKSCATTER  eval:check_rbl('sembackscatter-lastexternal', 'backscatter.spameatingmonkey.net')
tflags          RCVD_IN_SEMBACKSCATTER  net
describe        RCVD_IN_SEMBACKSCATTER  Received from an IP listed by SEM-BACKSCATTER
score           RCVD_IN_SEMBACKSCATTER  0.5

# SEM-BLACK
header          RCVD_IN_SEMBLACK        eval:check_rbl('semblack-lastexternal', 'bl.spameatingmonkey.net')
tflags          RCVD_IN_SEMBLACK        net
describe        RCVD_IN_SEMBLACK        Received from an IP listed by SEM-BLACK
score           RCVD_IN_SEMBLACK        0.5

# SEM-URI
urirhssub       SEM_URI                 uribl.spameatingmonkey.net. A 2
body            SEM_URI                 eval:check_uridnsbl('SEM_URI')
describe        SEM_URI                 Contains a URI listed by SEM-URI
tflags          SEM_URI                 net
score           SEM_URI                 0.5

# SEM-URIRED
urirhssub       SEM_URIRED              urired.spameatingmonkey.net. A 2
body            SEM_URIRED              eval:check_uridnsbl('SEM_URIRED')
describe        SEM_URIRED              Contains a URI listed by SEM-URIRED
tflags          SEM_URIRED              net
score           SEM_URIRED              0.5

# SEM-FRESH
urirhssub       SEM_FRESH               fresh.spameatingmonkey.net. A 2
body            SEM_FRESH               eval:check_uridnsbl('SEM_FRESH')
describe        SEM_FRESH               Contains a domain registered less than 5 days ago
tflags          SEM_FRESH               net
score           SEM_FRESH               0.5

# JunmkMailFilter (http://wiki.ctyme.com/index.php/Spam_DNS_Lists)
header          __RCVD_IN_JMF           eval:check_rbl('JMF-lastexternal','hostkarma.junkemailfilter.com')
describe        __RCVD_IN_JMF           Sender listed in JunkEmailFilter
tflags          __RCVD_IN_JMF           net

header          RCVD_IN_JMF_W           eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
describe        RCVD_IN_JMF_W           Sender listed in JMF-WHITE
tflags          RCVD_IN_JMF_W           net nice
score           RCVD_IN_JMF_W           -5

header          RCVD_IN_JMF_BL          eval:check_rbl_sub('JMF-lastexternal', '127.0.0.2')
describe        RCVD_IN_JMF_BL          Sender listed in JMF-BLACK
tflags          RCVD_IN_JMF_BL          net
score           RCVD_IN_JMF_BL          3.0

header          RCVD_IN_JMF_BR          eval:check_rbl_sub('JMF-lastexternal', '127.0.0.4')
describe        RCVD_IN_JMF_BR          Sender listed in JMF-BROWN
tflags          RCVD_IN_JMF_BR          net
score           RCVD_IN_JMF_BR          1.0
Title: Re: spamassassin vs groups
Post by: CharlieBrady on July 29, 2016, 05:39:35 PM
Quote from: Knuddi on July 29, 2016, 09:45:54 AM
I do not know how SME "handy" you are but I have just added NFR for SQLGrey which is very efficient for BOT type of senders.

However, as far as we know, that is no better than the already included greylisting plugin, which would be far simpler to use.
Title: Re: spamassassin vs groups
Post by: Jáder on August 10, 2016, 07:58:21 PM
I´m back!
This is statistics of my reference server:
Code: [Select]
[root@andorinha ~]# if [ -z $DAYS ]; then DAYS=1; fi; echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; awk -F"[\t]" ' /logterse plugin/ { svc=$6; count[svc]++; count["Total"]++; }  END  { for (j in count) print count[j] "\t" j; }' $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f) |sort -nr
Days of logfiles to scan [1]:
5113    Total
2482    queued
945     check_goodrcptto
872     rhsbl
415     dnsbl
266     tls
68      spamassassin
43      check_spamhelo
20      check_earlytalker
2       auth::auth_cvm_unix_local

 As you can see SpamAssassin is not helping a lot. :(
Our users receive a lot of SPAM. :( :(

I´ve tryed to use GREYLIST but users hated it... slow down e-mail delivered... and was shutdown as owner request.

I´ve NOT TESTED KNUDDI tip about "Try also to add these extra DNBDL in a new file called /etc/mail/spamassassin/Blacklists.cf and restart SpamAssassin (sv t spamd). Then check whether any of these gets triggered."

It was not clear to me what it should to do... so I´m afraid to implement.
Knuddi, could you elaborate about it ? Thanks

Regards

Jáder

Title: Re: spamassassin vs groups
Post by: Knuddi on August 10, 2016, 08:14:58 PM
Hi Jader,

The rules that I provided will simply add some additional lists to check in through SpamAssassin. I use these in a production environment with MANY mails so that will not trigger crazy wrong.

So create a new file in the directory /etc/mail/spamassassin called Blacklist.cf and add the content I provided. When saved then simply restart SpamAssassin with a "sv t spamd" and follow stats. You can monitor live through "tail -f /var/log/spamd/current" and see whether the rules are triggered.

You can grep in that file for the names that have a score in my list (e.g. RCVD_IN_MAILBLCOM_BL) to find matches.

Title: Re: spamassassin vs groups
Post by: Jáder on August 11, 2016, 04:12:19 PM
Hi Knuddi,

Thanks by your explanation. I´ve put your new rules in production.
Right now my stats are:

[root@andorinha ~]# if [ -z $DAYS ]; then DAYS=1; fi; echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; awk -F"[\t]" ' /logterse plugin/ { svc=$6; count[svc]++; count["Total"]++; }  END  { for (j in count) print count[j] "\t" j; }' $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f) |sort -nr
Days of logfiles to scan [1]:
3948    Total
1077    queued
1070    check_goodrcptto
949     rhsbl
537     dnsbl
124     tls
95      spamassassin
92      check_spamhelo
4       check_earlytalker


So just 2% are blocked by SA.
Are you seen different numbers ?
Could you post your numbers please ?

Regards
Title: Re: spamassassin vs groups
Post by: CharlieBrady on August 11, 2016, 08:36:12 PM
Quote from: jader on August 11, 2016, 04:12:19 PM
So just 2% are blocked by SA.

Depending on your settings, most SA positives will be sorted to junkmail rather than rejected. Those won't show up in the qpsmtpd stats.
Title: Re: spamassassin vs groups
Post by: Jáder on August 16, 2016, 12:06:01 PM
Yes, I'm aware about junkmail but that's not the case.

This is today statistics (at 7am):
[root@andorinha ~]# if [ -z $DAYS ]; then DAYS=1; fi; echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; awk -F"[\t]" ' /logterse plugin/ { svc=$6; count[svc]++; count["Total"]++; }  END  { for (j in count) print count[j] "\t" j; }' $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f) |sort -nr
Days of logfiles to scan [1]:
2654      Total
932       queued
713       rhsbl
566       check_goodrcptto
187       dnsbl
105       tls
101       check_spamhelo
32        spamassassin
18        check_earlytalker

So at monday we got 2654 messages and 932 went queued.

And yesterday:

[root@andorinha ~]# if [ -z $DAYS ]; then DAYS=1; fi; echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; awk -F"[\t]" ' /logterse plugin/ { svc=$6; count[svc]++; count["Total"]++; }  END  { for (j in count) print count[j] "\t" j; }' $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f) |sort -nr
Days of logfiles to scan [1]:
2518      Total
911       rhsbl
711       queued
438       check_goodrcptto
262       dnsbl
65        spamassassin
61        tls
44        check_spamhelo
24        check_earlytalker
2         count_unrecognized_commands
[root@andorinha ~]# if [ -z $DAYS ]; then DAYS=1; fi; echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; awk -F"[\t]" ' /logterse plugin/ { svc=$6; count[svc]++; count["Total"]++; }  END  { for (j in count) print count[j] "\t" j; }' $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f) |sort -nr
Days of logfiles to scan [1]: 2
4464      Total
1413      rhsbl
1173      queued
1154      check_goodrcptto
416       dnsbl
113       spamassassin
96        check_spamhelo
65        tls
26        check_earlytalker
5         auth::auth_cvm_unix_local
2         count_unrecognized_commands
1         check_badmailfrom

So at whole weekend We got 4464 messages (sat/sun) and 1173 went queued.
At weekend most (if not all) messages are spam.

SOMETHING IS WRONG. There are too many SPAM getting thru... even after several tweaks.
My users are getting as much as 100+ spam each day.

Can someone help me to improve Spamassassin hit count ?
Thanks

Jáder
Title: Re: spamassassin vs groups
Post by: Knuddi on August 16, 2016, 12:35:12 PM
Hi Jader,

If you would want to give it a go, then feel free tro try out ScanMailX filter - I can help you out with a free account if you please. Just to see whether your mail flow is truely unique and hard to filter.