Koozali.org: home of the SME Server

Obsolete Releases => SME Server 9.x => Topic started by: holck on November 24, 2016, 11:14:40 PM

Title: qpsmtpd statistics
Post by: holck on November 24, 2016, 11:14:40 PM: I have made a small Perl-script that takes the /var/log/qpsmtpd/current log-file as input and produces statistics like shown below. It gives a good overview, I think, of the reasons for denial.
Code: [Select]
Queued: 2710 (87 marked as spam) Denied: 10860 SMEOptimizer: 18 ( 0 %) Failed Authentication: 1059 (10 %) Relaying Denied: 151 ( 1 %) DNSBL: 7350 (68 %) RHSBL: 1299 (12 %) Invalid Host: 856 ( 8 %) Spamassassin: 27 ( 0 %) Virus: 2 ( 0 %) Early Talker: 90 ( 1 %) TLS negotion failed: 7 ( 0 %) Blacklists: Barracudacentral: 116 ( 1 %) Psbl.org: 4 ( 0 %) Spamcop: 66 ( 1 %) Spamhaus: 6695 (77 %) Surbl: 448 ( 5 %) Uceprotect: 469 ( 5 %) Uribl: 851 (10 %) Most active IP addresses: 185. 56. 82. 83: 126 185. 40. 4.121: 121 195. 22.127. 22: 83 185.125. 4.153: 65 93.174. 93. 84: 51 96.254.184. 26: 50 80. 82. 64.102: 49 77. 88.202. 42: 41 23.246.213.202: 38 117.218. 72. 93: 38

Here is the script:
Code: [Select]
#!/usr/bin/perl # use strict; # Sample, standard deny-line from qpsmtpd/current: # # @4000000057e971280f9518fc 7923 logging::logterse plugin (deny): ` 192.241.146.6 mta-wk-2.mk1.enchantitect.com mta-wk-2.mk1.enchantitect.com <c736fb27-sio-2IEpgeKf9g1V1Z0D@mk1.enchantitect.com> rhsbl 901 Blocked, enchantitect.com on lists [abuse], See: http://www.surbl.org/lists.html msg denied before queued # SMEOptimizer works by forcing a high spam score: # # @40000000582033e43560c98c 28003 smeoptimizer plugin (deny): SMEOptimizer SA hit: BAYES_00,DIGEST_MULTIPLE,HTML_MESSAGE,PYZOR_CHECK,RAZOR2_CF_RANGE_51_100,RAZOR2_CF_RANGE_E8_51_100,RAZOR2_CHECK,RCVD_IN_DNSWL_NONE,SMEOPTI_URI_SPAM,SPF_HELO_PASS,SPF_PASS # @40000000582033e43560d92c 28003 logging::logterse plugin (deny): ` 46.21.172.157 vserver3.axc.nl ashwinbihari.nl <freja_olsen@ashwinbihari.nl> <hsdsdc@ibsgaarden.dk> spamassassin 901 spam score exceeded threshold (#5.6.1) Yes, hits=13.1 required=3.0_ # # @4000000058207d3c2be3f834 8548 smeoptimizer plugin (queue): SMEOptimizer SA hit: BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_FONT_LOW_CONTRAST,HTML_MESSAGE,MIME_HTML_MOSTLY,MPART_ALT_DIFF,RCVD_IN_DNSWL_NONE,RCVD_IN_IADB_DK,RCVD_IN_IADB_LISTED,RCVD_IN_IADB_RDNS,RCVD_IN_IADB_SENDERID,RCVD_IN_IADB_SPF,RCVD_IN_IADB_VOUCHED,RP_MATCHES_RCVD,SMEOPTI_URI_SPAM,SPF_PASS # @4000000058207d3c2be40fa4 8548 logging::logterse plugin (queue): ` 91.235.232.1 smtp2-1.mailmailmail.net smtp2-1.mailmailmail.net <return-b6984-b202471-hefdfdlga.baaddfdfsgaard=ibsgaarden.dk@mailmailmail.net> <hedsdlga.badfdfadsgaard@ibsgaarden.dk> queued <72249250c81f557c67e6e65e6472b009@client2.mailmailmail.net> Yes, hits=4.6 required=3.0_ use warnings; my @denial = ( [0, "SMEOptimizer", qr/SMEOptimizer SA hit/], # Smeopti must be the first [0, "Failed Authentication", qr/authcvm/], [0, "Relaying Denied", qr/relaying/], [0, "DNSBL", qr/\sdnsbl\s/], [0, "RHSBL", qr/\srhsbl\s/], [0, "Invalid Host", qr/believe that you are/], [0, "Spamassassin", qr/exceeded threshold/], [0, "Virus", qr/Virus Found/], [0, "Early Talker", qr/earlytalk/], [0, "TLS negotion failed", qr/Negotiation Failed/] ); my %bl; my $queued = 0; my $unknown = ""; my $spam = 0; my ($line, $denied, $check, $smeoptimizer_plugin); sub count_black_lists { my $list = shift @_; $list =~ s!.*https?://!!; $list =~ s!/.*!!; $list =~ s!\w+\.(\w+)\..*!$1!; $bl{$list} ||= +0; $bl{$list}++; } Check: while ($line = <>) { chomp $line; if ($line =~ /smeoptimizer plugin/) { $smeoptimizer_plugin = 1; # Remember this and read the next line $line = <>; chomp $line } else { $smeoptimizer_plugin = 0 } $queued++ if ($line =~ /$queue$/); if ($line =~ /$deny$/) { $denied++; $ip = (split "`", $line)[1]; $ip =~ s/^\s+//; $ip =~ s/\s.*//; unless ($ip =~ /\d/) { print "Line = $line\nIP = $ip\n"; die } $attempts{$ip} ||= 0; $attempts{$ip}++; if ($smeoptimizer_plugin) { $check = $denial[0]; $check->[0]++ } else { foreach $check (@denial) { if ($line =~ $check->[2]) { $check->[0]++; if ($check->[1] =~ /BL/) { count_black_lists($line) } next Check } } } $line =~ s/.*`//; $unknown .= " $line\n" # Unidentified reason for deny } elsif ($line =~ /$queue$/) { if ($line =~ 'Yes, ') { $spam++ # Queued but marked as spam } } } print "\n\n"; printf "%-12s%5d", "Queued:", $queued; print " ($spam marked as spam)\n"; printf "%-12s%5d\n", "Denied:", $denied; foreach $check (@denial) { printf " %-25s%5d (%2d %%)\n", $check->[1].":", $check->[0], int(0.5 + $check->[0] / $denied * 100) } $bl_total = 0; foreach $list (keys %bl) { $bl_total += $bl{$list} } print "\nBlacklists:\n"; foreach $list (sort keys %bl) { printf " %-18s%5d (%2d %%)\n", ucfirst($list).":", $bl{$list}, int(0.5 + $bl{$list} / $bl_total * 100) } print "\nMost active IP addresses:\n"; $n = 1; foreach $ip (reverse (sort { $attempts{$a} <=> $attempts{$b} } keys %attempts)) { # @bytes = split (/\./, $ip); printf " %3d\.%3d\.%3d\.%3d", split(/\./, $ip); printf ": %5d\n", $attempts{$ip}; $n++; last if ($n > 10) } # print "\n\nUnknown reason for deny:\n", $unknown;
Title: Re: qpsmtpd statistics
Post by: brianr on November 25, 2016, 12:04:25 AM: Interesting - does this work on SME8 and SME9.1?

The newer version of qpsmtpd in SME9.2 (qpsmtpd version 0.96) has some differences in the log file format and the plugins.

Also the Current file is rotated to *.s files on post-upgrade and reboot, and also at some point depending on the logrotate rules. So by using just the "Current" file you only get a snapshot since the last rotate.

Have you seen my Mailstats program?

https://wiki.contribs.org/Mailstats
Title: Re: qpsmtpd statistics
Post by: Jean-Philippe Pialasse on November 25, 2016, 12:07:15 AM: have you tried smeserver-mailstats ?

most of your stats are present in it. You will receive a daily mail with statistics.

I am pretty sure that if any are missing, it would be easy to improve it asking the contributor to insert part of your code ( would you Brian ? :) ) !
Title: Re: qpsmtpd statistics
Post by: brianr on November 25, 2016, 08:17:57 AM: Quote from: Jean-Philippe Pialasse on November 25, 2016, 12:07:15 AM
I am pretty sure that if any are missing, it would be easy to improve it asking the contributor to insert part of your code ( would you Brian ? :) ) !

Certainly consider it.
Title: Re: qpsmtpd statistics
Post by: brianr on November 27, 2016, 10:37:39 PM: Quote from: brianr on November 25, 2016, 08:17:57 AM
Certainly consider it.

mailstats does not have a league table of IP addresses - not quite sure I understand the usefullness - anyone got an thoughts?
Title: Re: qpsmtpd statistics
Post by: Stefano on November 28, 2016, 10:45:09 AM: Quote from: brianr on November 27, 2016, 10:37:39 PM
mailstats does not have a league table of IP addresses - not quite sure I understand the usefullness - anyone got an thoughts?

I agree with you.. I don't see any plus in having such a list..
BTW, I strongly suggest everyone to use fail2ban contrib.. it helps a lot limiting connection from spam bots.. in my setup, an IP can try only 3 times, after it's blocked at firewall level..
Title: Re: qpsmtpd statistics
Post by: Jean-Philippe Pialasse on November 28, 2016, 04:19:45 PM: I think the IP could be interesting for bot and other attacks.

even with fail2ban I have seen attack where they run on 10 to 100 different IP and they do connexions at a rate fail2ban will not ban. Also even if ban , the default first ban is short, they will be able to try again soon.

Also it can show you that a server with noble intentions has problem sending you emails.
Title: Re: qpsmtpd statistics
Post by: holck on November 28, 2016, 10:18:09 PM: Thank you for the comments. I didn't know of Brian's contrib - I will certainly take a look at it :-)

As Brian says, using /var/log/qpsmtpd/current I only get a snapshot of the situation. But mostly, this is also what I'm interested in. For me, the situation changes day by day.

As Jean-Philippe mentions, fail2ban is certainly a great help, but some spammers get (partially) around this by using different IP-addresses, however often in the same subnets.

It has been surprising for me to see how effective Spamhaus' DNSBLs are. It is an indication, as I see it, that there is not much else for me to do than rely on these DNSBLs. For me, only very small amounts of spam are caught by spamassassin, SMEoptimizer, or clamAV. It also means that fail2ban is valuable for blocking repeated login-attacks, but fail2ban cannot contribute much to blocking spam email, sent to valid receivers.

Warm regards,
Jesper
Title: Re: qpsmtpd statistics
Post by: Stefano on November 29, 2016, 10:09:11 AM: I'm using barracudacentral (found it once in the wiki but can't find it now..) and I see it's very effective too
Title: Re: qpsmtpd statistics
Post by: Jean-Philippe Pialasse on December 07, 2016, 12:00:05 AM: the efficiency of every element in the chain is relative to the efficiency of the others placed before it.

the biggest problem with something too efficient is the risk of false positives ....

speaking of accounting the email incoming traffic I have worked on this :

(https://bugs.contribs.org/attachment.cgi?id=5729)

see https://bugs.contribs.org/show_bug.cgi?id=8818

all comments are welcomed!
Title: Re: qpsmtpd statistics
Post by: mmccarn on December 07, 2016, 12:25:55 PM: Quote from: Stefano on November 29, 2016, 10:09:11 AM
I'm using barracudacentral (found it once in the wiki but can't find it now..) and I see it's very effective too

There are fixes coming for support of A-record RBL's (like barracudacentral).

SME9 - smeserver-qpsmtpd-2.4.0-19.sme (my SME9 with latest patches is running smeserver-qpsmtpd-2.4.0-14)
Bug 9498 - qpsmtpd dnsbl does not allow configuration of blocklists without TXT records (https://bugs.contribs.org/show_bug.cgi?id=9498)

SME10 - smeserver-qpsmtpd-2.6.0-11.sme
Bug 8484 - qpsmtpd dnsbl does not allow configuration of blocklists without TXT records (https://bugs.contribs.org/show_bug.cgi?id=8484)

Here are my notes on getting barracudacentral working on SME8.2:
Topic: [Solved] Barracuda DNSBL doesn't seem to be working with qpsmtpd (https://forums.contribs.org/index.php?topic=50941.0)
Title: Re: qpsmtpd statistics
Post by: Stefano on December 07, 2016, 12:53:24 PM: mmmhh...

Code: [Select]
[root@fileserver ~]# rpm -qa | grep smeserver-qpsmtpd smeserver-qpsmtpd-2.4.0-32.el6.sme.noarch
more info:
Code: [Select]
[root@fileserver ~]# yum list smeserver-qpsmtpd Plugin abilitati:fastestmirror, needs-restarting, presto, security, smeserver Loading mirror speeds from cached hostfile * base: centos.fastbull.org * smeaddons: mirror.mab974.re * smeextras: mirror.mab974.re * smeos: mirror.mab974.re * smeupdates: mirror.mab974.re * updates: mirror.de.leaseweb.net base | 3.7 kB 00:00 smeaddons | 2.9 kB 00:00 smeextras | 2.9 kB 00:00 smeos | 3.1 kB 00:00 smeupdates | 2.9 kB 00:00 updates | 3.4 kB 00:00 updates/primary_db | 3.7 MB 00:01 Pacchetti installati smeserver-qpsmtpd.noarch 2.4.0-32.el6.sme @smeupdates-testing
and
Code: [Select]
[root@fileserver ~]# yum list qpsmtpd Plugin abilitati:fastestmirror, needs-restarting, presto, security, smeserver Loading mirror speeds from cached hostfile * base: centos.fastbull.org * smeaddons: mirror.mab974.re * smeextras: mirror.mab974.re * smeos: mirror.mab974.re * smeupdates: mirror.mab974.re * updates: mirror.de.leaseweb.net Pacchetti installati qpsmtpd.noarch 0.96-8.el6.sme @smeupdates-testingBTW, barracuda is working here..