Topic: Spamfilter seems to have stopped working!

[dave simmons]

I don't know if I should post here, or if it's a bug, but the spamfilter seems to have stopped working.

It was working perfectly until this weekend (24/8/07).  I have set the server up to mail me when updates are available.  On Friday I received a mail that there were updates available, so I installed them (one update I think).  I didn't change anything else, just the update and a reboot, and then this morning - bam - loads of spam in the mailboxes.  The spam filter hadn't caught anything since Friday's update - everything is coming through!

I didn't look closely at what was being update - I trust the machine and just apply the updates - but I don't think it was anything to do with Spam.

My questions are the following:

1. Is anyone else seeing this?
2. Did this update break spamassassin somehow?
3. Or is it coincidence -I don't really know how the filter works (I've never needed to)
4. How can I easily (and safely) remove the last update to see if the problem goes away - or is this impossible?

Grateful for any suggestions or comments.

[brianr]

Can you see the spamassassin text in the email headers?  - perhaps your could copy and paste one of your email headers here?

cheers

B.

[dave simmons]

Hello Brian,

Thanks for taking the time to reply.  Here is what I think you were asking:

X-Spam-Status: No, hits=1.4 required=4.0
   tests=SARE_ADULT2
X-Spam-Check-By: ********.be

If I understand this correctly, Spamassassin has awarded only 1.4 value and needs 4 to reject.  Are the spammers getting better? (and will the problem then go away automatically as spamassassin catches up?)

The weird thing is that before Friday everything was working perfectly here - the spam filter was catching about 100 spams per user per day (6 users) and letting maybe 3 or 4 spams per user through.  Nearly no false positives.  Unfortunatley our housekeeping is maybe too good because we have deleted all the old junk mails via web mail (so I can't check the headers etc.) - the server is not too powerful or large capacity so we try to clean up regularly.

If you need more info, please ask.

Thanks again,

Dave

[brianr]

Suggest you run

sa-update

and see what happens.

If you get some messages about a gpg key being wrong, then do what the message says, and then run sa-update again.

B.

[dave simmons]

I have just received a spam email - a classic spam text, which I know spamassassin has caught many times before:

"Hello! I am tired today. I am nice girl that would like to chat with you. Email me at pqxh@BestOnset.info only, because I am writing not from my personal email. Would you mind if I share some of my pictures with you?"

The spam header today is:
X-Spam-Status: No, hits=0.0 required=4.0
   tests=

I have looked back through the webmail (one of our guys is on holiday and therefore his junk mail has built up a bit).  I can find the same message (of course from a different address:  On 17 August the spam header (on the same machine) was:

"Hello! I am tired today. I am nice girl that would like to chat with you. Email me at a@mailmessageonline.info only, because I am writing not from my personal email. Will send some of my pictures "

X-Spam-Flag:  YES
X-Spam-Status:  Yes, hits=5.9 required=3.0 tests=DATE_IN_FUTURE_06_12,PYZOR_CHECK

Maybe this will give additional info.

I've just received a message that there's a new post, but will post this anyway in case it helps.

[dave simmons]

Sorry, I should have added, as far as I'm aware I haven't made any changes - certainly not manually or via the web interface.  Only applied updates.

[dave simmons]

Brian - will follow your suggestion and post results.

Thank you.

[dave simmons]

Brian, have followed your suggestion.  Indeed had the message about GPG...

Followed instructions on screen, and now have a command prompt.  Do I need to do the signal-event post-upgrade or reboot, or will it take care of itself?

[brianr]

Do I need to do the signal-event post-upgrade or reboot, or will it take care of itself?

no, you have just updated the SA rules. just stand back and wait for the spam..

B.

[dave simmons]

OK.  The spam's still flooding in.  Is this normal?

Sorry to ask - maybe they're stupid questions, but the beauty of the SME is that is has always just worked, so I have'nt really had to learn anything about the system.  I've been using it since version 5.6, and this is the first time it's done anything weird.

BTW.  The machine has not had a series of upgrades.  It was a new install of 7.? followed by an upgrade to 7.2 follwoing the instructions elsewhere on this site.

[brianr]

Look at the headers now..

{edit} perhaps you'd better try a post-upgrade and reboot, just incase..

B.

[dave simmons]

Here aer examples of the last 3 (since upgrade and a restart):

X-Spam-Status: No, hits=1.7 required=4.0
   tests=SARE_HTML_USL_OBFU

X-Spam-Status: No, hits=1.7 required=4.0
   tests=SARE_HTML_USL_OBFU

(above 2 x the same mail from different addresses - Cialis/Viagra)

X-Spam-Status: No, hits=0.0 required=4.0
   tests=
(this one is for Penis enlargement)

I'm searching through the junk mail file of my colleague to see if there are examples of the same ones, but they are very common spam mail.  If spamassassin can't catch this, there must be a problem.

Maybe my machine has been hacked?  How can I see this?

[dave simmons]

I don't know if this will give any further ideas, I found a command on the forum to show the spamassassin status, and this is what it gives:

BayesAutoLearnThresholdNonspam=0.10
BayesAutoLearnThresholdSpam=4.00
DNSAvailable=yes
MessageRetentionTime=90
OkLanguages=all
OkLocales=all
RejectLevel=12
ReportSafe=0
Sensitivity=custom
SkipRBLChecks=0
SortSpam=enabled
Subject=[SPAM]
SubjectTag=enabled
TagLevel=4
UseBayes=1
status=enabled

Am also doing a fresh install on an old server, and will try replacing the machine.

[brianr]

did you try a re-boot? and you did re-run sa-update after importing the gpg key?

B.

[dave simmons]

Have done reboot (twice).  Also did the signal-event post-upgrade before the reboot.

Spam is coming in about 1 every couple minutes.  Classic spam - Cialis/Viagra/penis enlargement etc. 

Must be something seriously wrong with installation/configuration of Spamassassin - this is very basic stuff.

Have already reinstalled 7.2 on our old server, and am restoring a backup.  Hopefully that will sort it out.

The only thing that worries me is what to do with the updates.  I am sure that I didn't change anything on the server - I only applied the updates via the server-manager.  But it can't be the updates which have caused this otherwise there would be other users with the same problem?

[dave simmons]

As an alternative, can you tell me/give me a link to how I can just remove and then reinstall spamassassin?

The positive side of this experience is that I'm learning more about the system.

[brianr]

My own production server is "totally" up to date and SA still works fine (and has been updated from pre-6 over the years)..

The reason that I suggested the sa-updata/gpg key thing was because I had a similar to you situation last week, with another server I look after, and the new set of updates (with the correct gpg key) seemed to fix it.  Spam that was getting 1.2 then started getting 20 odd!

Hope the re-install works

B.

[Mjohnson]

I am certainly experiencing the same problem as you, though not to your same degree.  Please check the following posts for other things to try.  Enabling the RBL seems to be of some assistance.  The RBL is a definite help for me, but the problem of SA not tagging obvious spam remains a puzzle.  Some days very little gets through and then others days the flood gates seem wide open.

http://forums.contribs.org/index.php?topic=37060.0

In my case (SME 7.2 fully updated), the server is being used as a front end to an Exchange Server.  As such I do not know how to "Teach" the Bayes filter mechanism in SME.  We rely upon the junk mail sorting ability of our Outlook Clients, which is frankly very good and is performing quite well.  Only a handful of spam makes it through untagged.

MJ

[dave simmons]

Hi,

Thanks for your reply mjohnson.

Am now up and running on a freshly-installed (but much less powerful) machine.  As the problem only started after I installed upgraded packages via the web interface, I am going to run it for a day or so without applying the patches.

On the one hand it's difficult to believe that only the upgrade caused the problem, but on the other, I don't touch the machine.  That's the beauty of the system -it just runs!  Also if it was the upgrade, loads of people would have the problem.

I had already seen the post referred to in your link, but to be honest, didn't fully understand it.  I'm also nervous about manually typing commands.  The rest of the machine was working fine - web sites, file server, print server etc.  It was just annoying about the spam.

I'm more comfortable doing a reinstall.  Will post my results - good or bad - later

[dunni]

Me too!  I have just migrated from 6.0 to a newly installed 7.2 using procedures described elsewhere.

In my case the whole thing was working perfectly until I installed some dungog contribs - namely smeserver-userpanel, smeserver-userpanels and smeserver-mailsorting.  After that, I get spamassassin behaviour as bad or worse than OP

e.g. on "Megadik" spam

X-Spam-Status: No, hits=1.4 required=4.0
   tests=SARE_ADULT2
X-Spam-Check-By: xyz.polyhedron.com

X-Spam-Status: No, hits=0.0 required=4.0
   tests=
X-Spam-Check-By: xyz.polyhedron.com

Most messages return a score of 0

The only reason I want these panels is for procmail - can I enable procmail without using these things?

[Brave Dave]

Me 3

I've checked and the spam log;
tail -f /var/log/spamd/current | tai64nlocal

reports amongst other things
spamd: result: . 0 - DRUGS_ERECTILE,HTML_MESSAGE,RDNS_DYNAMIC

Would have thought it should have attracted a higher score
SME 7.2
Done the sa-learn thing, reboot etc.

It's like it's not reading the scores properly

[p-jones]

If the spam filters are not working and you have ascertained it is not a "bug" in your system, take it to the bug tracker.

[Brave Dave]

Ok run yum update spamassassin and it seems to have rectified

spamassassin-3.2.2-43.el4 had problem

spamassassin-3.2.3-44.el4 seems good


Guess that's why there are updates 

[dave simmons]

Latest update - running 12 hours on freshly-installed machine.  No spam!

We haven't had a single spam message get through to users, but all the junkmail boxes are also empty.  Does this mean that:

1.  The new machine is throwing the messages away instead of sorting them into junkmail (if so, what about false positives?)
2.  There hasn't been any spam sent (difficult to believe when I see the volume we received over the last 2 days, even on our tiny domain)
3.  The problem is sorted.

Will switch the old machine back on this afternoon, and see what happens then - maybe it was a momentary blip.

Will also try the update suggested by davidbray.

The reason I didn't post to the bug tracker is that I don't know enough about the system to dare suggest there is a fault.  I prefer to believe that it's something I've done wrong.  I had waited 2 days to see if something came up about the problem on the forums, because something as basic as a spam filtering problem (where the system went from trapping several hundred per day to trapping nothing) should affect everyone if it was a bug.  There must be much bigger users (and therefore bigger spam volumes out there) than me.

[dave simmons]

Bizarre.  Have switched on old (problem) machine.  Spam is coming in again (3 mails in 5 minutes!)

Tried update, but yum says that it could not find an update for spamassassin, so I must be running the latest version.

Will have to reinstall the machine. Bugger!

[raem]

dave simmons

> Tried update, but yum says that it could not find an update for spamassassin, so I must be running the latest version.

What output does this give
rpm -qa spamassassin

[dave simmons]

Hello Ray,

Sorry for delay, had already switched off the machine.

Result is:  spamassassin-3.2.3-44.el4

[dave simmons]

A quick follow up - there must be something wrong with my spamassassin configuration.

Here is the header from the last spam mail received:

X-Spam-Status: No, hits=1.4 required=3.0
   tests=SARE_ADULT2

Here is the mail:

Good morning ashleigh
get a MASSIVE penis today

Alexey Siitonen
http://www.planetwo.com/


If I am interpreting the header correctly, sa needs a value of 3 to identify it as spam.  If I look at the e-mail configuration in the server-manager, this value is set at 4???  The reject level at 12.

My wife keeps receiving these mails about penis extensions.  She's threatening to register me???

Am trying to see the funny side.

Note to self:  Buy big hammer or axe

[raem]

dave

Result is:  spamassassin-3.2.3-44.el4

That's OK, what about
rpm -qa smeserver-spamassassin

I have
smeserver-spamassassin-1.4.0-14.el4.sme

Also is spamd running
sv s spamd

[dave simmons]

I have the same version of smeserver-spamassassin.

From the other command i get the following:

run: spamd: (pid 4530) 6304s, normally down; run: log: (pid 2476) 7441s

I have no idea what this means!

[raem]

dave simmons

Thats OK, spamd is running.

You said your other fresh install (I assume of 7.2) ran without any spam getting through. Maybe other settings are different.
What does this show
config show qpsmtpd

Can you run that on both servers and compare them

[dave simmons]

Hello Ray,

On problem machine I have:

qpsmtpd=service
Bcc=disabled
BccMode=cc
BccUser=maillog
DNSBL=enabled
LogLevel=8
MaxScannerSize=25000000
RBLList=sbl-xbl.spamhaus.org:whois.rfc-ignorant.org:dnsbl.njabl.org
RHSBL=enabled
RequireResolvableFromHost=no
SBLList=dsn.rfc-ignorant.org
access=public
status=enabled

On freshly installed machine I have exactly the same.

[raem]

dave simmons

On freshly installed machine I have exactly the same.

That config looks OK.
You must not be doing a fresh install of 7.2 though, as that is not the default configuration.

Time to list all the contribs, add ons or changes you have made to the server.

You could try to uninstall & reinstall spamassassin to see if that improves things.
Do all the following

rpm -e smeserver-spamassassin
rpm -e spamassassin
signal-event email-update

yum clean all
yum list updates
yum update

yum install spamassassin

[dave simmons]

Hello Ray,

The only difference between the machines is that the problem machine was installed with an older version of SME 7 and updated following the instructions elsewhere on this site.  The fresh install was carried out with a CD I downloaded yesterday, also from this site.  I chose the option to reformat the drive and perform a new install, so it should be a standard configuration.

I did restore from backup.  Does the restore from backup only bring over user files, or does it also change settings on the machine?

I haven't added any contribs to either machine - we're only a very small domain - a few web pages, 6 e-mail addresses plus an info@ address. 

I look sometimes at the contribs to see if there's anything which could be interesting for us, but because I don't know enough about the system I don't mess about.

That's what I've always liked about SME - it just works - I don't need to tinker with it!

I will follow your instructions and report back.

Thank you for taking the time for this.

[dave simmons]

Help!

First command failed:

rpm -e smeserver-spamassassin       

gives:

error: Failed dependencies:
         e-smith-spamassassin is needed by (installed) smeserver-support-1.6.0-39.el4.sme.noarch

I will wait for a reply before trying the commands in a different order

[newhopenet]

I'm trying to resolve what seems to be the exact symptoms that Dave has.

I get the same results on trying to re-install spamassassin.

[root@david ~]# rpm -e smeserver-spamassassin
error: Failed dependencies:
        e-smith-spamassassin is needed by (installed) smeserver-support-1.6.0-39.el4.sme.noarch
[root@david ~]#

[dave simmons]

I'm sorry to hear that someone else is having trouble, but also a bit relieved that it's not just me.

I just remembered a friend who also uses SME 7 - only for e-mail - and their domain gets loads more spam than us (many more users).

I gave him a call to ask if they were getting more spam the last few days -  and they're not.  He is using the same version as us - 7.2 - but he has not applied the last updates in the server-manager.  He didn't know that there were updates until I called.

He doesn't want to apply the updates to see if it breaks his machine (they really were having a big problem with spam before using SME).

Can it be the update that has caused the problem?

BTW.  Here is his config:

spamassassin=service
DNSAvailable=yes
MessageRetentionTime=90
OkLanguages=all
OkLocales=all
RejectLevel=0
ReportSafe=0
Sensitivity=medium
SkipRBLChecks=0
SortSpam=enabled
Subject=[SPAM]
SubjectTag=disabled
TagLevel=5
UseBayes=0
status=enabled

rpm -qa spamassassin gives:
spamassassin-3.2.2-43.el4

rpm -qa smeserver-spamassassin gives:
smeserver-spamassassin-1.4.0-14.el4.sme

config show qpsmtpd gives exactly the same output as on both my machines.

So his version of sa is older, but still working fine.

Could this be the problem?

[mudra]

I have been following this thread as we have been receiving more spam of late. I found that by changing my spam sensitivity to CUSTOM that the spam started to be blocked once again. Perhaps this was just conincidence as it has been working on and off for the last few days. I also had to import the GPG signature in order to update spam assassin. I hope that this may help someone. Mudra.

[brianr]

error: Failed dependencies:
         e-smith-spamassassin is needed by (installed) smeserver-support-1.6.0-39.el4.sme.noarch

that just means that there is another dependency - uninstall that first:

rpm -e smeserver-support

then do the other commands.

B.

[dave simmons]

Hi Brian, good to have you back.

Just one question - am I not risking breaking something else by uninstalling the smeserver-support?

Is it important?

What does it do?

[brianr]

Hi Brian, good to have you back.

Just one question - am I not risking breaking something else by uninstalling the smeserver-support?

Is it important?

What does it do?

I've been here all the tiime!

No real idea what it does, but i find it a little strange that it refers to "e-smith-spamassassin".

Please give us the result of

rpm -qa | grep spamassassin

here is mine:

smeserver-spamassassin-1.4.0-14.el4.sme
spamassassin-3.2.3-44.el4

Cheers

B.

[raem]

dave simmons

The fresh install was carried out with a CD I downloaded yesterday
I did restore from backup.

That means that all configuration is also restored. You have in effect created a replica of the server you originally backed up, including users, config & all server manager settings etc.
The only thing missing would be any add on contribs, and these would need to be reinstalled, but the config for those contribs would also have been restored. You said you did not install any extra contribs.

So that means one server has packages based on the 7.2 CD and it performed OK, whereas the other server has 7.2 + the latest updates and it performed problematically. The differences being the packages updated since the 7.2 CD was released. Hmmmm !

Do the uninstall & reinstall mentioned earlier, working around dependency issues that may arise.
I think you can use
rpm -e --force smeserver-spamassassin

I've lost contact with my server as my ISP is having a partial outage, so can't check.

Did you ever receive a message about importing the GPG key as suggested by Mudra & did you run the suggested command ?
Did you run sa-update as suggested earlier ?

Both the above seem to be issues that can cause the problem you are experiencing.

Look thoroughly in your /var/log/qpsmtpd/current log file for evidence that plugins are, or are not, working, and that messages detected as spam are being, or are not being, rejected. Follow through all the events associated with a message ID number.

[raem]

brianr

smeserver-spamassassin-1.4.0-14.el4.sme
spamassassin-3.2.3-44.el4

They were checked and reported as being the same earlier in this thread.

[dave simmons]

Hi Brian,

Sorry, didn't mean to offend!

Result is:

spamassassin-3.2.3-44.el4
smeserver-spamassassin-1.4.0-14.el4.sme
smeserver-spamassassin-features-0.0.2-0

I think this last one came in because of something I installed after reading another post on this forum.  By this I mean that it can't be the cause of the original problem because I only started tinkering after the spam filter stopped working.

[dave simmons]

In answer to Ray,

Yes - I have run all commands suggested in these posts, including the GPG.KEY import.

Will try again to remove spamassassin.

Have looked in the log file (via server-manager).  It seems that maybe some mails are being filtered out (no mails are arriving in junkmail folders- maybe they're just being thrown away).  I don't really understand what it says (I can post bits of it if you like).  I am seeing a 4-figure number in the 3rd column.  There are maybe 50 entries with the same number.  Is this referring to different operations on the same message?  The numbers are rising, but not consecutive - there are gaps in between.

[raem]

dave simmons

error: Failed dependencies:
         e-smith-spamassassin is needed by (installed) smeserver-support-1.6.0-39.el4.sme.noarch

Try
rpm -e --nodeps smeserver-spamassassin

[dave simmons]

OK Ray,

Used your command, and it worked.

I then continued the removal procedure as described in an earlier post, but:

The command rpm -e spamassassin then produces:

error: Failed dependencies:
         spamassassin >= 3.1.8 is needed by (installed) FuzzyOcr-3.5.1-3.el4.sme.noarch

should I try the nodeps option again, or are we just making the hole deeper?

I'm interested to keep trying, because I feel I'm learning something!

BTW.  I made a reply to your previous post about the log file, but you might not have seen it - it came on page 3 of this thread

[dave simmons]

Latest news - there are 3 spam mails in the junk mails box, so the spam filter is doing something.

It's just not spam scoring very well, because I've also received another copy of the classic 'Hello I'm tired today...' email which sa has caught in the past, but now they get through.

It seems therefore to be a question of spam scoring?

[brianr]

you can "try out" SA by piping the msg file through spamc, as follows

spamc < ~user/Maildir/cur/emailfilename

you have to "find" the email file first though (use mc search facility?).

See if spamc in command mode gives the same result, especially useful to try one that "ought" to be higher scored.

Just a thought

B.   

[CharlieBrady]

dave simmons

Try
rpm -e --nodeps smeserver-spamassassin

Ray, please don't try to diagnose problems in the forum. Direct people to the bug tracker, and help to diagnose issues in the bug tracker.

If "spamfilter seems to have stopped working", then clearly it's possible that there's a bug which needs to be fixed. Bug Tracker please.

Dave and newhopenet, if you even suspect that spam filtering is not working correctly, then please open a bug in the bug tracker, and report there exactly what you've seen which makes you have suspicions.

Thanks.

[dave simmons]

If you're not part of the solution you're part of the problem.

Sound familiar Charlie Brady?

Everyone else who answered has been so helpful.  I don't know if it's a bug - it seems weird that so very few people are experiencing it if its a bug.  Or maybe less people are using SME than I thought? 

Don't get me wrong, I'm very happy with SME.  I've used it for probably 5 years or more now, and it's the first time it's done anything weird.  We didn't used to have spam filtering - it's funny how you only miss something when it's gone! 

Ray, Brian and any others, if you want to continue advising me, I'll follow.  I've got a working SME machine (my old server) running now, so I can experiment further with the problem machine, and I'm grateful for what I've already learned from you guys.

 I assume that we can continue posting here, as long as we don't break any rules over decency, swearing etc.

[Mjohnson]

Dave,

f you're not part of the solution you're part of the problem.

Sound familiar Charlie Brady?

Charlie used the word "please" three times and "thanks" once.  That was by far the nicest response I have ever seen from him.   I think we owe him a tip-of-the-hat at the very least.

Dave and newhopenet, if you even suspect that spam filtering is not working correctly, then please open a bug in the bug tracker, and report there exactly what you've seen which makes you have suspicions.

Charlie and the moderators have been diligent, consistent and patient with their bug-tracker nudges.  From my experience, the response and advice doled out via the bug tracker is even better than the "warm 'n fuzzies" you received here on the forum.  Give it a shot...these guys are really good.

MJ

[byte]

I assume that we can continue posting here, as long as we don't break any rules over decency, swearing etc.

No please open a bug as per Charlie's comment, this way the issue gets proper attention. Thanks.

[newhopenet]

OK -- will open a bug for such things in the future. 

However, my particular issue has been resolved. 

A friend of mine, who knows nothing about SME but has been a linux admin for several years told me to do this:

signal-event post-upgrade
signal-event reboot

I know I already did that after I upgraded.  I really did.  But, for whatever reason, in my particular case, doing that resolved my spamassassin problem -- all our mail is now being properly scored, tagged, and sorted again.  Weird. 

[CharlieBrady]

If you're not part of the solution ...

I guess SME server software just writes itself ...

Anyway, if *you* want to be part of the solution, please post details about your problem to the bug tracker.

If you just want your problem solved, and to hell with everyone else, then continue to waste time here - and leave the problem unsolved for everyone else.

[byte]

I know I already did that after I upgraded.  I really did.  But, for whatever reason, in my particular case, doing that resolved my spamassassin problem -- all our mail is now being properly scored, tagged, and sorted again.  Weird.

That's where a bug could be, depending on how you upgraded.

[dave simmons]

And if I open a bug when it's just something on my machine, all the tech-nerds will call me an idiot.

No thanks, I think I'll just scratch the machine, re-install, but just copy over the web pages and recreate the users, instead of restoring from backup.

Thanks again to the guys who really tried to help.

Bye.

[byte]

And if I open a bug when it's just something on my machine, all the tech-nerds will call me an idiot.

I've not seen no-one called an idiot, care to show evidence of such ?!

No thanks, I think I'll just scratch the machine, re-install, but just copy over the web pages and recreate the users, instead of restoring from backup.

Quite a lack attitude there...it's only the end users who can make _your_ open source project more stable than it already is!

Thanks again to the guys who really tried to help.

And don't forget to thank the people are gave you the software in the first place

[newhopenet]

For anyone else who's been following this thread -- I've filed my particular issue as bug report 3350.  http://bugs.contribs.org/show_bug.cgi?id=3350

I noticed another existing bug report about a somewhat similar issue that was raised in this thread concerning a warning about no public key when upgrading from 7.1.3 to 7.2 -- This is an issue I experienced, and resolved it as described in the bug tracker report. That bug report is here:  http://bugs.contribs.org/show_bug.cgi?id=3191

[thallanor]

I am experiencing the same problem here.  spamassassin worked great and then suddenly started marking spam with unusually low tags.  I created a bug report and hope that I filled out everything appropriately, as it is my first submission.

[Mjohnson]

And if I open a bug when it's just something on my machine, all the tech-nerds will call me an idiot.

That's not true.  I know because I AM an IDIOT and this is the only place they have been kind enough to not point that out!!!

[dave simmons]

I'm also an idiot, which is why I preferred to think that it was just a problem on my machine.

I have had bad experiences posting on other forums (notably Mandrake/Mandriva, which I no longer use), where some people took great delight in ridiculing requests for help.  I apologise to members of this forum if they have been offended by my comments.

I didn't think I know enough about the system to make a sensible contribution on the bug forum.

I have however, spent enough time fiddling with the system, to the point that it is probably easier (and safer) to do a compete reinstall.

[CharlieBrady]

I didn't think I know enough about the system to make a sensible contribution on the bug forum.

Only you know what you've seen with your eyes on your system. And that's all you need to know to report a problem via the bug tracker.

I have however, spent enough time fiddling with the system, to the point that it is probably easier (and safer) to do a compete reinstall.

That's how you "maintain" a Windows system. That's not the way to fix problems with linux systems. To fix, we first need to diagnose. That requires the evidence which you would destroy by re-installing.

[dave simmons]

OK, but what do I need to do?  I can't give any more descriptions than I already have.  I've looked at the bug tracker and it seems much less user-friendly than this forum.

I don't mind working together if it also helps others, and I've got the time (not too mobile after an accident).

Can you take the info I've already given, or say what extra I need to give.  I can run on the old server for a while.

Maybe you want to come take a look.  I've got some spare public ip addresses, if you want I can set the machine up on one of the free addresses.  You'll need to tell me what I have to do - ports open etc.  It's late here now (22.30) .  I don't know where you are in the world, but if you would like to do this I would prefer to do it tomorrow if that is convenient.

[CharlieBrady]

Can you take the info I've already given ...

Please provide all details via the Bug Tracker. Most developers don't read the forums (and I'd rather not too, as its such a waste of time, but people will insist on reporting problems in the forums, despite "Don't report problems here ..." in bright red letters).

[dave simmons]

Please provide all details via the Bug Tracker. Most developers don't read the forums (and I'd rather not too, as its such a waste of time......

Oops, that was maybe also a bit offensive?

I was still writing my last post - so I'll restate it - I can put the problem machine on a free public ip - if you would like to take a look for yourself.  I'm sure with your experience you'll find the problem very quickly.  You'll need to tell me if (and how) to open any ports you need.

[chris burnat]

I was still writing my last post - so I'll restate it - I can put the problem machine on a free public ip - if you would like to take a look for yourself.  I'm sure with your experience you'll find the problem very quickly.  You'll need to tell me if (and how) to open any ports you need.

Dave, please report your issue on the Bug Tracker.  In this way, the information you provide will be available to the ALL of the development team for consideration.  As soon as you have posted your report, a copy will be automatically forwarded to each member of the dev team, and so will be each reply to your post.  In this way, the investigative process is streamlined, removing the need to log onto the forum to access latest developments, etc.  This is by far the most efficient way to deal with potential bugs - the average traffic on Bugzilla can be anything from 20 to 100 items per day....  Besides, all findings are then on the records, again in a centralised location, making it easy to search at a later date, link similar topics, etc.
Thank you
chris

[raem]

dave simmons & others

there are 3 spam mails in the junk mails box, so the spam filter is doing something.
It's just not spam scoring very well...It seems therefore to be a question of spam scoring?

Having done all that has been done and still having that result is a reasonable indication that something is not right and not a case of operator error or similar, so please report it in the bug tracker.
Additionally there have been 3 reports (at least) of this behaviour.
I personally have not experienced it, so wasn't initially convinced it was a real bug as such, and as there were a few "user level" resolutions that had been presented in the forums, my inclination was to give troubleshooting tips.

Regarding the issue of trying to resolve problems in the forums or going straight to the bug tracker, this has been mentioned before and opinions vary slightly.

Of course "if something in sme just doesn't work", then that's a problem of some sort, and the bug tracker is the official place to start the resolution process. It might end up being determined to be operator error, or localised system changes that will be resolved as invalid or it might go on to be found to be a real bug in the software that needs fixing, and does get fixed.

There was some discusssion/suggestion that potential bugs could be filtered in the forums where users felt they wanted to do so, and if no resolution was forthcoming then send the reporter to the bug tracker. The forums would act as a bugzilla pre filter so to speak.

I wasn't following the above theory as such, I was just OK about contributing to this particular topic to see if there was an early resolution possible that was not actually a bug.

At the time Charlie made his request I was also going to say it's time to send this to the bug tracker as anything obvious has been "tested for" and not found to be an issue or found to resolve the problem.

[CharlieBrady]

There was some discusssion/suggestion that potential bugs could be filtered in the forums where users felt they wanted to do so, and if no resolution was forthcoming then send the reporter to the bug tracker. The forums would act as a bugzilla pre filter so to speak.

Those suggestions were vigourously resisted by all of us who actually track down and fix bugs. Please co-operate with us and send all "issues" into the Bug Tracker. And please come over into the Bug Tracker and help to investigate problems there.

[judgej]

Ray, please don't try to diagnose problems in the forum. Direct people to the bug tracker, and help to diagnose issues in the bug tracker.

This thread is great!

It is answering lots of questions I have, due to three SME servers having exactly the same problem - very bad spam scoring that started just a few weeks ago. Some of us non-delelopers *don't* read bug trackers at bedtime. This is the other half of the community talking Mr. Brady, helping each other

-- JJ

[dmajwool]

Check out bug 3351 for some help that may fix this problem for you..
HTH, David.

[raem]

The suggestion by Shad Lords from http://bugs.contribs.org/show_bug.cgi?id=3351
that appears to resolve the issue (temporarily) is this:

I'd be interested to see if a simple:
sv t /service/spamd
on a broken system fixes the issue.  If it does then we just need to figure out
the best way to restart the daemon after a rules update.

and if that does not resolve the problem then:

- Take a message and send it through spamassassin and post the output along
with the time taken:

time spamc -R < ~user/Maildir/cur/message

- Then try ensuring you are fully updated and restart spamd:

/etc/cron.daily/sa_update
sv t /service/spamd
sv status /service/spamd

- And finally send the same message above through again and see if the score
changes:

time spamc -R < ~user/Maildir/cur/message


Would anyone who tries the above please update the bugtracker with your results.
http://bugs.contribs.org/show_bug.cgi?id=3351

The above steps will only fix the issue this time, and until the cause of the problem is fixed it may occur again with the next sa-update, so feedback to the bugtracker is important if you want a permanent fix.

[raem]

Charlie

Please co-operate with us and send all "issues" into the Bug Tracker. And please come over into the Bug Tracker and help to investigate problems there.

Yes Charlie, not a problem.
I realise/understand that making changes to a system in order to fix a problem does remove evidence needed to analyse & fix issues.
Your experience & knowledge puts you in a better position to more quickly judge what is likely to be a bug & what is not, and therefore when to give, & not give, troubleshooting advice in forums, versus doing so in the bugtracker.
If only we were all as knowledgeable & "on the ball" as you !

[swissis]

Same problem here.

I have read bugtracer 3304, 3350 and 3351 and can't make up whether it's solved or not and what to do.

Marcel

[raem]

swissis

Read my post two posts earlier, it's not yet permanently resolved, but there is a workaround suggested for immediate problems people are experiencing.
Try it and give feedback to the bug mentioned.
/etc/cron.daily/sa_update
sv t /service/spamd
sv status /service/spamd

[dave simmons]

Finally got it working again!!

Followed over to the bug forum, saw that it was already reported.

Unfortunately, the config on the server was pretty much screwed, so I couldn't get it going following the advice on the bug forum.  Did a fresh install and it works.

One question - if any of the technical guys are still following this link - I have been following the log of spamassassin, and I see that there are spams coming supposedly from our domain.  I can see they're not really coming from our domain (unknown IP address and .ru domain extension).  My question is the following - is our domain name in danger of being blacklisted because of this?

I realise that this is not really to do with SME but I am hoping that someone will still be able to answer. I have looked at spamcop, but there I can only search on IP address (our IP addresses are OK - not listed).

Thanks in advance!

Dave

[raem]

dave

...I see that there are spams coming supposedly from our domain.  I can see they're not really coming from our domain (unknown IP address and .ru domain extension). ....is our domain name in danger of being blacklisted because of this?

Spammers do fake the domain they send from, but the IP is usually real, so spammers who use your domain as their return address won't generally cause your domain to get listed, except perhaps by some overzealous list maintainers.

I believe that it is possible (but more difficult) to fake the IP they send from, so it is possible (although not so common) to get your static IP listed due to spammers activities.
Certainly spammers do use dynamic IPs, to attack & run, which usually results in temporary listing of the IP until it gets delisted. This can cause an inconvenience for users of dynamic IP accounts & it really depends how interested/active ISPs are to stop spammers activities & fix these sort of problems. Many dynamc IPs are listed automatically these days, necessitating the need for sme users with dynamic IP's, to send email via the ISP's smtp server.

The more likely way that your static IP will get listed is due to overzealous list maintainers, who can list your static IP for a variety of reasons. There have been numerous situations where good quality users who do not send spam get listed on "poor quality " lists (eg five-ten).
Spamcop blocks a lot of the big name free email accounts.
So for this reason choose the lists you use carefully, to avoid blocking senders you really want to receive from.

If you do get listed, then you need to request a delisting with that particular list maintainer, which can sometimes take a while. The workaround for this situation is to send your mail via your ISPs smtp server, even if you have a static IP.

Read my old howto, see the section Real Time Blacklist or Blocklist (RBL) Information
http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/rmitchell/smeserver/howto/Spam%20blocking%20HOWTO%20using%20qpsmtpd%20&%20RBL%20for%20sme%20server.htm

[drurydj]

Dave did you get this resolved?  I also had key mismatch.   Updated key and sa-update.

Running SME 7.2 upgraded along the way. 

I have same type of issue..

X-Virus-Checked: Checked by ClamAV on hvacc.net
X-Spam-Status: No, hits=1.4 required=4.0
   tests=SARE_ADULT2

<content removed>

alia harel

[drurydj]

Sorry.  Don't know how to read form (ack).

sv t /service/spamd fixed issue..

Dan

[dave simmons]

I eventually got it resolved by completely reinstalling the machine, and recreating users and re-transferring the website.  I have learned that reinstalling from backup also brings configuration files over, so I did not do that.  For me it wasn't much work - one domain and 6 users.  Of course, if there are many domains/users this could be a nightmare.

The solutions in this post, and also in bug report 3351 did not work for me. This was probably because I was also trying other things I found by searching the net, and that by trying other things I made the problem worse.

A clean reinstall (7.2 CD downloaded , followed by applying all the updates automatically via the server-manager) did not solve the problem - as soon as I created the domain and users, the spam started coming in again.  What I had to do was apply the sa update and import the GPG KEY manually (i.e. via a console session - not via the server-manager web interface).  I HAD activated the spam filter in the server-manager!

Only after I did the sa update manually did the machine start identifying spam correctly.  This means that a freshly-installed SME machine with updates will not filter spam without manual intervention.

***Should I report this as a bug?****

I would also be interested to hear from other users who have had this experience - whether they have been able to sort it out by following the solution either here or in the bug report.

I have a friend who is waiting to apply updates to his server, because the problem only started after we applied the last updates.  Obviously not applying updates is also not a good idea, but he has many more users & domains, and was having a massive problem with spam before moving to SME.

Ray Mitchell - if you're still following - what do you suggest regarding applying the updates?  I see a lot of technical discussion in the bug report, but no definite resolution.  

[raem]

dave

what do you suggest regarding applying the updates?  I see a lot of technical discussion in the bug report, but no definite resolution.

Read my earlier posts Reply #71 & #74, I think it's pretty clear what to do if you read those.
Of course you should apply updates, they are released to fix issues.
The fix looks to be one of either manually doing the combination of an sa_update & restart spamd, or restart spamd only, and the best way of achieving this is being looked at.

The issue appears to be identified although not yet conclusively, that's why you need to report any further observations or experiences to the bug tracker. Your feedback is important to help resolve the bug.
Please do so, and stop reporting the issue here, the developers are the people who fix the bugs and they are not reading these forum posts, but they are reading the bug reports.

Is that understandable ?

[raem]

dave

A clean reinstall (7.2 CD)... I HAD activated the spam filter.... Only after I did the sa update manually did the machine start identifying spam correctly....

The following comments are seperate to the bug issue being discussed here.
You make no mention of enabling DNS/RBL lists on your freshly installed sme7.2
If you don't enable RBL's & you have enabled the spam filter, then your server will be quite busy processing all the spam emails using spamassassin.
If you enable RBL's then many of those spam emails will be rejected by the smtp server ie your system won't accept them, thus giving spamassassin & your processor & memory a lot less work to do.
Enabling conservative RBL lists is highly recommended. I'm sure the instructions are in the FAQ or manual.

[chris burnat]

"Enabling conservative RBL lists is highly recommended. I'm sure the instructions are in the FAQ or manual."

Beware:
Conservative RBL are documented in the email section of the FAQ:
http://wiki.contribs.org/Email#Real-time_Blackhole_List_.28RBL.29

A more comprehensive list can be found at:
http://wiki.contribs.org/Updating_to_SME_7.2#RHSBL_Servers
They are not as conservative, and may not be what you want (as I have found out recently)...

[dave simmons]

Ray

Should I start a new bug report or try to add to the existing bug?

It may be part of the same problem or it may be different.  I don't have enough techical knowledge to know.

P.S.  Should my login for this forum work for the bug report or do I have to also register?  (I've tried with my login and pw for this forum and it didn't seem to work).  I received a mail about reactivation yesterday, and I thought that I had followed the instructions - I can post here .

Sorry if these seem to be stupid questions.

FYI - I don't think I've enabled the DNS/RBL lists, but I'm happy that everythings working.

[raem]

Enabling conservative RBL lists is highly recommended

By conservative I mean
do not enable RHSBL lists, and only enable
DNSBL lists and use zen.spamhaus.org and whois.rfc-ignorant.org

Anything more than that and you are likely to reject mail from legitimate senders.
Read my original howto for background which is a little out of date now with regard to lists ie some are defunct.
http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/rmitchell/smeserver/howto/Spam%20blocking%20HOWTO%20using%20qpsmtpd%20&%20RBL%20for%20sme%20server.html

config show qpsmtpd

qpsmtpd=service
    Bcc=disabled
    BccMode=cc
    BccUser=maillog
    DNSBL=enabled
    Instances=5
    LogLevel=8
    MaxScannerSize=55000000
    RBLList=zen.spamhaus.org:whois.rfc-ignorant.org
    RHSBL=disabled
    RequireResolvableFromHost=yes
    SBLList=dsn.rfc-ignorant.org
    access=public
    status=enabled

[raem]

dave

Add your comments/feedback to the existing bug.
The developers will advise if it is a different bug and suggest that a new bug be opened, if needed.
We are talking about commenting on the same bug referred to in the same bug report so I don't know why you would think to create a new bug report anyway.

Bugzilla has a seperate registration & login requirement, you will need to register a new account with loginame=your email address.

I don't think I've enabled the DNS/RBL lists, but I'm happy that everythings working.

You should enable the ones I have suggested as a minimum, you will be amazed at the reduction in spam that results ie 75 - 95% reduction in practice. See the FAQ or my old howto for the commands to use.