Topic: possible looping problem with spam assassin (Swerts-Knudsen)

[cmwi96]

I am using the Swerts-Knudsen install scripts for both ClamAV and SpamAssassin. I recently upgraded from older versions. When I enable Spam filtering from the panel my system becomes slow or even unresponsive. top show MANY instances of Perl5.6.1 running by the user qmailq. Left unchecked this will eventually consume all the memory resources of the machine. If I switch off the Spam filter the available memory returns to normal levels. Additionally a message sent while the filter was on but finally delivered after it was turned off contained the following :
Received: (qmail 4832 invoked by alias); 18 May 2005 15:34:27 -0000
Delivered-To: alias-localdelivery-edit@watkinsbros.net
Received: (qmail 4829 invoked from network); 18 May 2005 15:34:26 -0000
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
Received: from localhost (127.0.0.1)
  by box2.watkinsbros.net (127.0.0.1) with ESMTP; 18 May 2005 15:31:11 -0000
 
This leads me to believe that the mail is being run through this process several times. A test message sent while the Spam filter was off yealds the expected:
Received: (qmail 5211 invoked by alias); 18 May 2005 15:38:08 -0000
Delivered-To: alias-localdelivery-edit@watkinsbros.net
Received: (qmail 5208 invoked from network); 18 May 2005 15:38:08 -0000
X-Virus-Scanned: by amavis-ng-0.1.6.4-03dc on box2.watkinsbros.net
Received: from localhost (127.0.0.1)

Can anyone give me any suggestions?

[nate]

I am experiencing the same kind of thing.  
Admin account filling up with double bounce messages
and looping galore!  

I’m not going to be too quick to place blame however;
this could be bad DNS MX, (no reverse or duplicates)
or other bad config.
 
The only data point I know for sure is the thing runs like a pig
with Spamassassin turned on and just fine with it off.
 
(Ver. 6.0.1 w/smeplus)

[Knuddi]

Are you guys using the fetchmail contrib or ezl(something)?

[nate]

Are you guys using the fetchmail contrib or ezl(something)?

 
This is the one:http://sme.swerts-knudsen.com/howtos/howto_29.htm
unless it's been updated?
 
smeplus.sh - all extras installed with no errors about a month ago.

[cmwi96]

I don't remember insalling fetchmail but it may have been part of another package. (was it ever part of the original clamAV or SA package?). in any case
 rpm -qa |grep fetch
fetchmail-5.9.0-21.7.3

I couldn't say what its doing though.

[cmwi96]

did you by any chance mean exmlm?
ezmlm-idx-std-0.40-2
ezmlm-web-2.1-06
e-smith-ezmlm-web-1.1.1-05

[Knuddi]

Yes this is the one - I just recall a problem in the past with the combination of SA (amavis) and ezlm. Cannot recall that it looped by that it at least something was wrong. Try a search in the forum and maybe someone else can remember ??

[cmwi96]

I did find a post that says the combination of these causes the Spam filter to stop working, though there was no mention of the kind of processes or memory problems. Also I had been using your scripts on th same install with no issues until I installed the spamfilter_install.sh this morning. I had never experienced a problem having both.

[cmwi96]

another bit of info the Perl process that seems to open in several intsances seems to itterations of the /usr/bin/suidperl -w command if this helps. I did some looking and it seems as if a problem with this could cause messages to be scanned multiple times (which is what this seems like). Thoughts?

[raem]

Check your qmail-queue settings, these are with Jespers antivirus & spam filter panels installed.

# /sbin/e-smith/db configuration show smtpfront-qmail
smtpfront-qmail=service
    ExternalInterfacesFilter=/usr/bin/qmail-queue.amavis
    Instances=40
    InternalInterfacesFilter=/usr/bin/qmail-queue.amavis
    Patterns=enabled
    PatternsFile=/var/qmail/control/patterns.default
    Proxy=enabled
    RBLList=whois.rfc-ignorant.org:dnsbl.sorbs.net:relays.ordb.org:sbl-xbl.spamhaus.org
    access=public
    status=enabled

# /sbin/e-smith/db configuration show amavis-ng
amavis-ng=service
    MailFrom=antivirus@xxxxxxx.xxx.xxx
    Notifiers=Admin
    qmail-queue=/var/qmail/bin/qmail-spamc
    status=enabled

if necessary either correct the entries manually or
maybe run the install script(s) again so "things" get configured correctly ?

[cmwi96]

All of these properties read as they should with  the exception of the qmail-queue property which reads /var/qmail/bin/qmail-queue. I believe this is because I currently have SpamAssassin disabled in order to keep the box from crashing. I do believe enableing SA from the pannel changes this property however I have tried changing it manually and it just causes SA to start working but not correctly as I still have the original issue.

[cmwi96]

New info:
I unintalled all the ezMLM RPMs and ran a post-upgrade event and rebooted. While there is no EZMLM now I still have the same SA issue.
Also in the running processes during the "hang-up" I see quite a bit of this.
qmailq    3263  3260  5 09:29 ?        00:00:01 /usr/bin/suidperl -w /dev/fd/4//
qmailq    3264  3263  0 09:29 ?        00:00:00 [spamc <defunct>]
qmailq    3268  3263  7 09:29 ?        00:00:01 /usr/bin/suidperl -w /dev/fd/5//
qmailq    3269  3268  0 09:29 ?        00:00:00 [spamc <defunct>]
qmailq    3271  3268  9 09:29 ?        00:00:01 /usr/bin/suidperl -w /dev/fd/6//
qmailq    3272  3271  0 09:29 ?        00:00:00 [spamc <defunct>]
qmailq    3274  3271 14 09:29 ?        00:00:01 /usr/bin/suidperl -w /dev/fd/7//
qmailq    3275  3274  0 09:29 ?        00:00:00 [spamc <defunct>]
qmailq    3277  3274 23 09:30 ?        00:00:01 /usr/bin/suidperl -w /dev/fd/8//
qmailq    3278  3277  0 09:30 ?        00:00:00 [spamc <defunct>]

as I said lots of perl instances.

is anyone else having this issue?

[ltc6netspec]

Are you running messagewall?  If so I had to force a reinstall of it.  It ran for 10 minutes until I reinstalled the profiles I have used for a year.  Then it went back to your problem.  I just uninstalled messagewall to see if the perl problem would go away.  it did not.  Where is the file for spamassassin RBLs and to make sure spamassassin is not enabled.  

Configuration:  removed New spamassassin/ reinstalled old spamassassin still same problem.  Installed new clamav still same problem

HELP - 900 users

[cmwi96]

I'm not running messagewall.

[ltc6netspec]

Should perl be reinstalled?

[ltc6netspec]

This is really 'spamc'.  

Use ps -ef | more

It will show that spamc may be defunct

look at

Email slipping through without being scanned by SA
 
in the forums.

Pico'ed /home/e-smith/configuration

should have

|qmail-queue|/var/qmail/bin/qmail-queue|

amavis-ng= line

May email server is working now

[raem]

cmwi96

> I do believe enableing SA from the pannel changes
> this property however I have tried changing it
> manually and it just causes SA to start working
> but not correctly....

How about you actually enable spamassassin using the panel and then see what the qmail-queue settings are. No point assuming that they are something.

[cmwi96]

It wasn't an assumption I actually did this
It does indeed change to qmail-spamc when Spam filter is activated through the pannel. Then its chaged back when its disabled.

[cmwi96]

This is really 'spamc'.  

Use ps -ef | more

It will show that spamc may be defunct

look at

Email slipping through without being scanned by SA
 
in the forums.

Pico'ed /home/e-smith/configuration

should have

|qmail-queue|/var/qmail/bin/qmail-queue|

amavis-ng= line

May email server is working now

I think all you are doing by changing from qmail-spamc to qmail-queue is bypassing the spam scanner. Passing mail to spamc is how it works. This is how I have my mail server running as well. But I dont think you SA is actually working. Do your mail headers have a spam score?

[ltc6netspec]

I had to remove spamassassin by using knudsen script.  this then let me see what was creating the PERL5.6.1 (top -i & ps -ef |more)

That is when we noticed (spamc)defunct

I do not know enough to say if the /var/qmail/bin/qmail-spamc file is corrupted.  But when I did have the new spamassassin installed & enabled that is when I started to get the PERL5.6.1 and running out of memory.

Now should I delete this file so that I can reinstall spamassassin?  That is the question?

[meanlocha]

I had a similar problem recently. It seemed
to be caused by a SPAM email with more than 20 recipients.
I dont have an output of "ps ax" right now
but I recall there were multiple instances
of /usr/bin/smtpfront-qmail and suidperl.
Since my server is a low volume email server,
I did the following:

/sbin/e-smith/db configuration setprop smtpfront-qmail Instances 5

/sbin/e-smith/expand-template /var/service/smtpfront-qmail/runenv

/etc/init.d/smtpfront-qmail restart


Anthony

[cmwi96]

I had a similar problem recently. It seemed
to be caused by a SPAM email with more than 20 recipients.
I dont have an output of "ps ax" right now
but I recall there were multiple instances
of /usr/bin/smtpfront-qmail and suidperl.
Since my server is a low volume email server,
I did the following:

/sbin/e-smith/db configuration setprop smtpfront-qmail Instances 5

/sbin/e-smith/expand-template /var/service/smtpfront-qmail/runenv

/etc/init.d/smtpfront-qmail restart


Anthony

This is an easy fix so I will give it a try though I'm not sure that its that same issue. The user I am seeing is qmailq and it happens even if the message is NOT spam and is form a known user.

[cmwi96]

Changing the number of instances in smtpfront-qmail does not fix the issue. Amavis is still processing the message multiple times and I still have instances of perl that seem to hang open intil I disable the spam filter altogether. Its definatly looping.

[raem]

cmwi96

> I recently upgraded from older versions....

So it seems that you have NEVER got the newer version of spamassassin to work correctly.
What were the older versions you were using eg rpm name & number ?
There may be a clash with remnants of the older rpms eg an inappropriate custom template and/or configuration ??

Perhaps you should uninstall all rpms associated with antivirus, spam filter and any earlier spamassassin versions. You may also need to manually delete any custom templates associated with earlier versions that should not be there now, and any db's in /home/e-smith that should not be there.
When your system is "clean", then do a reinstall.

What system spec do you have, processor, RAM etc ?
Low end stuff can very quickly suffer when using spamassassin.

You may also want to look at these suggestions, although I don't think they are your current problem, see
http://mirror.contribs.org/smeserver/contribs/rmitchell/smeserver/howto/Mail%20system%20tweaks%20HOWTO%20for%20sme%20server.htm

[cmwi96]

You are correct I NEVER have got the newest version to work.
I uninstalled all the spam filter related packages and re installed with the current script. I noticed that in your post you mentioned un/re innstalling the ClamAV stuff as well. Can you or someone geive me a list or RPMs or keywords to grep for in order to try and uninstall both packages?

[raem]

Look here for clues, although these are for newer versions.
http://www.pagefault.org/code/e-smith/antivirus/
and
http://sme.swerts-knudsen.com/downloads/AntiVirus/

[Brenno]

I'm starting to encounter this too... fully upgraded SpamAssassin and ClamAV packages running on SME 6.01.  Hardware is a dual P4 1Ghz with 1GB RAM.  Box runs fine for 20-some hours, then grinds to a halt with all mail being dumped in the problems directory.

A check of "top" during slowdown shows >96% CPU load on both CPUs.  If I kill spamassassin, CPU usage returns to normal and mail flows.  I can then restart SpamAssassin and get another 20 - 26 hours of trouble-free working.

I even remember receiving an admin email saying something about something being defunct - listed over and over.  Sorry to say I didn't keep the email for posting:(

Any thoughts as to why this machine never had problems before, but now suffers under this config?  We're only processing 500 - 600 email/day for about 42 users.

[raem]

Brenno

> I'm starting to encounter this too

Check all the settings in configuration db for smtpfront-qmail and amavis-ng and comfirm correct.
See this thread
http://forums.contribs.org/index.php?topic=27599.0

also try these suggestions
http://mirror.contribs.org/smeserver/contribs/rmitchell/smeserver/howto/Mail%20system%20tweaks%20HOWTO%20for%20sme%20server.htm