Topic: Cron ClamAV on Mailboxes

[Smitro]

Hi all,

I've noticed that when Clam AV does its full system scans it picks up viruses in emails. They are mainly in the junkmail folder, but some are in the inbox. I know that email scanning is working as the spam stats contrib tells me which ones have been killed. So I'm guessing that these emails maybe missed as they are not yet included in the virus definitions. I'm guessing a day or 2 later, they are included and then the virus checker picks it up and kills it.

Since working this out I have set clam av do run a full system scan every day. This seems a little over kill to what use to be a weekly task. Is it possible to just run a daily scan on mailboxes? If so, how do I do this?

Is there any other recommendations to make sure that these viruses are killed faster?

[CharlieBrady]

So I'm guessing that these emails maybe missed as they are not yet included in the virus definitions. I'm guessing a day or 2 later, they are included and then the virus checker picks it up and kills it.

That would be my guess as well.

Since working this out I have set clam av do run a full system scan every day. This seems a little over kill to what use to be a weekly task. Is it possible to just run a daily scan on mailboxes? If so, how do I do this?

There's currently no provision in the software to do that.

[Stefano]

Code: [Select]

cd /home/e-smith/files/users
clamscan -r --include=./*/Maildir/

you can run it in cron.d.. be careful not to start a "manual" scan during weekly scan

HTH

[purvis]

I have been getting viruses notices too from email received on weekly scans also.
This has been happening for a while now. As far as i can remember, before SME 8 on my systems.
We have  a low volume of emails by most people and this has not been a problem but on 1 system.

I have now turned on the virus scans to do it once a day to monitor the activity and learn.
We do not have viruses quarantined or deleted on the normal full scan you can set in the server managerl panel.

I have seen files that do not contain viruses flagged as viruses(false positives) on the SME server.
Some program that i even wrote showed up as a viruses, because of the way they where accessing the hard drive.
Most where MSDOS 16-bit programs that I wrote and had some low level assembly for fast execution.

But i am now seeing more and more received emails flagged as being viruses with the dedicated virus scans and getting through the email server software.

So in an effort to do  something. I am now watching closely the emails.
I have the server setup to store(save) all email coming and going to server into the maillog account.
In the server manager panel, i have all received email that has been tagged as spam to have the word "[SPAM]" placed in the subject line with the default settings inside the server manger panel.

Most viruses, but not all,  I have seen have the [SPAM] in the subject line.

I do have a program written in windows that will attach to the server using POP3 and delete all email in an account with the word [SPAM] in the subject line.
Deleted emails are logged into files for viewing. Because all emails are also sent to the maillog email account, that is my backup in the case I delete a email message that was not actually spam.

But that does not remove the fact that there are emails that seem to get through and could possible be read on a system that can get infected.
We do things different here, we have computers just to for reading email and browsing the internet and those are not on our local LAN with workstations.
But we use other software, that is close to virus proof, to view the text and headers of received emails on the workstations. Any email can be safely deleted there and then, without any harm from some other email reader software that can be dangerous with an infected email.

here is the header of an email that was NOT marked as a virus when it was received but a dedicated scan detected it.
email addresses and server names have been edit for security measures
The email was received to the email account servicedesk@myserver.com
We have all emails received to servicedesk@myserver.com forwarded to johndoe@myserver.com(sorry for this extra confusion but i wanted to give and actual email header)

Code: [Select]

Return-Path: <silk2013@rfast.com>
Delivered-To: johndoe@server2.myserver.com
Received: (qmail 4603 invoked by alias); 20 Feb 2013 14:36:00 -0000
Delivered-To: alias-localdelivery-johndoe@myserver.com
Received: (qmail 4600 invoked by uid 5014); 20 Feb 2013 14:36:00 -0000
Delivered-To: servicedesk@server2.myserver.com
Received: (qmail 4594 invoked by alias); 20 Feb 2013 14:36:00 -0000
Delivered-To: alias-localdelivery-servicedesk@myserver.com
Received: (qmail 4589 invoked by uid 453); 20 Feb 2013 14:35:59 -0000
X-Spam-Level: *
X-Spam-Status: No, hits=0.0 required=5.0
tests=FSL_HELO_NON_FQDN_1,HTML_MESSAGE
X-Spam-Check-By: myserver.com
Received: from Unknown (HELO [125.143.21.236]) (125.143.21.236)
    by myserver.com (qpsmtpd/0.84) with ESMTP; Wed, 20 Feb 2013 08:35:53 -0600
Received: from (192.168.1.186) by rfast.com (125.143.21.236) with Microsoft SMTP Server id 8.0.685.24; Wed, 20 Feb 2013 23:35:50 +0900
Message-ID: <5124DAA4.408090@rfast.com>
Date: Wed, 20 Feb 2013 23:35:50 +0900
From: "cashproonline_notification@gcibemail.bankofamerica.com" <cashproonline_notification@gcibemail.bankofamerica.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.9) Gecko/20100921 Thunderbird/3.1.4
MIME-Version: 1.0
To: <servicedesk@myserver.com>
Subject: Your CashPro Online Digital Certificate
Content-Type: multipart/alternative;
 boundary="------------07020700104050901050607"
X-Virus-Checked: Checked by ClamAV on myserver.com


I have no idea whether this particular received email had an actual viruses or not. I do no plan on testing it to completeness.
But it does show up with daily or weekly clam scan.
Also, with the below

Code: [Select]

clamscan --infected -r /home/e-smith/files/users/
results from above command:
/home/e-smith/files/users/johndoe/Maildir/cur/1361370960.4605.server2:2,: Win.Trojan.Agent-200678 FOUND


[purvis]

We use only Ibay areas and not user areas.

I setup a hourly cron to to scan the directory of /home/e-smith/files/users/ and log those results into a file nameed /tmp/virusscan.
I am just using the programs and routines for monitoring now.
I will test Stefano command lines rather than the one i am using and hopefully it will only scan emails.

Code: [Select]

clamscan --infected -r /home/e-smith/files/users/
[\code]

In the end, what really bugs me is why does the program not detect the viruses when it is received.
And if it is the same program doing the virus testing, it is confusing and really unacceptable.
I do not want to have a virus scanner to be constantly scanning for viruses in all emails using some cron routine every few minutes either.


[purvis]

Stefano

you can run it in cron.d.. be careful not to start a "manual" scan during weekly scan

Is this dangerous to do or just for some other reason

Code: [Select]

cd /home/e-smith/files/users
clamscan -r --include=./*/Maildir/

I ask because i would like to  scan for viruses in emails often by many different means.
I worked on a program to read emails and save them in files on workstations.
The program is not complete as of yet and i am scared of viruses and such.
I never mind having a program to do more work if that is what it takes.

[purvis]

I came up with this to place in the directory /etc/cron.hourly
This will log all viruses scans and removals to a file called /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
The log file can be displayed with a browser. The information placed in the log file is about as small as i could get it.
create file

Code: [Select]

cd /etc/cron.hourly
nano cleanemailswithviruses

code in file

Code: [Select]

#!/bin/bash

mkdir -p /home/e-smith/files/ibays/Primary/html/serverstatus
chmod 755 /home/e-smith/files/ibays/Primary/html/serverstatus

cd /home/e-smith/files/users

TODAY=$(date +"%Y-%m-%d %T")
echo "$TODAY started of scanning for viruses in emails"  >>/home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/new/  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
TODAY=$(date +"%Y-%m-%d %T")
echo "$TODAY finished scan of /home/e-smith/files/users/*/Maildir/new/"  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/cur/  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
TODAY=$(date +"%Y-%m-%d %T")
echo "$TODAY finished scan of /home/e-smith/files/users/*/Maildir/cur/"  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
TODAY=$(date +"%Y-%m-%d %T")
echo "$TODAY finished scan of /home/e-smith/files/users/*/Maildir/"  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
echo "$TODAY end of scanning for viruses in emails"  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
echo '------------------------------------------------------------------'  >> /home/e-smith/files/ibays/Primary/html/serverstatus/virusscanemail.txt
exit 0

exit and save the file with  ctrl-x

set the file to execute
then execute the program for testing

Code: [Select]

cd /etc/cron.hourly
chmod 755 cleanemailswithviruses
./cleanemailswithviruses


This does not scan all the email messages for the first two passes for a reason.
I shorted it to help in speeding up the most important files first(the new and current emails)

The managing of the logged file is up to the those others for now.
Maybe a php script placed into the same directory can erase the file or rotate it.

[purvis]

I rewrote the previous cron script to compact the code and create a smaller log file and shorter lines in length for viewing with a mobile device such as an iphone.
Of course this is a work around for our location until a better solution is created.

I am not experienced in bash or linux programming but this seems to work and i need to get better acquainted with the anti-virus clamscan program.

I am seriously thinking a better bash routine to check for viruses would be better for our servers than what is offered in the generic manager's panel.
A better bash routine for checking for files with viruses might be a continuous loop and have blocks of code for running various virus checks.
That way, I could have a custom scan that maybe more efficient for our work periods and server file use and then I could disable the weekly or daily virus scan so that those would not conflict with custom virus scan being run from a bash routine that has looping code.

But for now here is a small rewrite. Any other rewrites that i will share will be in the contribs sections of the forum and not here.

Code: [Select]

#!/bin/bash

locationoflogfile="/home/e-smith/files/ibays/Primary/html/serverstatus"
logfilename="virusscanemail.txt"
logit="$locationoflogfile/$logfilename"

mkdir -p $locationoflogfile
chmod 755 $locationoflogfile

cd /home/e-smith/files/users
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY started scanning"  >> $logit
clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/new/  >> $logit
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY scanned new"  >> $logit
clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/cur/  >> $logit
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY scanned current"  >> $logit
clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/  >> $logit
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY scanned all" >> $logit
echo "$TODAY end of scanning"  >> $logit
echo '----------------------------------'  >> $logit
exit 0
[code]
 

[purvis]

remove post by user

[purvis]

I found out that clamscan processes will slow the server down.
In looking for options to reduce the cpu overhead and priority of the clamscan program so that other programs have a higher disk priority.
I found that the linux programs renice, nice, ionice  may be a solution and i am going to post the rewrite of the routine above again to what i have running.

The most important change i have made was to add the line below in front of the program clamscan.
/bin/nice -n 20 /usr/bin/ionice -c3 -n7

I have also place in the routine, 2 sleep commands, to allow for other processes to have additonal cpu time.

Code: [Select]

#!/bin/bash

/usr/bin/renice 20 $$

locationoflogfile="/home/e-smith/files/ibays/Primary/html/serverstatus"
logfilename="clamscanemails.txt"
logit="$locationoflogfile/$logfilename"

mkdir -p $locationoflogfile
chmod 755 $locationoflogfile

cd /home/e-smith/files/users
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY started scanning"  >> $logit
/bin/nice -n 20 /usr/bin/ionice -c3 -n7 /usr/bin/clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/new/  >> $logit
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY scanned new"  >> $logit
sleep 10
/bin/nice -n 20 /usr/bin/ionice -c3 -n7 /usr/bin/clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/cur/  >> $logit
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY scanned current"  >> $logit
sleep 10
/bin/nice -n 20 /usr/bin/ionice -c3 -n7 /usr/bin/clamscan --no-summary --remove=yes --infected -r --include=./*/Maildir/  >> $logit
TODAY=$(date +"%Y%m%d %T")
echo "$TODAY scanned all" >> $logit
echo "$TODAY end of scanning"  >> $logit
echo '----------------------------------'  >> $logit
exit 0


[purvis]

this line. placed in the above bash file routine located in the /etc/cron.hourly

Code: [Select]

/usr/bin/renice 20 $$
Causes a unnecessary email message to be sent to the admin email account when the routine is executed at the top of every hour.
either comment out the line  with a "#" sign like the below line. If there is another way for the cron.hourly bash routine to not send an email, I am not aware of it, but will search for one.

Code: [Select]

#/usr/bin/renice 20 $$
or just remove the line
 
After observing the routine running, I  do not believe the line is needed.
The line was placed there in a effort to reduce the priority the bash routine and also hopefully reduce the priority of the clamscan, being i thought it might be a child process to the bash.
I understand child processes use the same priority of the parent program.
The clamscan uses way to much of the computer resources in  CPU and INPUT/OUTPUT use and will very likely slow down a computer.
It would seem that how i am executing the clamscan program in the bash routine does make some concessions for clamscan to use less CPU AND IO resources.

Also if you do not want to log anything.
setting logit to /dev/null should give you that effect

Code: [Select]

logit="/dev/null"


[purvis]

After further testing, making the change in the line below will solve the problem of the email message being sent to the admin that was not wanted.

Code: [Select]

/usr/bin/renice 20 $$ > /dev/null


[purvis]

I have to wonder if the problem of email viruses getting past a virus scan at time of email delivery is because the current virus definitions are not loading.
Then later the clamscan identifies the email file as having a virus because it loaded up current virus definitions. 

This may or may not be a side effect if freshclam is not working proper or freshclam is working proper because of clamd is not working proper.

See other threads about freshclam.

[purvis]

from purvis above
I had wrote

I have been getting viruses notices too from email received on weekly scans also.
This has been happening for a while now. As far as i can remember, before SME 8 on my systems.
We have  a low volume of emails by most people and this has not been a problem but on 1 system.

After doing hourly scans of emails, erasing those emails during scans that Clamav identified as having viruses, and logging the virus scans on emails date and time.
Has led me see where the emails are passing thru the ClamAV  antivirus scans at the time the emails are received because the virus definitions of those particular viruses have not been updated and/or included in the signature cvd files of ClamAV.

The lag time of of detecting those emails seeems to be about 2 and 1/2 days,  or just a few more.

[Smitro]

Some great research here Purvis, I'm looking to implement this as soon as I have a chance. Good work!