Topic: Email Filtering For Checking Attachments!!!

[byte]

Hey guys...

I have searched on here and found many ways to filter attachments and block them i.e checkhab, assp so on, I was just wondering what success people have had using them?

Thanks for any info!!

[mbachmann]

I am about to try out http://tech-geeks.org/contrib/loveless/bad-attach/bad-attach-0.2.tar. But until now nothing is done with the attachements.

[warren]

I've been using checkhab on 5.6U5 for the last 4-5 mnths without problems ; had to remove blocking of html mail though. All in all very satisfied with the workings of checkhab

Warren

[warren]

I've been using checkhab on 5.6U5 for the last 4-5 mnths without problems ; had to remove blocking of html mail though. All in all very satisfied with the workings of checkhab

Warren

[nate]

I’ve used checkhab for almost two years in a 60-user org.  Too many false positives from HTML formatted mail.  Mostly when they are composed and/or copy/pasted from various MSOffice apps.  My users screamed bloody murder so I have to allow HTML formatting.  I end up picking false positives out of the junk mail folder far too often.  I have done some tweaking to the config but nothing really helps.  I am finally going to dump checkhab for a less laborious solution (that I have yet to find.).  
 

 
 
(burp!)

[raem]

byte

There is an add on contrib for sme 6.0.x for pattern matching filtering that works very well
http://mirror.contribs.org/smeserver/contribs//rmitchell/smeserver/howto/Virus%20and%20file%20blocking%20HOWTO%20using%20smtpfront-qmail%20for%20sme%20server.htm


This has been incorporated into default v6.5 (just needs enabling) and also is part of default v7.0xxx & can be controlled & enabled using the server manager panel.

You can add your own file types as required (see howto) and block any type of file

[byte]

That post i put was 2 years ago and I did end up using that how-to as per Ray M and now the built in feature in 7.0pre1

[nate]

Thanks, I will give this a go this week and post my findings.
---
I am curious if the smtpfront pattern matching “solves” the problem of the false positives ???

[raem]

nate

> I am curious if the smtpfront pattern matching “solves” the problem of the false positives ???

By that I assume you are referring to:
"Too many false positives from HTML formatted mail......I end up picking false positives out of the junk mail folder far too often"

Pattern matching will by default reject email messages that have attachments with executable content (you can select the basic types of executable content to block).
If you want to block other file type content then you generate the file magic as per the howto and add that to the pattern database (as per howto).

If you exclude all html content then you will probably be in big trouble with your users.

Note that you can copy the patterns database from smev7.0 (which has been updated) and use that in v6.x for more effective blocking.
See a forun post a few months ago about this

So the way to manage all this is to use layers.

1) RBL rejection (configured in jespers Spam Filter or manually as per my howto)

2) Pattern matching rejection (as per howto or default in sme 6.5 & 7)

3) Spamassassin with a senstitive score setting like 4 (use jespers Spam Filter panel)

4) Antivirus filtering - clamav (use jespers antivirus panel)

5) other tweaks to reject spam type messages
eg Reject messages with bad HELO/EHLO entries
Control doublebounce messages
Control mail to invalid addresses & configure mail blocking rules

all as per this howto
http://mirror.contribs.org/smeserver/contribs//rmitchell/smeserver/howto/Mail%20system%20tweaks%20HOWTO%20for%20sme%20server.htm

I get virtually no junk mail into my Inbox as it all gets blocked or filtered to the junkmail folder.
I don't recall having a false positive in the junkmail folder for many months now as the current version of spammassassin is quite good.

You can use whitelist or blacklists in the Spam filter panel to help reduce any false positives or negatives particular to your system & your users.

Make sure you keep spamassassin Spam Filter & the antivirus panel updated whenever jesper issues a new release or update.

[nate]

Ray,
 
Thank you very much for taking the time to write this response.  I do use Spamassassin w/RBL and ClamAV (can’t live without them), as well as other tweaks like some custom procmail rules and other things.  My users get almost no Spam and I am very proud of the fact that my network has seen 100% uptime and has had zero viruses for the past four years!  -thanks to helpful folks like you in the open source community.
 
Originally, my reasoning for using checkhab was that it looked like a very straightforward system to allow/disallow specific file types, an on/off switch for attachments.  However, the script has flaws!

Banning attachments by file types is important for many reasons, virus control, as well as resource usage and user rules.  The number one system setting that keeps everything working everyday is, absolutely NO system modification by users on their workstations.  If you can’t install anything – you can’t break the thing!  -- I had to wade through a lot of griping and whining, but I got them trained!
 
Pattern matching using a database still seems a bit “clunky” …but if it’s the best deal for now that’s what I’ll go with.  Soon I hope to explore the seemingly more elegant Greylisting approach….
 
I think you have pointed me in the right direction for the next step.  Now I just have to find the time to run through all this on my test server in case I blow something up in the processes of swapping checkhab for smtpfront.  All in a good day (and nights) work…
 
Cheers and thanks again,
Nate

[raem]

nate

Pattern matching is "well proven" technology, the developers have incorporated it into v6.5 and v7 as standard.

> Banning attachments by file types is important for many reasons....

Keep in mind that pattern matching does not examine the attached file suffix, it analyses the code in all attached files and if the beginning of that code matches a pattern (that has been selected), then the whole message is rejected.

The file may purport to be a bat file or a pif file or a mpg file but if it includes executable code then the message will be rejected (where exe has been selected) etc etc etc.

If you want to reject all mpg file attachments, then simply determine the "magic" for mpg and add that to the patterns database.
Note that a file with the suffix mpg may not in fact be rejected if it does not have the mpg magic code, as it is not really an mpg file but just pretending to be an mpg (and vice versa) ie something pretending to be a zip file but was actually a mpg will be rejected as it has the mpg magic.

Full details are in the howto, although if applying the concepts to v7.0 there may be some slight differences in the implementation.
Note in sme 7.0 that you just need to enable pattern matching in the server manager panel as it is already installed and fully configured.


>Pattern matching using a database still seems a bit “clunky”...

I don't think it's clunky for the reasons outlined above, you don't specify a file type, you specify a pattern type.
Whatever type of file that has that pattern will be "caught".
It's all encompassing rather than clunky.
There is no need for regular updates like clamav etc, but only ocassionally as new "all encompassing" patterns are utilised by spammers etc, which in practise has proven to be infrequently.


> Soon I hope to explore the seemingly more elegant Greylisting approach….
 
Well there is also a contrib developed by Gordon Rowell that does work quite effectively. During my tests absolutely zero spam and viruses were received.
Unfortunately some admin interaction is regularly required to build up a good whitelist & check for messages that were not delivered again (retried).
The problem is that some external mail systems do not retry again and others send from multiple IP's, which can seriously delay a message getting to your system.
Whitelists would take care of these issues though, the main problem is you have to keep watching log files to pick up these problems and add those senders to your whitelist.
If everybodies mail systems worked "correctly" greylisting would be essier to reliably use.
If your users regularly receive mail from the same or repeat sources, then greylisting will work very well & there will be no delays introduced either, as those senders are put on the auto generated "whitelist" which is maintained for 24 hours or so.

Search for some interesting posts in devinfo lists re greylisting.

[nate]

Executable content blocking:


I was not able to get smtpfront pattern matching (as per Ray Mitchell how to) working on first attempt.

I carefully worked through the how-to on my test machine.  After executing the final reboot, the new “menu box” in the server manager Email panel never appeared.  From my tests there was no indication that the system was working or that executable attachments were being rejected.

Notes:
 
* Test box = smeserver 6.0.1 with PLUS.  
* Installed all rpms from how-to except for perl-libnet because a higher number already installed on system.
* rpm –e e-smith-checkhab + reboot before I started the how to.  I don’t think this should have caused any problems??

[nate]

Success! ...on Second attempt.
 
I restored my test machine from an image and tried the how-to again.  This time I did not uninstall checkhab first and everything worked!
 
However, from a few tests it seems that the pattern matching does not catch everything executable!  I swapped the db with the mailpatterns db from smeserver 7pre1- noted a lot more patterns defined.  I sent a bunch of emails attaching several different .exe files from windows os.  Some files were stopped and some were let through.  The system did stop .zip files as advertised.
 
I think it works good enough to deploy on my production server.  I still need to find “the right way” or at least an elegant way to disable checkhab.  For now, I simply removed all entries from the ban files list.
 
My system is far from “perfect” but I think this system along with SA and ClAV will serve as adequate protection and greatly reduce false positives.

Nate

[raem]

nate

> ...from a few tests it seems that the pattern matching does not catch everything executable!
> ... I sent a bunch of emails attaching several different .exe files from windows os.  
> Some files were stopped and some were let through.  
> The system did stop .zip files as advertised.
 

Can you supply some examples of the exe magic code for the exe files that are "getting through" ?

A default install will stop zip v1 format files but not stop zip v2 format files. You need to enable zipv2 in the server manager panel.

Can you also supply the file magic for the zip files that are getting through ?


Remember the pattern matching contrib does not stop files by name, it examines the content. Your exe or zip files that got through may not really be executable or zip format, that's why I'm asking for an example of the file magic for those files that got through ie to see if they really are exe or zip Vxx.

Instructions for getting the magic are in the HOWTO.

[nate]

Ray,
 
First, I want to say thank you so very much!  I have seen the light and I am now a big fan; Pattern Matching Rocks!  The system is working great.
 
> Can you supply some examples of the exe magic code for the exe files that are "getting through" ?

Okay, so to test it I simply grabbed a couple random .exe files that were sitting on the root of a Windows box – I don’t even know what they do, but they were small and I sent them to my e-smith box from an outside (yahoo mail) account.  --  These file did get through!
 
Here they are with their magic numbers:

arcldr.exe =

TAEKACK5JT4AAAAAAAAAAOAADwELAQUMAOgBAABgAAAAuAIAPgYAAAAAAAAAAAIAAAAwAAAQAAAA

arcsetup.exe =
 
TAEHAMi6Uz4AAAAAAAAAAOAADwELAQUMADICAABKAAAAsgIAJwwAAAAAAAAAQAIAAAAwAAAQAAAA

> Can you also supply the file magic for the zip files that are getting through ?

No zip files are getting through that I have seen.  They are stopped at the gate. – Perfect!
 
I have enabled zip1 and 2 in the server manager.  Also, I did what you said and copied the mailpatterns file from smeserver-7pre1.
 
I will experiment with creating my own patterns now.

Thank you again Ray.
 
Nate Hartman

[raem]

nate
 
> TAEKACK5JT
> TAEHAMi6Uz

Those patterns are not in the default database, so messages with those types of file attachments will not be rejected.

To quote from the HOWTO:

"A default pattern matching database is created with common executable file patterns, which cover the majority of currently known Windows type executable viruses.
Email messages are rejected if the attachment content matches an entry in the patterns database. By default this includes the majority of *.exe files, older v1.0 *.zip files and some *.gif files."


So you can see that the default patterns for exe type files match those for known viruses that mimic certain exe files. Not all exe files are included by default as they don't need to be.

Of course you can easily add any pattern to the database, including all exe type patterns including those mentioned above.

[nate]

Got it. It all makes sence.  Thanks again.
 
- N

[idyll]

Hello.

I followed the instructions for the Virus and file blocking HOWTO using smtpfront-qmail for sme server.

I also copied the 7.0 Pre1 mailpatterns per another post.

I do not have any additional GUI configuration options for this contrib on my web server panel. I see clearly where it resides on the 7.0 server, but not my retro-fitted 6.0.1-01.

Any pointers or suggetions?

thanks
regards,

patrick

[raem]

idyll

> I do not have any additional GUI configuration options for this contrib..

Look in the existing E-mail panel, some new fields are added.

Failing that try checking if the rpms are installed
rpm -q rpmname

Failing that check all the steps in the HOWTO.

[gordonr]

Ray,
 
First, I want to say thank you so very much!  I have seen the light and I am now a big fan; Pattern Matching Rocks!  The system is working great.
 

Yep - I like it

Would you please raise a bug in the bug tracker against 7.0, noting the EXE patterns which got through? Thanks.