Topic: Saving a copy of all outgoing e-mail

[raydias]

I would like to be able to save a copy of all outgoing e-mail for security reasons. How would I do that in SME 5.5? Is this information saved anywhere already?

Thanks

Ray

[Kelvin]

>Is this information saved anywhere already?

No it isn't.

This has been a long on-going issue which unfortunately, the SME guys chose not to address.

I have been working on this exact same issue. The "proper" way to do this is to download the qmail source, change one line in one of the files (as per the qmail FAQ)  and recompile. Then install the newly compiled version into SME and add something to your .qmail file or something and that'll do it. Unfortunately, that's beyond me.

I was pointed in another direction which involves using mailsnarf (part of a network sniffer tool collection). Mailsnarf watches the nominated interfaces and captures all mail traffic and saves them into a text file which conforms to the mbox format. That part works fine, and it captures all mails (both incoming and outgoing) which is what I was after. What I have not yet worked out (as no one replied my post on this) is how to have the captured mails e-mailed to a nominated mail account for archiving / auditing.

Any ideas ?

Kelvin

[Nathan Fowler]

There is a mbox2maildir program which will convert your MBOX format to Maildir format.  Then it's simply a matter of moving the individual messages to their appropriate /home/e-smith/files/users/#USERNAME#/Maildir/new/ directory (I believe).

MBOX2Maildir:  http://www.firstpr.com.au/web-mail/mb2md/

Hope this helped,
Nathan

[Nathan Fowler]

As a side note, it appears the mb2md program (http://www.firstpr.com.au/web-mail/mb2md/) will automatically move the messages once extracted from the MBOX.

Nathan

[Kelvin]

Hi Nathan,

The problem I face is how to get the system to :-

1. Tell mailsnarf to close the capture file
2. Copy the file / e-mail the file
3. Restart mailsnarf with a new capture file

You can tell mailsnarf to create the capture file just about anywhere (even in an existing mailbox). However, if mailsnarf creates its capture file in a mailbox and that mailbox gets cleared by a mail client, the capture file gets delivered as well, but, the file is also cleared and mailsnarf does not recreate the file to continue capturing.

I don't want to split the capture file back to the original senders mailboxes, just to e-mail those mails to an archive mailbox (for archival and auditing).

Any ideas ?

Kelvin

[Nathan Fowler]

Possible Solution:

[Create a cron script]
1)  Kill mailsnarf using kill in combination with awk and grep
2)  Move the MBOX file to another users directory, perhaps a BigBrother account.
3)  SU to BigBrother account
4)  Execute mbox2maildir with CWD of /home/e-smith/files/users/BigBrother
5)  Restart mailsnart.

[Christopher]

Kelvin,

Have you had any luck with this? Have you successfully created a script? I ask so as not to re-invent the wheel if at all possible.

TIA,
Christopher

[drew]

Hello everyone...

Kelvin:  I know you have had some success with your mail archiving pursuit.  Have you been able to get it functioning to a point where it is ready for "production"?  If so, could you provide us with a detailed HOW-TO for installing and setting this up?

Nathan:  Have you created or worked on anything like what Kelvin has managed to put together?  If not, would you be willing to help put the finishing touches on a "packaged" solution to this email archiving issue?  (I'd be willing to 'fund' some of this work depending on the cost, since I am not a programmer/scripting guru myself...and this might be the only way for me to help/contribute.)

I'm surprised that Mitel hasn't yet focused on this issue.  Thanks for all your ideas/input, etc.  Let me know what you think.

Drew

[Charlie Brady]

drew wrote:

> I'm surprised that Mitel hasn't yet focused on this issue.

Mitel makes decisions about what its development priorites are based on the requirements of its customers. Those customers obviously haven't been clamouring for this feature.

Regards

Charlie

[Kelvin]

Charlie wrote :

>> I'm surprised that Mitel hasn't yet focused on this issue.
>
>Mitel makes decisions about what its development priorites are based on the
>requirements of its customers. Those customers obviously haven't been
>clamouring for this feature.

Yes, this is true. As a business, I agree.

However, given that qmail has a mechanism to do this already, but requires 1 simple change to one source file (as given by qmail's FAQ) and then recompile, I'm disappointed that this was not offered as either a default or an option. Having only limited experience in recompiling entire software packages (drivers, etc. with simple and complete make files, yes -- entire packages, no), I do not have enough background to do this on my own, nor enough spare time to work it out. While I acknowledge there are licensing restrictions on distributing a modified qmail package (discussed on these forums a while back), an option would be to make the modified binaries available as a download rpm or blade or whatever would be a great help to all.

Yes, I have had a little success with achieving the goal of archiving outgoing mails. However, I have not made this public yet for personal reasons.

Kelvin

[Nathan Fowler]

By customers, Charlie means paying customers.  We're all customers of the E-Smith/SME product, but to continue being a business Mitel must create development priorities based on the demand of the paying customer, which only makes sense.

However, other not-for-profit developers have created some very useful application and extensions for the SME product.  I would be more than willing to assist in the creation of a howto and/or any necessary code to make this feature reality.

Kelvin, are you willing to combine our efforts?  Drew, contact me via E-mail and we'll get something set up.

Nathan

[Charlie Brady]

Kelvin wrote:

> However, given that qmail has a mechanism to do this already,
> but requires 1 simple change to one source file (as given by
> qmail's FAQ) and then recompile, I'm disappointed that this
> was not offered as either a default or an option.

The qmail license prohibits distribution of binary packages containing any changes to the source code. Making this change just isn't an option for Mitel (or anyone else).

OTOH, making a change with similar intent to mailfront just might be possible. That's something you could investigate.

Charlie

[Charlie Brady]

Kelvin wrote:

>While I
> acknowledge there are licensing restrictions on distributing
> a modified qmail package (discussed on these forums a while
> back), an option would be to make the modified binaries
> available as a download rpm

That would be (quite obviously I would have thought) "a method of distributing a modified qmail package".

Charlie

[Kelvin]

Charlie Wrote :

>That would be (quite obviously I would have thought) "a method of distributing a
>modified qmail package".

That depends. I cannot remember the exact wording of the license now (it's been a while).

However, if by package you mean the entire qmail package, then, it might be the case where the modification to the one line in the source might only affect one or a few binaries and not "the entire package". And if the license does not explicitly exclude you from doing so, you could possibly package only the modified binaries and not the entire qmail system.

And yes, obviously, if the wording of the license explicitly excludes us from doing this then we might go down the path suggested in the previous discussion and work on "patching" the binaries instead. I don't recall that this was explicitly mentioned in the license either.

Kelvin

[drew]

Gentlemen,

I appreciate your posts/responses - thank you.  I am not an expert on Linux binary/package licensing/distribution, but based on Charlie's  responses it appears there is an issue with qmail which will not allow this "archiving/auditing" change to be made.  Have we determined this to be a fact?  If not - how do we go about getting this answered?  Do we need to contact the qmail folks?

If indeed the license prohibits this change/modification that is too bad, since the technical aspects apparently could be worked out with very little work or changes based on Kelvin's research and effort.  Isn't this one of the main reasons for using Linux/open source software - it allows for "us" (the end users and the development community) to make changes to increase the functionality...which in turn makes the product better and everyone benefits?  Obviously if qmail won't allow it then folks are going to move to a product which does - so why would the folks at qmail take this approach?  I don't want to get into a whole debate on licensing, but this sure seems to be a thorn which doesn't seem to make sense.

I am quite certain that as Linux makes more inroads into small/medium businesses that this email archiving/auditing feature will be an absolute requirement for legal and compliance issues.  I am actually surpirised that Mitel hasn't been hearing from customers about this yet, but I understand Mitel dollars need to be spent on the features customers want.

So where does that leave us now?  Do we need to look into another mail product (Charlie mentioned mailfront, which I think is in SME 5.5/5.6)?  I have several 5.1.2 servers running though, and wasn't planning on upgrading at the moment - so how would that work?  Remove qmail (?!?!) and insert a different mail product?  Is this even possbile or a realistic option?

Kelvin/Nathan:  As I said - I am willing to help with funds and testing if that is the only thing that I can do.  I will need to defer to your programming/scripting abilities and expertise in the technical areas of this project.  The question is - where do we go now?

Charlie:  I assume Mitel has at least looked at this email archiving/auditing issue - what have those discussions entailed and what direction are you headed in?  I assume whatever it is is based on mailfront?  If so, I guess we should direct our efforts and resources in that direction.

(I will try to move this over to the developers area if that is what you guys want, but I thought this might attract a better audience and wider thoughts in this forum.)

Thanks,

Drew

[Kelvin]

If you had a look at the qmail web site, it clearly mentions that if you wish to distribute modified binaries, you could ask permission from Dan, the author. As the change we are after is clearly outlined in qmail's own FAQ, I don't see why Dan would not allow it. So, we could ask permission, although it would sound a lot better coming from Mitel. Of course, if Mitel does not deem this important enough, then they won't bother.

Kelvin

[Charlie Brady]

Kelvin wrote:

> As the change we are
> after is clearly outlined in qmail's own FAQ, I don't see why
> Dan would not allow it.

You obviously know very little about Dan Bernstein. Google will tell you more.

> So, we could ask permission...

You would be wasting your time. You are almost certain to get no reply.

Charlie

[Kelvin]

Charlie Wrote :

>You obviously know very little about Dan Bernstein. Google will tell you more.

Granted. Then again, I know very little about just about everyone at Mitel, Microsoft, Symantec, etc. yet I still choose to put faith in their products, support (if any), etc.

Anyway, from what I can tell by reading Dan's licensing as well as other discussions on the net, patching is not disalllowed and therefore if anyone out there with enough knowledge to do so should be free to distribute a patch file / program to accomplish what we want as long as the patch is applied to the original binary distribution.

Obviously, patching a patched version is asking for trouble, but as clearly implied, the qmail program distributed with SME has to be the original unmodified version. So anyone producing a patch for the qmail in the SME server should not have an issue with this (unless they have already applied some other patch !).


> You would be wasting your time. You are almost certain to get no reply.

Won't know till you try. It might take a 5 minute e-mail and if he does reply positively, this would save days, even weeks of working around the problem. Like the problem with bind in 5.1.2 I brought up a while back. If there is a simple enough way to fix the issue in an existing 5.1.2 server (in this case upgrade bind), why would you force the client to upgrade their server to a newer version (free software or otherwise, there are issues with inconvenience, downtime, etc.). Plus a risk that the new version might not work correctly on their existing hardware, etc. (been there, done that!).

Kelvin

[Luk Vermeylen]

About monitoring incoming and outgoing e-mail:
I did this yesterday and it seems to work. It even war relatively
simple.
I installed a few development packages (make, gcc,...)downloaded qmail
tarball extracted this one ,stopped the qmail service, changed the file
according qmail faq, did a make setup check, made a
/var/qmail/alias/.qmail-log file with in that one a line ./msg-log
afterwards created 2 e-mail aliasses for the user who had to receive the
monitored e-mail: log@... and alias-localdelivery-log@...

everything seems to work. The only thing I would like to change is that
the subject line in the monitored e-mails (send to log@ and
alias-localdelivery-log@ schould be modified with monitor outgoing or
incoming acoording the mail is incoming or outgoing, but found until now
no solution for this

Luk

[Arby Edi]

Luk,

What version of SME did you do this on?  I'm about to attempt this on 5.1.2 but I really don't know too much.  Anything you can do to help (step byu step instructions?) would be greatly appreciated.

[Luk Vermeylen]

Did this on a 5.5

I did it like mentionned in the faq of qmail

created 1 user acccounts (copy)
created 2 pseudonyms for 'copy' alias-localdelivery-log and log (if you
don't create them, you get undelivered mail- seems that the change in qmail
delivers the mails to them)

for qmail: http://cr.yp.to/qmail/faq/admin.html#copies

I downloaded qmail tarball, untarred it, changed the extra.h file as
mentionned, did a ./config (mentionned in the install.ctl file)
put the line ./msg-log in the /var/qmail/alias/.qmail-log file

restarted the server and it worked
I don't know if this is the exact way, but it works for me

If my guidelines are not totally correct, let me know, and i'll try to make
a howto

don't try this on a working box, do this first on a test box or a box not
yet running as production server

Luk

[Drew]

Kelvin/Luk/Nathan,

I am sending you guys an email on this topic off phorum in order to come to some form of conclusion on this whole outbound mail logging issue.

Thanks,
Drew

[ChanChong]

Sorry to write in.

But is there any conclusion upon this issue? Seems like this is the thread that is really working on this issue.

There isn't any other threads that gives this issue so much coverage. If there is, can someone also help to point me to the right direction.

Thanks.

[Kelvin]

Hi ChanChong,

Try :-

http://www.quasarsoft.com.au/howtos/html/mailcapture_howto.html

Not perfect, but see how far this gets you.

Kelvin

[Michiel]

From a posting in dev-info by Gordon Rowell:

- Install Charlie Brady's mailfront-0.86-1es2 (see his contrib area on any of the mirror servers)
- Create a mail log user (e.g. maillog)
- echo maillog > /var/service/smtpfront-qmail/env/LOG_RECIPIENT

That's all. All your incoming & outgoing mail will be copied to user maillog. With some procmail coding you can then sort the mail to different folders or exclude certain mails from being archived. A bit limited, but it works quite well.

Michiel

[Luk Vermeylen]

how to let it work in 5.6 or 6.0?
mailfront rpm is for 5.5, will it work also on 5.6 and/or 6.0?

Thanks

Luk