Topic: Can't stop qmail

[uniqsys]

Hi all,

Here is a real problem.  I have a server with 1.6 million emails in the message queue. It seems someone brought in an infected laptop to the network and it is generating SPAM emails.  The workstations are all shutdown so no email is being generated currently.  What I am trying to do is stop qmail so I can delete the messages with qmHandle. (The only way I know how at the moment.) The command from these forums is "sv d /service/qmail" to stop the service but it won't stop it.  I check it with "sv s /service/qmail".  I have even tried to Kill the process via htop but to no avail.  Advice on the forum from developers is to never mess with the message queue without stopping qmail first.  But it won't stop for me, what do I do?

The server is SME 7.3 with all updates and with no contribs except for SME7Admin and qmHandle which I installed for handling this problem.

BTW, at this point I have the server isolated from anything except itself.

Thanks.

[william_syd]

Do you have to disable it in the configuration db before you stop it?



[root@tiger ~]# config show qmail
qmail=service
    FilterType=procmail
    MaxMessageSize=15000000
    status=enabled
[root@tiger ~]#

Although, stopping it correctly (like other services) should keep it stopped.

[uniqsys]

Do you have to disable it in the configuration db before you stop it?

That is an idea.  I'll try it.  I just wonder if qmHandle will delete messages on a disabled service.  We'll find out.

[CharlieBrady]

The command from these forums is "sv d /service/qmail" to stop the service but it won't stop it.

It will. You might just need to be patient, waiting for existing outbound connections to terminate.

[CharlieBrady]

Do you have to disable it in the configuration db before you stop it?

No.

[william_syd]

I remember a post on here between CharlieBrady and other(s) about qmail and qmHandle.

This one >> http://forums.contribs.org/index.php?topic=40959.0

Was some good info in it from memory.


Warning - while you were typing 2 new replies have been posted. You may wish to review your post.

[uniqsys]

It will. You might just need to be patient, waiting for existing outbound connections to terminate.

When you say "existing outbound connections", does that include the MX contacts it is trying to send?  If so, I think that is my problem.  The queue is so overwhelmed that there is no time slot to interrupt it; if it works that way.  Besides how patient must we be?  I waited 15 minutes for it to stop after sending the command to stop the qmail and the status still said it was running.  Is temporarily disabling the service OK to do?

[mmccarn]

I had a non-sme server compromised in a similar manner 2 or 3 years ago. 

Since the pending outbound messages were (clearly) pure-d bona-fide SPAM, quite a bit of it was getting caught by "tarpit" servers. 

A tarpit server will let your server establish an SMTP connection, then never let any traffic pass on that connection.  Pretty soon, all of the configured out-bound SMTP connections are tied up by tarpits, and the spam host (your SME, in this case) cannot deliver spam (or email) to anyone.

One side effect of this was that the qmail service could not be stopped - because it won't stop while there are any open smtp connections, and the tarpits won't allow the connections to close.

In fact, I learned about tarpits by running netstat -an | grep :25.*EST, then doing some online research on the remote systems my server was connected to.

You need to find some way to disable qmail internet connectivity until you can get your queue cleaned out - either by unplugging your WAN cable or by some other means.

Finally, the advice about not playing with the qmail queue while qmail is running is mostly aimed at preserving valid emails that are in the queue while eliminating the unwanted emails.  If you are willing to wipe out the whole queue you may have less to worry about...

[uniqsys]

I could not get qmail to stop as a service, so I disabled qmail and did a signal-event post-upgrade and signal-event reboot so the service does not start on reboot.  The tarpit seems the most likely problem here.  Thanks for the info MMCCARN.  Using qmHandle -D to dump all the queues, I get "message # slotted for deletion" scrolling on the screen and then it hits one particular message and hangs.  Is there any way to do a mass deletion other than with qmHandle?

By the way, regarding:

Finally, the advice about not playing with the qmail queue while qmail is running is mostly aimed at preserving valid emails that are in the queue while eliminating the unwanted emails.  If you are willing to wipe out the whole queue you may have less to worry about...

I thought that the developer meant it might corrupt qmail completely, so that qmail would not work at all.  I try not to take too many chances with email issues when I don't know much.

[mmccarn]

I thought that you should be able to fix qmail by doing an upgrade install (for example), but a little google searching leads me to believe that qmail is more finicky than I thought...

I think you can clear out your qmail entirely (ALL queued messages GONE FOREVER) as follows:

1. Make sure qmail is stopped (make sure the result of the command is "ok: down: qmail...")

Code: [Select]

sv stop qmail
2. Remove all pending messages from all qmail queue folders:

Code: [Select]

cd /var/qmail/queue
rm -f mess/*/*
rm -f info/*/*
rm -f local/*/*
rm -f remote/*/*


[uniqsys]

Thanks MMCCARN.  I too did some searching too and came across:
http://www.debianadmin.com/delete-qmail-server-messages-queue.html
and http://www.debianhelp.co.uk/qmailqueue.htm

which talk about the same thing.  It looks as if forcing the removal of the directories does it as you point out and give helpful commands for.
I am trying it now.

Thanks.

[uniqsys]

This has been a real problem.  In order to clean out the queues you have to stop qmail. OK But the "sv d /service/qmail" or "sv stop qmail"  commands could not break in to be invoked.  Instead I had to set the qmail status to "disabled" and signal-event post-upgrade, signal-event reboot the machine to keep qmail from starting at startup.

I then cleaned the queues with the commands

Code: [Select]

find /var/qmail/queue/mess -type f -exec rm {} \;
find /var/qmail/queue/info -type f -exec rm {} \;
find /var/qmail/queue/local -type f -exec rm {} \;
find /var/qmail/queue/intd -type f -exec rm {} \;
find /var/qmail/queue/todo -type f -exec rm {} \;
find /var/qmail/queue/remote -type f -exec rm {} \;
and then later

Code: [Select]

find /var/qmail/queue/pid -type f -exec rm {} \;because that was loaded and prevented any mail from processing.  I tried  mmccarn's suggested commands

Code: [Select]

cd  /var/qmail/queue
rm -f mess/*/*
etc, but they gave an error that there were too many files to delete.  So I used the find command.  It took a while to delete 1.6 million emails!


But now I need some help with how to get rid of the SPAM generator in the network.  I have isloated 2 machines.  Running Norton and CA security suite has not detected anything.  Any other pointers on tools I could use?  I am googling SPAM generators now.

Thanks.

[CharlieBrady]

This has been a real problem.  In order to clean out the queues you have to stop qmail. OK But the "sv d /service/qmail" or "sv stop qmail"  commands could not break in to be invoked.  Instead I had to set the qmail status to "disabled" and signal-event post-upgrade, signal-event reboot the machine to keep qmail from starting at startup.

I'm sure that you did not need to do that. qmail will always stop, after it has tried to finish sending mail which it is in the process of sending. The problem is that qmail-remote, when connected to a tarpit SMTP server, will take a very long time to finish. Unless you disconnect your WAN connection, in which case they should die quickly.

But now I need some help with how to get rid of the SPAM generator in the network.  I have isloated 2 machines.

Reformat them, and install linux

[I think the topic of how to remove malware from Windows PCs is off-topic for this forum.]

[Stefano]

But now I need some help with how to get rid of the SPAM generator in the network.  I have isloated 2 machines.  Running Norton and CA security suite has not detected anything.  Any other pointers on tools I could use?  I am googling SPAM generators now.

a bit OT but my 2c:

- put the infected pcs on a lan segment without internet connection (or use a firewall rule)
- try tools as sophos rootkit revealer (it's free): some time ago I've had a problem with a pc with no virus.. but with a rootkit
- try to do a offline av scan with a boot cd like bartpe or a linux live cd otherwise you could find unremovable files

HTH

Ciao
Stefano

[uniqsys]

Reformat them, and install linux

I would love to!!   If only I could get them to understand...

[I think the topic of how to remove malware from Windows PCs is off-topic for this forum.]

I see your point and will let it drop here.  Thanks for everyone's help.

[uniqsys]

I'm sure that you did not need to do that. qmail will always stop, after it has tried to finish sending mail which it is in the process of sending. The problem is that qmail-remote, when connected to a tarpit SMTP server, will take a very long time to finish. Unless you disconnect your WAN connection, in which case they should die quickly.

BTW, one last comment on this. I do not doubt this, but I had to support this server remotely so I had to use the WAN connection to instruct the server!  Catch 22 This really limits your options. 

[mmccarn]

A couple notes:

* You might thwart your infected LAN systems by forcing your LAN clients to use SMTP authentication when sending: http://wiki.contribs.org/Email#How_do_I_disable_SMTP_relay_for_unauthenticated_LAN_clients

* You might be able to successfully stop qmail when trapped by tarpits if you can identify the tarpit hosts, then use iptables to block all traffic to those hosts.  I suspect that once you issue the command to shutdown qmail you could use netstat -an | grep :25.*EST to identify the connections that are stuck open, then figure out the required iptables commands to block those connections.

[phil_elvey]

Hi I have had a similar problem on my Server with spam being sent out.

I have enabled SMTP authentication and disabled unauthenticated SMTP relay as per the instructions on this site.  I have a small question - in the server manager the SMTP proxy is disabled.  Does it affect any of the changes I have made if I enable it?  Are there any disadvantages, other than being forced to use the server as SMTP (or rather SSMTP)?

[mmccarn]

If all of your LAN workstations use the SME server for outbound SMTP, turning on SMTP proxy will have no ill effects.

If you have any LAN clients configured to send email through outside servers, turning on SMTP proxy will force you to reconfigure those workstations.