Topic: [solved] High Load Average after last upgrade: qpsmtpd-forkser

[Jáder]

Hi,

I've seeing a high LA after last upgrade
I register every upgrade/update so this is what I've up0grade yesterday:

Code: [Select]

16.jul.16 - updates SO
Removed:
  kernel.x86_64 0:2.6.32-573.22.1.el6                          kernel-devel.x86_64 0:2.6.32-573.22.1.el6

Installed:
  kernel.x86_64 0:2.6.32-642.3.1.el6                           kernel-devel.x86_64 0:2.6.32-642.3.1.el6

Updated:
  kernel-firmware.noarch 0:2.6.32-642.3.1.el6                      kernel-headers.x86_64 0:2.6.32-642.3.1.el6
  mdadm.x86_64 0:3.3.4-1.el6_8.5                                   nfs-utils.x86_64 1:1.2.3-70.el6_8.1
  nss-softokn.x86_64 0:3.14.3-23.3.el6_8                           nss-softokn-freebl.i686 0:3.14.3-23.3.el6_8
  nss-softokn-freebl.x86_64 0:3.14.3-23.3.el6_8                    sendmail-milter.x86_64 0:8.14.4-9.el6_8.1
  squid.x86_64 7:3.1.23-16.el6_8.5                                 tar.x86_64 2:1.23-15.el6_8
  tzdata.noarch 0:2016f-1.el6                                      yum.noarch 0:3.2.29-75.el6.centos

Using TOP I can see:

Code: [Select]

6848 qpsmtpd   20   0  175m  95m 2544 R 100.0  1.2   0:04.15 qpsmtpd-forkser
 6849 qpsmtpd   20   0  175m  95m 2544 R 99.5  1.2   0:03.94 qpsmtpd-forkser
 6841 qpsmtpd   20   0  177m  97m 2616 S 93.1  1.2   0:04.62 qpsmtpd-forkser
 6850 qpsmtpd   20   0  168m  88m 2476 R 69.2  1.1   0:02.08 qpsmtpd-forkser
Most of time I see a qpsmtpd-forkser usin 90-95% CPU!

Any tips?

Regards

Jáder

[Jáder]

Not sure what was the cause, but after google search I find out about problems with clamd.
So I stopped it... "config clamd status disabled" and problem went away.
So I was without receiving e-mail... and an signal-event email-update fixed it.

[Jean-Philippe Pialasse]

clamd log and qpsmtpd logs might help to understand what was happening.

Mostly I would say you were receiving a huge email that clamd  had difficulty to scan, not related to your update.


ALso what you where willing is not to disable clamav, but rather disable mail scan by clamav (config setprop smtpd VirusScan disabled), unless you also wanted to remove periodically scan of files on the server.
You should disable it from the server manager in email setting , and reenable it for files in server manager. That is why you stop receiving email after disabling clamav, qpsmtpd was waiting for a clamav process. The signal event disabled Viruscan.

you might also try to reduce the MaxScannerSize (config getprop qpsmtpd MaxScannerSize ; default 25000000) depending on the amount of memory available on your system (free -m) and for for clamd (config getprop clamd MemLimit ; default 1400000000). If you have less than 3 Gig of memory, yes you have to disable clamd totally, as it will need more memory than what is available to run.

[Jáder]

Yes, I´ll try to find out why it was using 90-100% of CPU.

In /var/log/clamd/current (and old files) I just found this message:

Code: [Select]

2016-07-15 14:42:45.341865500 LibClamAV Warning: cli_tnef: file truncated, returning CLEAN

The qpsmtpd log is rotating so quick I just have files from today... a lot of files (12!) all from same date.
Every step of e-mail delivered is logged!
Anyways, I found this message there:

Code: [Select]

@40000000578e05ba299a0064 2365 cleaning up after 51518
@40000000578e05bb035c774c 51524 virus::clamav plugin (data_post): Changing permissions on file to permit scanner access
@40000000578e05bb0bbf416c 51524 virus::clamav plugin (data_post): clamscan results: /var/spool/qpsmtpd/1468925358:51524:0: OK
@40000000578e05bb0bc8cad4 51524 logging::logterse plugin (queue): ` 127.0.0.2   Unknown mydomain.tld   
Note the 127.0.0.2   Unknown followed by mydomain.tld
Should I care about this message ?

I have 8GB RAM on this server. All other limits you pointed are default (14M e 25M).
I´m sure the non-receiving-email-problem was caused by wrong way to stop clamd (I´m a moron some times: "I just wanna to stop it eating all CPU right now" way)

I have qmHandle installed and used it to see outgoing queue, but do not watch for incoming queue.

[CharlieBrady]

Note the 127.0.0.2   Unknown followed by mydomain.tld
Should I care about this message ?

That probably means you are using fetchmail.

[Jáder]

That probably means you are using fetchmail.

No, I'm not using fetchmail. This server has many new RPMs (including VirtualBox) but is server with full SMTP funcionality so no fetchmail at all.

BTW I tracked the problem to latest kernel update (kernel.x86_64 0:2.6.32-642.3.1.el6). I'm unable to go back and test old kernel (I'm 4.000 km away from server!) but my logs and sysmon graph confirm it.
EDIT1: I forgot to say it may be a VirtualBox update also. I'm investigating it right now!

[Jean-Philippe Pialasse]

maybe not related as you pointed qpsmtpd, but when using Virtualbox on SME I have seen that sometime the process is using a lot of CPU if you have only one VM running, everything is ok if you have two or more.

as pointed by Charlie 127.0.0.2  is normally a sign that a fetchmail instance is running as this is the local adress used by it.

[Jáder]

maybe not related as you pointed qpsmtpd, but when using Virtualbox on SME I have seen that sometime the process is using a lot of CPU if you have only one VM running, everything is ok if you have two or more.

as pointed by Charlie 127.0.0.2  is normally a sign that a fetchmail instance is running as this is the local adress used by it.

I think about it (VirtualBox being the cause) but then I use top I allways can see several (2 or more) qpsmtpd-forsker at top using +90% CPU.

[Jáder]

I need URGENTE HELP.

I'd like to know WHAT SERVICE is running on my server (or network) and identifying himself as 127.0.0.2 because this service is sending mail to  one fixed address  ending at @p10361243119.s2599.jianzhanapp.com

I'm unable to block these e-mails to leave server/network.
Any tips where to start looking for and what to do to block this till I fix it ?

Thanks

Jáder

[Jean-Philippe Pialasse]

Code: [Select]

netstat -an |grep EST
also, as you mentionned having qmhandle, what gives

Code: [Select]

# qmHandle  -l -NI would guess your 127.0.0.2 is only dnscache looking to resolve adresses and connecting to port 53

I would suspect a machine on your lan sending tons of emails (qpsmtpd working) , then qmail should have to work  but as we only saw really small chunk of logs this is a guess.


if you fear mails are getting out

Code: [Select]

service qmail stop
the you can try to fix what happen, use qmhandler to purge selectively and  then

Code: [Select]

service qmail start

[Jáder]

Hi  Jean-Philippe

I think the 127.0.0.2 may be the DNS query of dnscache.
But I 'd like to how to find out:
1) from where those strange message are coming from ?
2) What's the message content

I got some info on server:

Code: [Select]

[root@andorinha etc]# lsof -i TCP:25
COMMAND    PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
qmail-rem 1889  qmailr    3u  IPv4 599027      0t0  TCP 127.0.0.2:42664->127.0.0.2:smtp (ESTABLISHED)
qpsmtpd-f 1890 qpsmtpd    0u  IPv4 599028      0t0  TCP 127.0.0.2:smtp->127.0.0.2:42664 (ESTABLISHED)
qpsmtpd-f 1890 qpsmtpd    1u  IPv4 599028      0t0  TCP 127.0.0.2:smtp->127.0.0.2:42664 (ESTABLISHED)
qpsmtpd-f 1890 qpsmtpd    6u  IPv4 599028      0t0  TCP 127.0.0.2:smtp->127.0.0.2:42664 (ESTABLISHED)
qpsmtpd-f 1897 qpsmtpd    0u  IPv4 599250      0t0  TCP andorinha.xxxxx.com.br:smtp->116.75.140.24:12005 (ESTABLISHED)
qpsmtpd-f 1897 qpsmtpd    1u  IPv4 599250      0t0  TCP andorinha.xxxxx.com.br:smtp->116.75.140.24:12005 (ESTABLISHED)
qpsmtpd-f 1897 qpsmtpd    6u  IPv4 599250      0t0  TCP andorinha.xxxxx.com.br:smtp->116.75.140.24:12005 (ESTABLISHED)
qpsmtpd-f 2551 qpsmtpd    4u  IPv4  15405      0t0  TCP *:smtp (LISTEN)



Code: [Select]

[root@andorinha etc]# qmHandle  -l -N
Total messages: 2
Messages with local recipients: 0
Messages with remote recipients: 2
Messages with bounces: 0
Messages in preprocess: 0
[root@andorinha etc]#


Code: [Select]

[root@andorinha etc]# netstat -an |grep EST|grep -v 993|grep -v \:139|grep -v \:1521
tcp        0      0 127.0.0.1:9001              127.0.0.1:57876             ESTABLISHED
tcp        0     40 192.168.47.5:22             187.114.196.2:54885         ESTABLISHED
tcp        0      0 127.0.0.1:57876             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:57856             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57854             ESTABLISHED
tcp        0      0 127.0.0.1:57858             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57860             ESTABLISHED
tcp        0      0 127.0.0.1:57868             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:57854             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:57866             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57846             ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57870             ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57866             ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57868             ESTABLISHED
tcp        0      0 127.0.0.1:57860             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57856             ESTABLISHED
tcp        0      0 127.0.0.1:57846             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:57870             127.0.0.1:9001              ESTABLISHED
tcp        0      0 127.0.0.1:9001              127.0.0.1:57858             ESTABLISHED
udp        0      0 192.168.47.5:46049          203.158.18.33:53            ESTABLISHED
udp        0      0 192.168.47.5:33134          192.168.47.5:53             ESTABLISHED
udp        0      0 127.0.0.2:49498             127.0.0.2:53                ESTABLISHED


[Jean-Philippe Pialasse]

all I see there is that the indian IP 116.75.140.24 is sending to you emails with 3 simultaneous connections.

Are you using  openswan or  libreswan as a VPN ?

check you /var/log/qpsmtpd/current log if there are a lot of connexions, this might rather be a mailbomb fromt he outside.


I would give a try to smeserver-fail2ban enabling the qpsmtpd jail, however if you are away be really cautious on the procedure to isntall it to avoid to be locked out of the server .

[Daniel B.]

I'd like to know WHAT SERVICE is running on my server (or network) and identifying himself as 127.0.0.2 because this service is sending mail to  one fixed address  ending at @p10361243119.s2599.jianzhanapp.com

127.0.0.2 is your SME Server, it cannot be anything on your network. We don't know what is running on it. You should know better than us. Connections from 127.0.0.2 most likely means you have a fetchmail instance running (or something similar). You might also have a SMTP loop somewhere. Have you played with /var/qmail/control/smtproutes ? Are you using a smarthost ?  We can see you have a qmail-remote instance connecting on 127.0.0.2:25, which could well be a routing loop issue

[CharlieBrady]

I think the 127.0.0.2 may be the DNS query of dnscache.

The MX record for p10361243119.s2599.jianzhanapp.com points to 127.0.0.2:

Code: [Select]

;; ANSWER SECTION:
p10361243119.s2599.jianzhanapp.com. 43200 IN MX 0 mail.p10361243119.s2599.jianzhanapp.com.

;; AUTHORITY SECTION:
p10361243119.s2599.jianzhanapp.com. 43200 IN NS cm1.xrnet.cn.
p10361243119.s2599.jianzhanapp.com. 43200 IN NS cm2.xrnet.cn.

;; ADDITIONAL SECTION:
mail.p10361243119.s2599.jianzhanapp.com. 43200 IN A 127.0.0.2

So it is looking back through your mail system. If you wait enough loops, then qmail will dump the message.

Alternatively, you can stop qmail and delete the message from the queue. But you need to work out where that message came from and try to prevent this happening again.

One possibility is that it is a bounce message in response to a spam.

[Jáder]

I´m afraid the SMTP is being used to send out information.

I´ve seen a big amount of data in output... I´d love to see one of those messages.
I´ve created an entry to p10361243119.s2599.jianzhanapp.com pointing to  10.10.10 in /etc/hosts file trying to avoid the e-mail to leave my server.
It appears do not work!


see:

Code: [Select]

[root@andorinha ~]# cat /etc/hosts
#------------------------------------------------------------
#              !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------
127.0.0.1       localhost
192.168.47.5     andorinha.xxxxxx.com.br andorinha
10.0.10.10      p10361243119.s2599.jianzhanapp.com

Right now I have qmail stopped most of time.

How I can find out what is running as a service on 127.0.0.2 (if there any !)... yes, I know, it´s my server... but I fear do not to know how be sure I´m not being hacked!!!