Koozali.org: home of the SME Server
Obsolete Releases => SME Server 9.x => Topic started by: Jáder on July 17, 2016, 03:18:25 PM
-
Hi,
I've seeing a high LA after last upgrade
I register every upgrade/update so this is what I've up0grade yesterday:
16.jul.16 - updates SO
Removed:
kernel.x86_64 0:2.6.32-573.22.1.el6 kernel-devel.x86_64 0:2.6.32-573.22.1.el6
Installed:
kernel.x86_64 0:2.6.32-642.3.1.el6 kernel-devel.x86_64 0:2.6.32-642.3.1.el6
Updated:
kernel-firmware.noarch 0:2.6.32-642.3.1.el6 kernel-headers.x86_64 0:2.6.32-642.3.1.el6
mdadm.x86_64 0:3.3.4-1.el6_8.5 nfs-utils.x86_64 1:1.2.3-70.el6_8.1
nss-softokn.x86_64 0:3.14.3-23.3.el6_8 nss-softokn-freebl.i686 0:3.14.3-23.3.el6_8
nss-softokn-freebl.x86_64 0:3.14.3-23.3.el6_8 sendmail-milter.x86_64 0:8.14.4-9.el6_8.1
squid.x86_64 7:3.1.23-16.el6_8.5 tar.x86_64 2:1.23-15.el6_8
tzdata.noarch 0:2016f-1.el6 yum.noarch 0:3.2.29-75.el6.centos
Using TOP I can see:
6848 qpsmtpd 20 0 175m 95m 2544 R 100.0 1.2 0:04.15 qpsmtpd-forkser
6849 qpsmtpd 20 0 175m 95m 2544 R 99.5 1.2 0:03.94 qpsmtpd-forkser
6841 qpsmtpd 20 0 177m 97m 2616 S 93.1 1.2 0:04.62 qpsmtpd-forkser
6850 qpsmtpd 20 0 168m 88m 2476 R 69.2 1.1 0:02.08 qpsmtpd-forkser
Most of time I see a qpsmtpd-forkser usin 90-95% CPU!
Any tips?
Regards
Jáder
-
Not sure what was the cause, but after google search I find out about problems with clamd.
So I stopped it... "config clamd status disabled" and problem went away.
So I was without receiving e-mail... and an signal-event email-update fixed it.
-
clamd log and qpsmtpd logs might help to understand what was happening.
Mostly I would say you were receiving a huge email that clamd had difficulty to scan, not related to your update.
ALso what you where willing is not to disable clamav, but rather disable mail scan by clamav (config setprop smtpd VirusScan disabled), unless you also wanted to remove periodically scan of files on the server.
You should disable it from the server manager in email setting , and reenable it for files in server manager. That is why you stop receiving email after disabling clamav, qpsmtpd was waiting for a clamav process. The signal event disabled Viruscan.
you might also try to reduce the MaxScannerSize (config getprop qpsmtpd MaxScannerSize ; default 25000000) depending on the amount of memory available on your system (free -m) and for for clamd (config getprop clamd MemLimit ; default 1400000000). If you have less than 3 Gig of memory, yes you have to disable clamd totally, as it will need more memory than what is available to run.
-
Yes, I´ll try to find out why it was using 90-100% of CPU.
In /var/log/clamd/current (and old files) I just found this message:
2016-07-15 14:42:45.341865500 LibClamAV Warning: cli_tnef: file truncated, returning CLEAN
The qpsmtpd log is rotating so quick I just have files from today... a lot of files (12!) all from same date.
Every step of e-mail delivered is logged!
Anyways, I found this message there:
@40000000578e05ba299a0064 2365 cleaning up after 51518
@40000000578e05bb035c774c 51524 virus::clamav plugin (data_post): Changing permissions on file to permit scanner access
@40000000578e05bb0bbf416c 51524 virus::clamav plugin (data_post): clamscan results: /var/spool/qpsmtpd/1468925358:51524:0: OK
@40000000578e05bb0bc8cad4 51524 logging::logterse plugin (queue): ` 127.0.0.2 Unknown mydomain.tld
Note the 127.0.0.2 Unknown followed by mydomain.tld
Should I care about this message ?
I have 8GB RAM on this server. All other limits you pointed are default (14M e 25M).
I´m sure the non-receiving-email-problem was caused by wrong way to stop clamd :) (I´m a moron some times: "I just wanna to stop it eating all CPU right now" way)
I have qmHandle installed and used it to see outgoing queue, but do not watch for incoming queue.
-
Note the 127.0.0.2 Unknown followed by mydomain.tld
Should I care about this message ?
That probably means you are using fetchmail.
-
That probably means you are using fetchmail.
No, I'm not using fetchmail. This server has many new RPMs (including VirtualBox) but is server with full SMTP funcionality so no fetchmail at all.
BTW I tracked the problem to latest kernel update (kernel.x86_64 0:2.6.32-642.3.1.el6). I'm unable to go back and test old kernel (I'm 4.000 km away from server!) but my logs and sysmon graph confirm it.
EDIT1: I forgot to say it may be a VirtualBox update also. I'm investigating it right now!
-
maybe not related as you pointed qpsmtpd, but when using Virtualbox on SME I have seen that sometime the process is using a lot of CPU if you have only one VM running, everything is ok if you have two or more.
as pointed by Charlie 127.0.0.2 is normally a sign that a fetchmail instance is running as this is the local adress used by it.
-
maybe not related as you pointed qpsmtpd, but when using Virtualbox on SME I have seen that sometime the process is using a lot of CPU if you have only one VM running, everything is ok if you have two or more.
as pointed by Charlie 127.0.0.2 is normally a sign that a fetchmail instance is running as this is the local adress used by it.
I think about it (VirtualBox being the cause) but then I use top I allways can see several (2 or more) qpsmtpd-forsker at top using +90% CPU.
-
I need URGENTE HELP.
I'd like to know WHAT SERVICE is running on my server (or network) and identifying himself as 127.0.0.2 because this service is sending mail to one fixed address ending at @p10361243119.s2599.jianzhanapp.com
I'm unable to block these e-mails to leave server/network.
Any tips where to start looking for and what to do to block this till I fix it ?
Thanks
Jáder
-
netstat -an |grep EST
also, as you mentionned having qmhandle, what gives
# qmHandle -l -NI would guess your 127.0.0.2 is only dnscache looking to resolve adresses and connecting to port 53
I would suspect a machine on your lan sending tons of emails (qpsmtpd working) , then qmail should have to work but as we only saw really small chunk of logs this is a guess.
if you fear mails are getting out
service qmail stop
the you can try to fix what happen, use qmhandler to purge selectively and then
service qmail start
-
Hi Jean-Philippe
I think the 127.0.0.2 may be the DNS query of dnscache.
But I 'd like to how to find out:
1) from where those strange message are coming from ?
2) What's the message content
I got some info on server:
[root@andorinha etc]# lsof -i TCP:25
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
qmail-rem 1889 qmailr 3u IPv4 599027 0t0 TCP 127.0.0.2:42664->127.0.0.2:smtp (ESTABLISHED)
qpsmtpd-f 1890 qpsmtpd 0u IPv4 599028 0t0 TCP 127.0.0.2:smtp->127.0.0.2:42664 (ESTABLISHED)
qpsmtpd-f 1890 qpsmtpd 1u IPv4 599028 0t0 TCP 127.0.0.2:smtp->127.0.0.2:42664 (ESTABLISHED)
qpsmtpd-f 1890 qpsmtpd 6u IPv4 599028 0t0 TCP 127.0.0.2:smtp->127.0.0.2:42664 (ESTABLISHED)
qpsmtpd-f 1897 qpsmtpd 0u IPv4 599250 0t0 TCP andorinha.xxxxx.com.br:smtp->116.75.140.24:12005 (ESTABLISHED)
qpsmtpd-f 1897 qpsmtpd 1u IPv4 599250 0t0 TCP andorinha.xxxxx.com.br:smtp->116.75.140.24:12005 (ESTABLISHED)
qpsmtpd-f 1897 qpsmtpd 6u IPv4 599250 0t0 TCP andorinha.xxxxx.com.br:smtp->116.75.140.24:12005 (ESTABLISHED)
qpsmtpd-f 2551 qpsmtpd 4u IPv4 15405 0t0 TCP *:smtp (LISTEN)
[root@andorinha etc]# qmHandle -l -N
Total messages: 2
Messages with local recipients: 0
Messages with remote recipients: 2
Messages with bounces: 0
Messages in preprocess: 0
[root@andorinha etc]#
[root@andorinha etc]# netstat -an |grep EST|grep -v 993|grep -v \:139|grep -v \:1521
tcp 0 0 127.0.0.1:9001 127.0.0.1:57876 ESTABLISHED
tcp 0 40 192.168.47.5:22 187.114.196.2:54885 ESTABLISHED
tcp 0 0 127.0.0.1:57876 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:57856 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57854 ESTABLISHED
tcp 0 0 127.0.0.1:57858 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57860 ESTABLISHED
tcp 0 0 127.0.0.1:57868 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:57854 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:57866 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57846 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57870 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57866 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57868 ESTABLISHED
tcp 0 0 127.0.0.1:57860 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57856 ESTABLISHED
tcp 0 0 127.0.0.1:57846 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:57870 127.0.0.1:9001 ESTABLISHED
tcp 0 0 127.0.0.1:9001 127.0.0.1:57858 ESTABLISHED
udp 0 0 192.168.47.5:46049 203.158.18.33:53 ESTABLISHED
udp 0 0 192.168.47.5:33134 192.168.47.5:53 ESTABLISHED
udp 0 0 127.0.0.2:49498 127.0.0.2:53 ESTABLISHED
-
all I see there is that the indian IP 116.75.140.24 is sending to you emails with 3 simultaneous connections.
Are you using openswan or libreswan as a VPN ?
check you /var/log/qpsmtpd/current log if there are a lot of connexions, this might rather be a mailbomb fromt he outside.
I would give a try to smeserver-fail2ban enabling the qpsmtpd jail, however if you are away be really cautious on the procedure to isntall it to avoid to be locked out of the server .
-
I'd like to know WHAT SERVICE is running on my server (or network) and identifying himself as 127.0.0.2 because this service is sending mail to one fixed address ending at @p10361243119.s2599.jianzhanapp.com
127.0.0.2 is your SME Server, it cannot be anything on your network. We don't know what is running on it. You should know better than us. Connections from 127.0.0.2 most likely means you have a fetchmail instance running (or something similar). You might also have a SMTP loop somewhere. Have you played with /var/qmail/control/smtproutes ? Are you using a smarthost ? We can see you have a qmail-remote instance connecting on 127.0.0.2:25, which could well be a routing loop issue
-
I think the 127.0.0.2 may be the DNS query of dnscache.
The MX record for p10361243119.s2599.jianzhanapp.com points to 127.0.0.2:
;; ANSWER SECTION:
p10361243119.s2599.jianzhanapp.com. 43200 IN MX 0 mail.p10361243119.s2599.jianzhanapp.com.
;; AUTHORITY SECTION:
p10361243119.s2599.jianzhanapp.com. 43200 IN NS cm1.xrnet.cn.
p10361243119.s2599.jianzhanapp.com. 43200 IN NS cm2.xrnet.cn.
;; ADDITIONAL SECTION:
mail.p10361243119.s2599.jianzhanapp.com. 43200 IN A 127.0.0.2
So it is looking back through your mail system. If you wait enough loops, then qmail will dump the message.
Alternatively, you can stop qmail and delete the message from the queue. But you need to work out where that message came from and try to prevent this happening again.
One possibility is that it is a bounce message in response to a spam.
-
I´m afraid the SMTP is being used to send out information.
I´ve seen a big amount of data in output... I´d love to see one of those messages.
I´ve created an entry to p10361243119.s2599.jianzhanapp.com pointing to 10.10.10 in /etc/hosts file trying to avoid the e-mail to leave my server.
It appears do not work! :(
see:
[root@andorinha ~]# cat /etc/hosts
#------------------------------------------------------------
# !!DO NOT MODIFY THIS FILE!!
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at http://www.contribs.org/development/
#
# Copyright (C) 1999-2006 Mitel Networks Corporation
#------------------------------------------------------------
127.0.0.1 localhost
192.168.47.5 andorinha.xxxxxx.com.br andorinha
10.0.10.10 p10361243119.s2599.jianzhanapp.com
Right now I have qmail stopped most of time.
How I can find out what is running as a service on 127.0.0.2 (if there any !)... yes, I know, it´s my server... but I fear do not to know how be sure I´m not being hacked!!!
-
well as pointed Charlie, the mail will never leave your server anyway, as the DNS is
mail.p10361243119.s2599.jianzhanapp.com. 43200 IN A 127.0.0.2
somebody did manage to put this mail in you system, and it keeps being sent and resent as 127.0.0.2 is localhost also
qpsmtpd pass it to qmail
qmail pass it to qpsmtpd
then qmail pass it to qpsmtpd
which pass it to qmail ....
try telnet 127.0.0.2 25 , you qpsmtd will answer
# telnet 127.0.0.2 25
Trying 127.0.0.2...
Connected to 127.0.0.2.
Escape character is '^]'.
220 me.moi.pialasse.com ESMTP
helo localhost
250 moi.pialasse.com Hi Unknown [127.0.0.2]; I am so happy to meet you.
quit
221 moi.pialasse.com closing connection. Have a wonderful day.
Connection closed by foreign host.
this look like a good way to attack a system sending hundred of mails this way ...
now what is the best way to stop this.... and also prevent this from happening
-
I´d love to see one of those messages.
With qmail stopped, you can find one in the qmail queue, using 'grep -r'.
How I can find out what is running as a service on 127.0.0.2 (if there any !)...
The issue is that qmail-remote is sending the message to qpsmtpd running on port 25, 127.0.0.2 (because qpsmtpd is listening to *). The problem is the message, and DNS, not any programs running on your server.
We could possibly solve the load issue by denying relaying on the 127.0.0.2 address. We could also have a plugin in qpsmtpd which looks for loopback addresses in the MX records of senders and recipients.
-
I stopped qmail.
I have seen the message using:
find /var/qmail/queue -name 21234028| xargs cat | less
but there are HUGE headers...just 127.0.0.2 over and over again.
I´d like to know how to remove the message...I could delete that 21234028 file... but not sure if that is the right way.
I´m sure I´ll need remove more than just one file from qmail queue.
Can someone help me with this ?
About pluging about loopback addresss: I AGREE!!
-
there is definitively a bug here we need to resolve, as there is an infinity loop
I do not know how safe it is to delete from the qpsmtpd queue
I just tested and confirmed the bug on a server, I had the opposite approach,
define mail.p10361243119.s2599.jianzhanapp.com to an ip of another server I control or any ip of your lan in /etc/hosts, and then used qmail handle to delete the mail
MESSAGE NUMBER 147776
--------------
Received: (qmail 31579 invoked by uid 453); 27 Jul 2016 14:41:12 -0000
Received: from Unknown (HELO moi.pialasse.com) (127.0.0.2)
by moi.pialasse.com (qpsmtpd/0.84) with SMTP; Wed, 27 Jul 2016 10:41:12 -0400
Received: (qmail 31574 invoked by uid 453); 27 Jul 2016 14:41:10 -0000
Received: from Unknown (HELO moi.pialasse.com) (127.0.0.2)
by moi.pialasse.com (qpsmtpd/0.84) with SMTP; Wed, 27 Jul 2016 10:41:10 -0400
Received: (qmail 31564 invoked by uid 453); 27 Jul 2016 14:41:08 -0000
Received: from Unknown (HELO moi.pialasse.com) (127.0.0.2)
by moi.pialasse.com (qpsmtpd/0.84) with SMTP; Wed, 27 Jul 2016 10:41:08 -0400
Received: (qmail 31559 invoked by uid 453); 27 Jul 2016 14:41:07 -0000
Received: from Unknown (HELO moi.pialasse.com) (127.0.0.2)
by moi.pialasse.com (qpsmtpd/0.84) with SMTP; Wed, 27 Jul 2016 10:41:07 -0400
Received: (qmail 31552 invoked by uid 453); 27 Jul 2016 14:41:05 -0000
Received: from Unknown (HELO moi.pialasse.com) (127.0.0.2)
by moi.pialasse.com (qpsmtpd/0.84) with SMTP; Wed, 27 Jul 2016 10:41:05 -0400
...
Received: from Unknown (HELO moi.pialasse.com) (127.0.0.2)
by moi.pialasse.com (qpsmtpd/0.84) with SMTP; Wed, 27 Jul 2016 10:40:22 -0400
in only 2 minutes I had hundreds of lines , if it is days you have this message it might weight hundreds of megabytes.
-
bug openned there: https://bugs.contribs.org/show_bug.cgi?id=9704
-
bug openned there: https://bugs.contribs.org/show_bug.cgi?id=9704
You should have asked jader to open the bug. We need to ensure that he is following the bug, and providing all the information required.
-
but there are HUGE headers...just 127.0.0.2 over and over again.
You need to identify the earliest Received: header, to determine how the message was first injected into the queue.
-
We could possibly solve the load issue by denying relaying on the 127.0.0.2 address.
That should already be the case:
[root@sdfdsf ~]# cd /service/qpsmtpd/peers/
[root@sdfdsf peers]# ls -l
total 8
-rw-r--r-- 1 root root 13 Jan 20 2016 0
lrwxrwxrwx 1 root root 5 Jan 20 2016 127.0.0.1 -> local
lrwxrwxrwx 1 root root 5 Jan 20 2016 192.168.122 -> local
lrwxrwxrwx 1 root root 1 Jul 8 12:52 192.168.122.1 -> 0
-rw-r--r-- 1 root root 13 Jan 20 2016 local
[root@sdfdsf peers]#
-
define mail.p10361243119.s2599.jianzhanapp.com to an ip of another server I control or any ip of your lan in /etc/hosts
jader has already tried modifying /etc/hosts - I don't think qmail looks there - just in DNS. A custom entry in /var/qmail/control/smtproutes should work.
-
jader has already tried modifying /etc/hosts - I don't think qmail looks there - just in DNS. A custom entry in /var/qmail/control/smtproutes should work.
it will need a service qmail restart anyway in both cases
-
That should already be the case:
[root@sdfdsf ~]# cd /service/qpsmtpd/peers/
[root@sdfdsf peers]# ls -l
total 8
-rw-r--r-- 1 root root 13 Jan 20 2016 0
lrwxrwxrwx 1 root root 5 Jan 20 2016 127.0.0.1 -> local
lrwxrwxrwx 1 root root 5 Jan 20 2016 192.168.122 -> local
lrwxrwxrwx 1 root root 1 Jul 8 12:52 192.168.122.1 -> 0
-rw-r--r-- 1 root root 13 Jan 20 2016 local
[root@sdfdsf peers]#
I feel it misses a
ln -s local 127.0.0 to have an effect on 127.0.0.2, 127.0.0.3 ...
-
I feel it misses a
ln -s local 127.0.0 to have an effect on 127.0.0.2, 127.0.0.3 ...
That's exactly what we don't want, since we want to prevent relaying via 127.0.0.2 (etc), not allow it.
-
then ln -s 0 127.0.0 ?
-
but still if we have 127.0.0.200 for fetchmail running do you consider it local or remote ?
-
I have several info to update on BUG... but need to know how to remove this message from queue.
Can someone post a howto here ?
The message has more than 21MB right now!
21238600 (9, 9/21238600)
Return-path: gcontratos@xxxxx.com.br
From: Renata Bassoa <gcontratos@xxxx.com.br>
To: destinatarios-nao-revelados: ;
Subject: =?UTF-8?Q?Pesquisa_de_satisfa=c3=a7=c3=a3o_-_1=c2=ba_Semestre_2016?=
Date: Mon, 18 Jul 2016 08:53:25 -0300
Size: 21744867 bytes
[root@andorinha ~]# cat /var/qmail/queue/mess/22/21237394|grep X-Virus-Checked|wc -l
53702
[root@andorinha ~]#
-
First backup the message to get latter the headers to know how it started before the 50000 loop
With qmail running
Service qpsmtpd stop
Then go to server manager and check for the qmail queue with qmail handle contrib
Select it and delete it, repeat it as many time as needed if you have more than one
Service qpsmtpd start
-
Jean-Philippe
Thank you by pointing the qpsmtpd daemon... I was getting crazy with qmail.kkk
I´ve stopped qpsmtpd, copy message parts (3of them) in a tar file and now can upload in bug.
Thank you
Thank you
Thank you
Regards
Jáder