Koozali.org: home of the SME Server
Contribs.org Forums => Koozali SME Server 10.x => Topic started by: david000 on February 22, 2022, 10:59:45 AM
-
Subject: Cron <root@sme10> squid -k rotate
squid: ERROR: Could not send signal 10 to process 1849: (3) No such process
After the last couple of server updates I'm seeing Cron squid email alerts. A quick google suggested a restart should fix it, which it did on the first occasion. I need to restart again to fix the current one.
I wondered how to ID the root cause ?
-
We know there are issue with logging.
JP had been doing a huge rewrite of the logging code whist working stupid hours a day in hospital and with two young children.
https://bugs.koozali.org/show_bug.cgi?id=10484
https://bugs.koozali.org/show_bug.cgi?id=946
https://bugs.koozali.org/show_bug.cgi?id=11873
https://bugs.koozali.org/show_bug.cgi?id=11867
https://bugs.koozali.org/show_bug.cgi?id=11866
https://bugs.koozali.org/show_bug.cgi?id=11813
And probably some others I have missed. All of this will be coming out soon. Currently I think ka lot of rpms are in testing - please feel free to try them on a TEST VM and report back.
You should REALLY follow all bugs to keep abreast of thing via Bugzilla, some of the wiki bug pages, or the bug mailing list:
https://lists.contribs.org/mailman/listinfo/bugteam
-
Thanks, ReetP. I'll fire up the VM and have a look at testing some RPMs when I can get to it.
Best
David
-
Currently have a sytem running with all the latest logging updates, test box only at this stage, so far so good, hasnt blown up as yet :-) still a work in progress, please jump in and test more hands etc
-
squid error is probably not related. it is more related to log rotate.
just noise.
unless you get issue like unable to browse internet after the cron email.
speaking of updates it works smoothly. probably needs to add few more filter because systemd spams a lot to messages now :)
-
squid error is probably not related. it is more related to log rotate.
just noise.
unless you get issue like unable to browse internet after the cron email.
speaking of updates it works smoothly. probably needs to add few more filter because systemd spams a lot to messages now :)
For my implementation we only collect mail and view locally using the web mail. We also use Ibays for small bits of user storage. I'm not seeing any issues in terms of functionality.
On the updates, they seem to work smoothly with the exception where a couple of times it seems to have hung during the server manager install, where IIRC it was doing a php update (maybe). There was a note on here which mentioned it from a different user. If it happens again I'll note how far it gets before I restart\reconfigure.
-
so just noise in your case.
note however if sme server is your gateway by default it is transparent proxying your web browsing so it squid fails you might have issue browsing.
manager update is not that reliable currently because of added security level over the years like csrf tokens, session timeout, and probably httpd restart following php or web related rpms. few bugs are open about that.
-
On updates is it better not use the server manager and instead go with #yum update from Putty ?
-
yes.
probably future will use an update method similar to what proxmox do: a webconsole with the command running on loading.
-
That was the error that alerted me to the squid problem I had a couple of weeks ago - squid wouldn't start at all because of a squid.pid file that had been left behind for some reason.
It's interesting that you are having the problem of having to restart squid after doing updates. On my own machine I have the same problem, but with dovecot and fail2ban. I've not had time to investigate any further, though I'm sure I came across a post somewhere that said that fail2ban would not start after an update.
-
That was the error that alerted me to the squid problem I had a couple of weeks ago - squid wouldn't start at all because of a squid.pid file that had been left behind for some reason.
here this is not just log noise but we have something that actually fails.
any template custom for squid ?
any change tot he default squid db values ?
-
here this is not just log noise but we have something that actually fails.
any template custom for squid ?
any change tot he default squid db values ?
It was a clean install on a new server, and I had just completed a restore from backup made from the Server Manager. The first sign of a problem was that Opera and Edge browsers couldn't connect, but Firefox worked fine. Then I got the error email the next morning, and on further investigation found that squid had not started.
After a bit of googling I found that this could be caused by a squid.pid file, and I eventually found it, deleted it, and everything has been fine since. Squid started OK on the next reboot. However I've not done any updates yet, so that will be the next test. The thread is here https://forums.koozali.org/index.php/topic,54768.0.html (https://forums.koozali.org/index.php/topic,54768.0.html)
The problem on my machine is dovecot not starting after installing updates - as I said, I've not had time to do any diagnostics, or look at log files to see what is going on. I've not even had time to see if it will start after a straight reboot. Fail2ban doesn't start either, but I'm sure I've seen a thread on that elsewhere.
-
let’s use the same wording.
if there is a restore this is not a clean install.
Would you say this is a scoop of unflavored ice cream mixed with chocolate and mint flavors ?
So you have a SME 10 with workstation restore from your v9 and squid was not working. Symptoms were browsers which were configured to use the proxy were not able to display any website and in place were displaying a network access error.
While a browser not configured to use the proxy was fine browsing the internet.
First question have you issued the post-upgrade and rebooted after your restored ?
if not this is normal to have all expected system down.
Second question, what returns /sbin/e-smith/templates
Third question, what returns config show squid
-
let’s use the same wording.
if there is a restore this is not a clean install.
Would you say this is a scoop of unflavored ice cream mixed with chocolate and mint flavors ?
So you have a SME 10 with workstation restore from your v9 and squid was not working. Symptoms were browsers which were configured to use the proxy were not able to display any website and in place were displaying a network access error.
While a browser not configured to use the proxy was fine browsing the internet.
First question have you issued the post-upgrade and rebooted after your restored ?
if not this is normal to have all expected system down.
Second question, what returns /sbin/e-smith/templates
Third question, what returns config show squid
I stand corrected, you are right, it wasn't a clean install. In my defence I had followed the sequence required to restore a new server to the letter. You were very good in helping me some time ago on this, and gave me detailed instructions.
Question 1: Yes, I had done signal-event post-upgrade; signal-event reboot from the command line on the server, twice.
I won't have the answer to either of the other questions for a few days, as the machine is remote, and although I can access the server manager, I cannot access the server using a terminal.
-
An update from my side,
updating via yum update seems to have been more effective and the squid emails have gone away.
-
Second question, what returns /sbin/e-smith/templates
Third question, what returns config show squid
Question 2:
[root@sme ~]# /sbin/e-smith/templates
-bash: /sbin/e-smith/templates: No such file or directory
Question 3:
[root@sme ~]# config show squid
squid=service
EnforceSafePorts=no
SafePorts=21,70,80,81,119,210,443,563,980,1024-65535
TCPPort=3128
TCPProxyPort=80:3128
TransparentPort=3128
access=private
status=enabled
It all seems to be working fine now. I have run updates from the command line, and they all went through fine. Squid is running normally.
-
Try
/sbin/e-smith/audittools/templates
-
OK, this has just happened on my own server after an update from the server manager:
UNIT LOAD ACTIVE SUB DESCRIPTION
acpid.service loaded active running ACPI Event Daemon
atd.service loaded active running Job spooling tools
auditd.service loaded active running Security Auditing Service
blk-availability.service loaded active exited Availability of block device
clamd.service loaded active running clamd scanner () daemon
crond.service loaded active running Command Scheduler
cvm-unix.service loaded active running Credential Validation Module
dbus.service loaded active running D-Bus System Message Bus
dhcpd.service loaded active running DHCPv4 Server Daemon
dnscache.forwarder.service loaded active exited dnscache.forwarder,
dnscache.service loaded active exited dnscache,
dovecot.service loaded active running Dovecot IMAP/POP3 email serv
fail2ban.service loaded active running Fail2Ban Service
freshclam.service loaded active running ClamAV virus database update
getty@tty1.service loaded active running Getty on tty1
gssproxy.service loaded active running GSSAPI Proxy Daemon
httpd-admin.service loaded active running httpd-admin The Koozali SME
httpd-e-smith.service loaded active running httpd-e-smith The Koozali SM
irqbalance.service loaded active running irqbalance daemon
kmod-static-nodes.service loaded active exited Create list of required stat
ldap.init.service loaded active exited Koozali SME Server ldap.init
ldap.service loaded active running Koozali SME Server OpenLDAP
local.service loaded active exited Local service for Koozali SM
lpd.service loaded active exited lpd, LPRng print spool
lvm2-lvmetad.service loaded active running LVM2 metadata daemon
lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors,
lvm2-pvscan@9:1.service loaded active exited LVM2 PV scan on device 9:1
mariadb.service loaded active running MariaDB database server
masq.service loaded active exited masq, the Koozali SME Server
mdmonitor.service loaded active running Software RAID monitoring and
mysql.init.service loaded active exited Koozali SME Server mysql DB
networking.service loaded active exited Network management for Kooza
nmbd.service loaded active running nmbd.service
ntpd.service loaded active running Network Time Service
php-fpm.service loaded active running The PHP FastCGI Process Mana
php55-php-fpm.service loaded active running The PHP FastCGI Process Mana
php56-php-fpm.service loaded active running The PHP FastCGI Process Mana
php70-php-fpm.service loaded active running The PHP FastCGI Process Mana
php71-php-fpm.service loaded active running The PHP FastCGI Process Mana
php72-php-fpm.service loaded active running The PHP FastCGI Process Mana
php73-php-fpm.service loaded active running The PHP FastCGI Process Mana
php74-php-fpm.service loaded active running The PHP FastCGI Process Mana
php80-php-fpm.service loaded active running The PHP FastCGI Process Mana
php81-php-fpm.service loaded active running The PHP FastCGI Process Mana
qmail.service loaded active exited qmail, message transfer agen
qpsmtpd.service loaded active exited qpsmtpd
radiusd.service loaded active running FreeRADIUS high performance
rc-local.service loaded active exited /etc/rc.d/rc.local Compatibi
rhel-dmesg.service loaded active exited Dump dmesg to /var/log/dmesg
rhel-domainname.service loaded active exited Read and set NIS domainname
rhel-loadmodules.service loaded active exited Load legacy module configura
rhel-readonly.service loaded active exited Configure read-only root sup
rsyslog.service loaded active running System Logging Service
runit.service loaded active running Process Supervising Daemon
smartd.service loaded active running Self Monitoring and Reportin
smb.service loaded active exited Samba SMB Daemon global serv
smbd.service loaded active running Samba SMB Daemon
smtp-auth-proxy.service loaded active running Koozali SME Server SMTP auth
spamassassin.service loaded active running Spamassassin daemon
sqpsmtpd.service loaded active exited sqpsmtpd
● squid.service loaded failed failed Squid caching proxy
sshd.service loaded active running OpenSSH server daemon
systemd-journal-flush.service loaded active exited Flush Journal to Persisten
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-random-seed.service loaded active exited Load/Save Random Seed
systemd-readahead-collect.service loaded active exited Collect Read-Ahead Dat
systemd-readahead-replay.service loaded active exited Replay Read-Ahead Data
[root@botham ~]# systemctl start squid
Job for squid.service failed because the control process exited with error code. See "systemctl status squid.service" and "journalctl -xe" for details.
I then ran journalctl -xe, and got:
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: Unit squid.service entered failed state.
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: squid.service failed.
Apr 19 13:02:58 botham.blackfingernail.co.uk ntpd[967]: 0.0.0.0 0613 03 spike_detect -0.403493 s
lines 2407-2429/2429 (END)
Apr 19 13:00:46 botham.blackfingernail.co.uk fail2ban-server[5019]: Server ready
Apr 19 13:01:01 botham.blackfingernail.co.uk crond[5057]: pam_unix(crond:session): session opened for user root by (uid=0)
Apr 19 13:01:01 botham.blackfingernail.co.uk CROND[5058]: (root) CMD (run-parts /etc/cron.hourly)
Apr 19 13:01:01 botham.blackfingernail.co.uk CROND[5057]: pam_unix(crond:session): session closed for user root
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: Starting Squid caching proxy...
-- Subject: Unit squid.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit squid.service has begun starting up.
Apr 19 13:01:19 botham.blackfingernail.co.uk squid[5105]: squid: ERROR: Could not send signal 15 to process 2270: (1) Operation
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: squid.service: control process exited, code=exited status=1
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: Failed to start Squid caching proxy.
-- Subject: Unit squid.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit squid.service has failed.
--
-- The result is failed.
Based on what I did last time on the other server, I went to /var/log/squid and looked for a squid.pid file:
[root@botham ~]# cd /var/log/squid
[root@botham squid]# ls -l
total 10388
-rw-r----- 1 squid squid 1394149 Apr 19 12:55 access.log
-rw-r----- 1 squid squid 4596767 Mar 20 03:34 access.log-20220320.gz
-rw-r----- 1 squid squid 512879 Mar 27 03:19 access.log-20220327.gz
-rw-r----- 1 squid squid 2210238 Apr 3 03:21 access.log-20220403.gz
-rw-r----- 1 squid squid 704405 Apr 10 03:27 access.log-20220410.gz
-rw-r----- 1 squid squid 1102692 Apr 17 03:05 access.log-20220417.gz
-rw-r----- 1 squid squid 14625 Apr 19 13:01 cache.log
-rw-r----- 1 squid squid 14487 Mar 20 01:35 cache.log-20220320.gz
-rw-r----- 1 squid squid 15707 Mar 27 03:19 cache.log-20220327.gz
-rw-r----- 1 squid squid 6766 Apr 3 02:46 cache.log-20220403.gz
-rw-r----- 1 squid squid 13266 Apr 9 18:59 cache.log-20220410.gz
-rw-r----- 1 squid squid 1804 Apr 17 02:54 cache.log-20220417.gz
-rw------- 1 root root 112 Jan 29 12:54 squid.log-20220130.gz
-rw------- 1 root root 113 Feb 8 10:42 squid.log-20220209.gz
-rw------- 1 root root 137 Mar 4 10:08 squid.log-20220305.gz
-rw------- 1 root root 113 Mar 23 09:03 squid.log-20220324.gz
-rw------- 1 root root 113 Apr 12 07:56 squid.log-20220413.gz
-rw-r--r-- 1 root squid 5 Apr 12 07:56 squid.pid
I ran the command rm squid.pid, and then restarted squid with no problems.
Now, I'm also having problems with dovecot not starting up after an update, and also fail2ban. I know there have been problems with fail2ban, and I only have the problem with dovecot on my own machine.
I'm busy this afternoon, but will raise a bug as soon as I get the time. You'll all have to bear with me though, as things are a bit hectic at the moment. Sorry.
-
There are some huge changes in the logging system under development right now - I think that may be at heart of your issues.
Before you open a bug please read everything on this bug, and the linked ones as well:
https://bugs.koozali.org/show_bug.cgi?id=11403
If you talk to Terry on Rocket he can give you some guidance on helping test this.
-
Thanks John, I'll have a look. Though I'm not sure that mine is a logging issue as such, but I'll read and speak to Terry first.
Cheers,
-
I think that the hanging pid is likely related to logs rotating and services not restarting correctly, hence a logging issue.
This has been a long standing issue but a massive amount of work and JP has only just got round to it - he's wrestling with really important exams right now.
It will be worth testing it first as I suspect that may eliminate at least some of the issues.
-
That would make sense thanks. OK, I'll see what I can do over the next few weeks. Been a bit busy farmering again...
-
A quick update. I'm seeing "Cron squid -k rotate" email alerts again after the last update\restart.
"squid: ERROR: Could not send signal 10 to process 1872: (3) No such process"
-
funny thing is the script should not send error as it is redirected to /dev/null
probably this error is redirected to stdout would need to add &1>2
postrotate
# Asks squid to reopen its logs. (logfile_rotate 0 is set in squid.conf)
# errors redirected to make it silent if squid is not running
/usr/sbin/squid -k rotate 2>/dev/null
# Wait a little to allow Squid to catch up before the logs is compressed
sleep 1
endscript
however the real issue is squid not validating its pid file.
we could update the /etc/squid/squid.conf template so pid file moves
pid_filename /var/log/squid/squid.pid
to
pid_filename /var/run/squid/squid.pid
or better
pid_filename /run/squid/squid.pid
this will at least remove those when squid pid was not deleted on reboot.
we could template the logrotate.d/squid to add the &1>2
adding
PIDFile= to the dropin for squid.service might also help there by cleaning it of pid not belonging to the right process
see https://bugs.koozali.org/show_bug.cgi?id=11454
-
I've added a brief note to the bug report, but the error has gone away for me in recent weeks.
-
It's back after the last yum update, coincidentally alongside an email saying the bug was closed. I've added a note in the bug report.
ah, Just noticed that Terry has seen it.
-
the updated package is still in smeupdates-testing repo /smeupdates-testing/x86_64/RPMS/e-smith-proxy-5.6.0-12.el7.sme.noarch.rpm
what version of e-smith-proxy do you currently have installed
# rpm -q e-smith-proxy
just getting all the ducks lined up and this and other updates will be moved into the updates repo..if you want you can update from updates-testing being fully aware it IS smeupdates-testing :-)
-
what version of e-smith-proxy do you currently have installed
# rpm -q e-smith-proxy
Cheers Terry,
e-smith-proxy-5.6.0-11.el7.sme.noarch
It's not a problem, I'll wait on the update and see how it goes.
-
ta thanks, yep update coming shortly..