Koozali.org: home of the SME Server

Squid

Offline david000

  • ****
  • 193
  • +0/-0
Squid
« on: February 22, 2022, 10:59:45 AM »
Code: [Select]
Subject:  Cron <root@sme10> squid -k rotate

squid: ERROR: Could not send signal 10 to process 1849: (3) No such process

After the last couple of server updates I'm seeing Cron squid email alerts.   A quick google suggested a restart should fix it, which it did on the first occasion. I need to restart again to fix the current one.

I wondered how to ID the root cause  ? 

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Squid
« Reply #1 on: February 24, 2022, 03:46:10 PM »
We know there are issue with logging.

JP had been doing a huge rewrite of the logging code whist working stupid hours a day in hospital and with two young children.

https://bugs.koozali.org/show_bug.cgi?id=10484
https://bugs.koozali.org/show_bug.cgi?id=946
https://bugs.koozali.org/show_bug.cgi?id=11873
https://bugs.koozali.org/show_bug.cgi?id=11867
https://bugs.koozali.org/show_bug.cgi?id=11866
https://bugs.koozali.org/show_bug.cgi?id=11813

And probably some others I have missed. All of this will be coming out soon. Currently I think ka lot of rpms are in testing - please feel free to try them on a TEST VM and report back.

You should REALLY follow all bugs to keep abreast of thing via Bugzilla, some of the wiki bug pages, or the bug mailing list:

https://lists.contribs.org/mailman/listinfo/bugteam

...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #2 on: February 24, 2022, 04:33:52 PM »
Thanks, ReetP.  I'll fire up the VM and have a look at testing some RPMs when I can get to it.

Best


David

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Squid
« Reply #3 on: February 24, 2022, 08:36:03 PM »
Currently have a sytem running with all the latest logging updates, test box only at this stage, so far so good, hasnt blown up as yet :-) still a work in progress, please jump in and test more hands etc
--
qui scribit bis legit

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Squid
« Reply #4 on: February 25, 2022, 12:20:32 AM »
squid error is probably not related.  it is more related to log rotate. 
just noise.
unless you get issue like unable to browse internet after the cron email. 


speaking of updates it works smoothly. probably needs to add few more filter because systemd spams a lot to messages now :)

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #5 on: February 25, 2022, 10:24:51 AM »
squid error is probably not related.  it is more related to log rotate. 
just noise.
unless you get issue like unable to browse internet after the cron email. 


speaking of updates it works smoothly. probably needs to add few more filter because systemd spams a lot to messages now :)

For my implementation we only collect mail and view locally using the web mail. We also use Ibays for small bits of user storage. I'm not seeing any issues in terms of functionality.

On the updates, they seem to work smoothly with the exception where a couple of times it seems to have hung during the server manager install, where IIRC it was doing a php update (maybe).  There was a note on here which mentioned it from a different user.  If it happens again I'll note how far it gets before I restart\reconfigure.



 

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Squid
« Reply #6 on: February 25, 2022, 01:11:03 PM »
so just noise in your case. 
note however if sme server is your gateway by default it is transparent proxying your web browsing so it squid fails you might have issue browsing.

manager update is not that reliable currently because of added security level over the years like csrf tokens, session timeout, and probably httpd restart following php or web related rpms. few bugs are open about that. 

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #7 on: February 25, 2022, 01:20:48 PM »
On updates is it better not use the server manager and instead go with #yum update from Putty ?

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Squid
« Reply #8 on: February 25, 2022, 01:41:53 PM »
yes. 

probably future will use an update method similar to what proxmox do: a webconsole with the command running on loading. 

Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #9 on: March 04, 2022, 08:00:41 PM »
That was the error that alerted me to the squid problem I had a couple of weeks ago - squid wouldn't start at all because of a squid.pid file that had been left behind for some reason.

It's interesting that you are having the problem of having to restart squid after doing updates. On my own machine I have the same problem, but with dovecot and fail2ban. I've not had time to investigate any further, though I'm sure I came across a post somewhere that said that fail2ban would not start after an update.
Jim

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Squid
« Reply #10 on: March 05, 2022, 02:53:36 AM »
That was the error that alerted me to the squid problem I had a couple of weeks ago - squid wouldn't start at all because of a squid.pid file that had been left behind for some reason.


here this is not just log noise but we have something that actually fails. 

any template custom for squid ?
any change tot he default squid db values ?


Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #11 on: March 05, 2022, 03:30:30 PM »
here this is not just log noise but we have something that actually fails. 

any template custom for squid ?
any change tot he default squid db values ?

It was a clean install on a new server, and I had just completed a restore from backup made from the Server Manager. The first sign of a problem was that Opera and Edge browsers couldn't connect, but Firefox worked fine. Then I got the error email the next morning, and on further investigation found that squid had not started.

After a bit of googling I found that this could be caused by a squid.pid file, and I eventually found it, deleted it, and everything has been fine since. Squid started OK on the next reboot. However I've not done any updates yet, so that will be the next test. The thread is here https://forums.koozali.org/index.php/topic,54768.0.html

The problem on my machine is dovecot not starting after installing updates - as I said, I've not had time to do any diagnostics, or look at log files to see what is going on. I've not even had time to see if it will start after a straight reboot. Fail2ban doesn't start either, but I'm sure I've seen a thread on that elsewhere.

Jim

Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Squid
« Reply #12 on: March 05, 2022, 03:47:43 PM »
let’s use the same wording.

if there is a restore this is not a clean install.

Would you say this is a scoop of unflavored ice cream mixed with chocolate and mint flavors ?

So you have a SME 10 with workstation restore from your v9 and squid was not working.  Symptoms were browsers which were configured to use the proxy were not able to display any website and in place were displaying a network access error.
While a browser not configured to use the proxy was fine browsing the internet. 

First question have you issued the post-upgrade and rebooted after your restored ?
if not this is normal to have all expected system down.

Second question, what returns /sbin/e-smith/templates

Third question, what returns config show squid

Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #13 on: March 05, 2022, 03:54:23 PM »
let’s use the same wording.

if there is a restore this is not a clean install.

Would you say this is a scoop of unflavored ice cream mixed with chocolate and mint flavors ?

So you have a SME 10 with workstation restore from your v9 and squid was not working.  Symptoms were browsers which were configured to use the proxy were not able to display any website and in place were displaying a network access error.
While a browser not configured to use the proxy was fine browsing the internet. 

First question have you issued the post-upgrade and rebooted after your restored ?
if not this is normal to have all expected system down.

Second question, what returns /sbin/e-smith/templates

Third question, what returns config show squid

I stand corrected, you are right, it wasn't a clean install. In my defence I had followed the sequence required to restore a new server to the letter. You were very good in helping me some time ago on this, and gave me detailed instructions.

Question 1: Yes, I had done signal-event post-upgrade; signal-event reboot from the command line on the server, twice.

I won't have the answer to either of the other questions for a few days, as the machine is remote, and although I can access the server manager, I cannot access the server using a terminal.
Jim

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #14 on: March 06, 2022, 10:14:31 PM »
An update from my side,

updating via yum update seems to have been more effective and the squid emails have gone away.

Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #15 on: March 22, 2022, 07:13:11 PM »

Second question, what returns /sbin/e-smith/templates

Third question, what returns config show squid

Question 2:
Code: [Select]
[root@sme ~]# /sbin/e-smith/templates
-bash: /sbin/e-smith/templates: No such file or directory

Question 3:
Code: [Select]
[root@sme ~]# config show squid
squid=service
    EnforceSafePorts=no
    SafePorts=21,70,80,81,119,210,443,563,980,1024-65535
    TCPPort=3128
    TCPProxyPort=80:3128
    TransparentPort=3128
    access=private
    status=enabled

It all seems to be working fine now. I have run updates from the command line, and they all went through fine. Squid is running normally.
Jim

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Squid
« Reply #16 on: March 22, 2022, 08:19:40 PM »
Try

Code: [Select]
/sbin/e-smith/audittools/templates
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #17 on: April 19, 2022, 02:20:04 PM »
OK, this has just happened on my own server after an update from the server manager:
Code: [Select]
UNIT                        LOAD   ACTIVE SUB     DESCRIPTION
  acpid.service               loaded active running ACPI Event Daemon
  atd.service                 loaded active running Job spooling tools
  auditd.service              loaded active running Security Auditing Service
  blk-availability.service    loaded active exited  Availability of block device
  clamd.service               loaded active running clamd scanner () daemon
  crond.service               loaded active running Command Scheduler
  cvm-unix.service            loaded active running Credential Validation Module
  dbus.service                loaded active running D-Bus System Message Bus
  dhcpd.service               loaded active running DHCPv4 Server Daemon
  dnscache.forwarder.service  loaded active exited  dnscache.forwarder,
  dnscache.service            loaded active exited  dnscache,
  dovecot.service             loaded active running Dovecot IMAP/POP3 email serv
  fail2ban.service            loaded active running Fail2Ban Service
  freshclam.service           loaded active running ClamAV virus database update
  getty@tty1.service          loaded active running Getty on tty1
  gssproxy.service            loaded active running GSSAPI Proxy Daemon
  httpd-admin.service         loaded active running httpd-admin The Koozali SME
  httpd-e-smith.service       loaded active running httpd-e-smith The Koozali SM
  irqbalance.service          loaded active running irqbalance daemon
  kmod-static-nodes.service   loaded active exited  Create list of required stat
  ldap.init.service           loaded active exited  Koozali SME Server ldap.init
  ldap.service                loaded active running Koozali SME Server OpenLDAP
  local.service               loaded active exited  Local service for Koozali SM
  lpd.service                 loaded active exited  lpd,  LPRng print spool
  lvm2-lvmetad.service        loaded active running LVM2 metadata daemon
  lvm2-monitor.service        loaded active exited  Monitoring of LVM2 mirrors,
  lvm2-pvscan@9:1.service     loaded active exited  LVM2 PV scan on device 9:1
  mariadb.service             loaded active running MariaDB database server
  masq.service                loaded active exited  masq, the Koozali SME Server
  mdmonitor.service           loaded active running Software RAID monitoring and
  mysql.init.service          loaded active exited  Koozali SME Server mysql DB
  networking.service          loaded active exited  Network management for Kooza
  nmbd.service                loaded active running nmbd.service
  ntpd.service                loaded active running Network Time Service
  php-fpm.service             loaded active running The PHP FastCGI Process Mana
  php55-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php56-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php70-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php71-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php72-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php73-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php74-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php80-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  php81-php-fpm.service       loaded active running The PHP FastCGI Process Mana
  qmail.service               loaded active exited  qmail, message transfer agen
  qpsmtpd.service             loaded active exited  qpsmtpd
  radiusd.service             loaded active running FreeRADIUS high performance
  rc-local.service            loaded active exited  /etc/rc.d/rc.local Compatibi
  rhel-dmesg.service          loaded active exited  Dump dmesg to /var/log/dmesg
  rhel-domainname.service     loaded active exited  Read and set NIS domainname
  rhel-loadmodules.service    loaded active exited  Load legacy module configura
  rhel-readonly.service       loaded active exited  Configure read-only root sup
  rsyslog.service             loaded active running System Logging Service
  runit.service               loaded active running Process Supervising Daemon
  smartd.service              loaded active running Self Monitoring and Reportin
  smb.service                 loaded active exited  Samba SMB Daemon global serv
  smbd.service                loaded active running Samba SMB Daemon
  smtp-auth-proxy.service     loaded active running Koozali SME Server SMTP auth
  spamassassin.service        loaded active running Spamassassin daemon
  sqpsmtpd.service            loaded active exited  sqpsmtpd
● squid.service               loaded failed failed  Squid caching proxy
  sshd.service                loaded active running OpenSSH server daemon
  systemd-journal-flush.service loaded active exited  Flush Journal to Persisten
  systemd-journald.service    loaded active running Journal Service
  systemd-logind.service      loaded active running Login Service
  systemd-random-seed.service loaded active exited  Load/Save Random Seed
  systemd-readahead-collect.service loaded active exited  Collect Read-Ahead Dat
  systemd-readahead-replay.service loaded active exited  Replay Read-Ahead Data
[root@botham ~]# systemctl start squid
Job for squid.service failed because the control process exited with error code. See "systemctl status squid.service" and "journalctl -xe" for details.

I then ran journalctl -xe, and got:
Code: [Select]
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: Unit squid.service entered failed state.
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: squid.service failed.
Apr 19 13:02:58 botham.blackfingernail.co.uk ntpd[967]: 0.0.0.0 0613 03 spike_detect -0.403493 s
lines 2407-2429/2429 (END)
Apr 19 13:00:46 botham.blackfingernail.co.uk fail2ban-server[5019]: Server ready
Apr 19 13:01:01 botham.blackfingernail.co.uk crond[5057]: pam_unix(crond:session): session opened for user root by (uid=0)
Apr 19 13:01:01 botham.blackfingernail.co.uk CROND[5058]: (root) CMD (run-parts /etc/cron.hourly)
Apr 19 13:01:01 botham.blackfingernail.co.uk CROND[5057]: pam_unix(crond:session): session closed for user root
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: Starting Squid caching proxy...
-- Subject: Unit squid.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit squid.service has begun starting up.
Apr 19 13:01:19 botham.blackfingernail.co.uk squid[5105]: squid: ERROR: Could not send signal 15 to process 2270: (1) Operation
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: squid.service: control process exited, code=exited status=1
Apr 19 13:01:19 botham.blackfingernail.co.uk systemd[1]: Failed to start Squid caching proxy.
-- Subject: Unit squid.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit squid.service has failed.
--
-- The result is failed.

Based on what I did last time on the other server, I went to /var/log/squid and looked for a squid.pid file:
Code: [Select]
[root@botham ~]# cd /var/log/squid
[root@botham squid]# ls -l
total 10388
-rw-r----- 1 squid squid 1394149 Apr 19 12:55 access.log
-rw-r----- 1 squid squid 4596767 Mar 20 03:34 access.log-20220320.gz
-rw-r----- 1 squid squid  512879 Mar 27 03:19 access.log-20220327.gz
-rw-r----- 1 squid squid 2210238 Apr  3 03:21 access.log-20220403.gz
-rw-r----- 1 squid squid  704405 Apr 10 03:27 access.log-20220410.gz
-rw-r----- 1 squid squid 1102692 Apr 17 03:05 access.log-20220417.gz
-rw-r----- 1 squid squid   14625 Apr 19 13:01 cache.log
-rw-r----- 1 squid squid   14487 Mar 20 01:35 cache.log-20220320.gz
-rw-r----- 1 squid squid   15707 Mar 27 03:19 cache.log-20220327.gz
-rw-r----- 1 squid squid    6766 Apr  3 02:46 cache.log-20220403.gz
-rw-r----- 1 squid squid   13266 Apr  9 18:59 cache.log-20220410.gz
-rw-r----- 1 squid squid    1804 Apr 17 02:54 cache.log-20220417.gz
-rw------- 1 root  root      112 Jan 29 12:54 squid.log-20220130.gz
-rw------- 1 root  root      113 Feb  8 10:42 squid.log-20220209.gz
-rw------- 1 root  root      137 Mar  4 10:08 squid.log-20220305.gz
-rw------- 1 root  root      113 Mar 23 09:03 squid.log-20220324.gz
-rw------- 1 root  root      113 Apr 12 07:56 squid.log-20220413.gz
-rw-r--r-- 1 root  squid       5 Apr 12 07:56 squid.pid

I ran the command rm squid.pid, and then restarted squid with no problems.

Now, I'm also having problems with dovecot not starting up after an update, and also fail2ban. I know there have been problems with fail2ban, and I only have the problem with dovecot on my own machine.

I'm busy this afternoon, but will raise a bug as soon as I get the time. You'll all have to bear with me though, as things are a bit hectic at the moment. Sorry.
Jim

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Squid
« Reply #18 on: April 19, 2022, 04:06:31 PM »
There are some huge changes in the logging system under development right now - I think that may be at heart of your issues.

Before you open a bug please read everything on this bug, and the linked ones as well:

https://bugs.koozali.org/show_bug.cgi?id=11403

If you talk to Terry on Rocket he can give you some guidance on helping test this.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #19 on: April 19, 2022, 07:19:59 PM »
Thanks John, I'll have a look. Though I'm not sure that mine is a logging issue as such, but I'll read and speak to Terry first.

Cheers,
Jim

Offline ReetP

  • *
  • 3,722
  • +5/-0
Re: Squid
« Reply #20 on: April 19, 2022, 07:42:46 PM »
I think that the hanging pid is likely related to logs rotating and services not restarting correctly, hence a logging issue.

This has been a long standing issue but a massive amount of work and JP has only just got round to it - he's wrestling with really important exams right now.

It will be worth testing it first as I suspect that may eliminate at least some of the issues.
...
1. Read the Manual
2. Read the Wiki
3. Don't ask for support on Unsupported versions of software
4. I have a job, wife, and kids and do this in my spare time. If you want something fixed, please help.

Bugs are easier than you think: http://wiki.contribs.org/Bugzilla_Help

If you love SME and don't want to lose it, join in: http://wiki.contribs.org/Koozali_Foundation

Offline Peasant

  • *
  • 143
  • +2/-0
Re: Squid
« Reply #21 on: April 19, 2022, 07:58:55 PM »
That would make sense thanks. OK, I'll see what I can do over the next few weeks. Been a bit busy farmering again...
Jim

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #22 on: August 31, 2022, 10:17:53 AM »
A quick update. I'm seeing "Cron  squid -k rotate" email alerts again after the last update\restart.

Code: [Select]
"squid: ERROR: Could not send signal 10 to process 1872: (3) No such process"


Offline Jean-Philippe Pialasse

  • *
  • 2,743
  • +11/-0
  • aka Unnilennium
    • http://smeserver.pialasse.com
Re: Squid
« Reply #23 on: August 31, 2022, 02:21:03 PM »
funny thing is the script should not send error as it is redirected to /dev/null

probably this error is redirected to stdout would need to add &1>2


Code: [Select]
postrotate
      # Asks squid to reopen its logs. (logfile_rotate 0 is set in squid.conf)
      # errors redirected to make it silent if squid is not running
      /usr/sbin/squid -k rotate 2>/dev/null
      # Wait a little to allow Squid to catch up before the logs is compressed
      sleep 1
    endscript


however the real issue is squid not validating its pid file.


we could update the /etc/squid/squid.conf template so pid file moves

pid_filename /var/log/squid/squid.pid
to
pid_filename /var/run/squid/squid.pid
or better
pid_filename /run/squid/squid.pid


this will at least remove those when squid pid was not deleted on reboot. 

we could template the logrotate.d/squid to add the &1>2

adding
PIDFile= to the dropin for squid.service might also help there by cleaning it of pid not belonging to the right process

see https://bugs.koozali.org/show_bug.cgi?id=11454

« Last Edit: August 31, 2022, 02:24:21 PM by Jean-Philippe Pialasse »

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #24 on: October 12, 2022, 10:11:06 AM »
I've added a brief note to the bug report, but the error has gone away for me in recent weeks.

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #25 on: February 13, 2023, 10:08:29 AM »
It's back after the last yum update, coincidentally alongside an email saying the bug was closed.  I've added a note in the bug report.

ah, Just noticed that Terry has seen it.

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Squid
« Reply #26 on: February 13, 2023, 10:59:42 AM »
the updated package is still in smeupdates-testing  repo  /smeupdates-testing/x86_64/RPMS/e-smith-proxy-5.6.0-12.el7.sme.noarch.rpm

what version of e-smith-proxy do you currently have installed

# rpm -q e-smith-proxy

just getting all the ducks lined up and this and other updates will be moved into the updates repo..if you want you can update from updates-testing being fully aware it IS smeupdates-testing :-)
--
qui scribit bis legit

Offline david000

  • ****
  • 193
  • +0/-0
Re: Squid
« Reply #27 on: February 13, 2023, 11:26:46 AM »
what version of e-smith-proxy do you currently have installed

# rpm -q e-smith-proxy

Cheers Terry,

e-smith-proxy-5.6.0-11.el7.sme.noarch

It's not a problem, I'll wait on the update and see how it goes. 

Offline TerryF

  • grumpy old man
  • *
  • 1,821
  • +6/-0
Re: Squid
« Reply #28 on: February 13, 2023, 01:17:51 PM »
ta thanks, yep update coming shortly..
--
qui scribit bis legit