Topic: Squid Fatal

[derekb]

Hi

We are having the below problem with i believe squid (HTTP Caching on our SME server below version)

Code: [Select]

SME Server 8.1
Copyright 1999-2006 Mitel Corporation
All rights reserved.
Copyright (C) 2013 Koozali Foundation, Inc.

Code: [Select]

FATAL: logfileWrite: /var/log/squid/access.log: (5) Input/output error
Squid Cache (Version 2.6.STABLE21): Terminated abnormally.
CPU Usage: 0.123 seconds = 0.069 user + 0.054 sys
Maximum Resident Size: 25888 KB
Page faults with physical i/o: 10
Memory usage for squid via mallinfo():
total space in arena:    3168 KB
Ordinary blocks:         3150 KB      3 blks
Small blocks:               0 KB      0 blks
Holding blocks:           244 KB      1 blks
Free Small blocks:          0 KB
Free Ordinary blocks:      17 KB
Total in use:            3394 KB 99%
Total free:                17 KB 0%
2014/07/30 11:33:38| Starting Squid Cache version 2.6.STABLE21 for i686-redhat-linux-gnu...
2014/07/30 11:33:38| Process ID 4302
2014/07/30 11:33:38| With 1024 file descriptors available
2014/07/30 11:33:38| Using epoll for the IO loop
2014/07/30 11:33:38| DNS Socket created at 0.0.0.0, port 47682, FD 6
2014/07/30 11:33:38| Adding domain tagsa.co.za from /etc/resolv.conf
2014/07/30 11:33:38| Adding nameserver 10.0.0.1 from /etc/resolv.conf
2014/07/30 11:33:38| User-Agent logging is disabled.
2014/07/30 11:33:38| Referer logging is disabled.
2014/07/30 11:33:38| Unlinkd pipe opened on FD 11
2014/07/30 11:33:38| Swap maxSize 102400 + 8192 KB, estimated 0 objects
2014/07/30 11:33:38| Target number of buckets: 1843
2014/07/30 11:33:38| Using 8192 Store buckets
2014/07/30 11:33:38| Max Mem  size: 8192 KB
2014/07/30 11:33:38| Max Swap size: 102400 KB
2014/07/30 11:33:38| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2014/07/30 11:33:38| Rebuilding storage in /var/spool/squid (CLEAN)
2014/07/30 11:33:38| Using Least Load store dir selection
2014/07/30 11:33:38| Current Directory is /var/service/squid
2014/07/30 11:33:38| Loaded Icons.
2014/07/30 11:33:55| Accepting transparently proxied HTTP connections at 10.0.0.1, port 3128, FD 13.
2014/07/30 11:33:55| Accepting transparently proxied HTTP connections at 127.0.0.1, port 3128, FD 14.
2014/07/30 11:33:55| Accepting ICP messages at 10.0.0.1, port 3130, FD 15.
2014/07/30 11:33:55| Outgoing ICP messages on port 3130, FD 16.
2014/07/30 11:33:55| WCCP Disabled.
2014/07/30 11:33:55| Ready to serve requests.
2014/07/30 11:33:56| Store rebuilding is 83.3% complete
2014/07/30 11:33:57| Done reading /var/spool/squid swaplog (4917 entries)
2014/07/30 11:33:57| Finished rebuilding storage from disk.
2014/07/30 11:33:57|      4917 Entries scanned
2014/07/30 11:33:57|         0 Invalid entries.
2014/07/30 11:33:57|         0 With invalid flags.
2014/07/30 11:33:57|      4917 Objects loaded.
2014/07/30 11:33:57|         0 Objects expired.
2014/07/30 11:33:57|         0 Objects cancelled.
2014/07/30 11:33:57|         0 Duplicate URLs purged.
2014/07/30 11:33:57|         0 Swapfile clashes avoided.
2014/07/30 11:33:57|   Took 19.0 seconds ( 258.9 objects/sec).
2014/07/30 11:33:57| Beginning Validation Procedure
The issue arose after the system was shutdown without a proper halt due to power failures and our UPS failing.

From this i have tried to reset the process using a few threads i found online "http://superuser.com/questions/589438/squid3-proxy-cannot-write-log-file" but the system is still having problems (to the measure that we loose our Local network as well as any http access.

Has anyone ever experienced something similar or has any advise to restore squid?

Code: [Select]

[root@netkeeper squid]# ls -la
total 191944
drwxrwxr-x  2 squid squid     4096 Jul 31 13:59 .
drwxrwxrwx 53 root  root     20480 Jul 31 11:57 ..
-rwxrwxr-x  1 squid squid 24218466 Jul 31 14:24 access.log
-rwxrwxr-x  1 squid squid  7615853 Jul 27 04:02 access.log.1.gz
-rwxrwxr-x  1 squid squid  9180025 Jul 20 04:02 access.log.2.gz
-rwxrwxr-x  1 squid squid  8256428 Jul 13 04:02 access.log.3.gz
-rwxrwxr-x  1 squid squid  6477164 Jul  6 04:02 access.log.4.gz
-rwxrwxr-x  1 squid squid  5561742 Jun 29 04:02 access.log.5.gz
-rwxrwxr-x  1 squid squid  2824697 Jul 31 14:24 cache.log
-rwxrwxr-x  1 squid squid   122804 Jul 27 04:02 cache.log.1.gz
-rwxrwxr-x  1 squid squid    78990 Jul 20 04:02 cache.log.2.gz
-rwxrwxr-x  1 squid squid    64406 Jul 13 04:02 cache.log.3.gz
-rwxrwxr-x  1 squid squid    51292 Jul  6 04:02 cache.log.4.gz
-rwxrwxr-x  1 squid squid    40404 Jun 29 04:02 cache.log.5.gz
-rwxrwxr-x  1 squid squid        6 Jul 31 14:23 squid.pid
-rwxrwxr-x  1 squid squid 40925866 Jul 31 13:23 store.log
-rwxrwxr-x  1 squid squid 19276058 Jul 27 04:02 store.log.1.gz
-rwxrwxr-x  1 squid squid 22415101 Jul 20 04:02 store.log.2.gz
-rwxrwxr-x  1 squid squid 19589976 Jul 13 04:02 store.log.3.gz
-rwxrwxr-x  1 squid squid 15544897 Jul  6 04:02 store.log.4.gz
-rwxrwxr-x  1 squid squid 13966178 Jun 29 04:02 store.log.5.gz
[root@netkeeper squid]#


[Daniel B.]

You should check your disk isn't full, and run fsck to be sure your FS has no error

[derekb]

Thanks Daniel

Regarding the disk i can confirm its not full (1.3Tb available).

we will try a forced fsck and revert any results.

Is there any other way to maybe fully reconfigured (take to default).

Am i right in that squid is controlling that specific feature (HTTP Caching - Server-manager panel)?

Kind regards

[CharlieBrady]

Code: [Select]

FATAL: logfileWrite: /var/log/squid/access.log: (5) Input/output error


That almost certainly means that you have a disk I/O error. Search for 'sda' in /var/log/messages.

Do you have only a single disk? It is very highly recommended that you always use a two disk RAID1 pair.

I'm surprised that you say that you are also losing your local network. A hard disk error would not usually cause any disruption in ethernet networking.

[CharlieBrady]

Is there any other way to maybe fully reconfigured (take to default).

Fresh install and restore from backup. You would check your hardware before doing the fresh install.

Am i right in that squid is controlling that specific feature (HTTP Caching - Server-manager panel)?

squid should not be involved in accesses to the server-manager.

[derekb]

Hi CharlieBrady

thanks for the below.

i think a fresh install will do good but for the time being it would be good to know what is actually causing the problem.

From Server Panel -> At Proxy Settings -> HTTP proxy status -> Enabled/Disabled

Enabled = the http service on the server slows to a point were websites doesnt even want to open (extremely delay). Google searches do though work quite well. The local network (file server side) also slows down extremely. HTTPS works fine.
Disabled = the system return to normal. HTTP service is fine and the local network access speeds up to what is expected.

From the trouble shooting the only item we found that failed or pointed to a failure is squid and subsequently this thread.

By your last message i understand squid isnt really handling this.

Is there anything else i can trouble shoot to find the problem for future instances/reference. All the logs are clean except for this error.

Kind regards

[Jáder]

Maybe you have a slow disk problem.
Try this:

Code: [Select]

[root@leopardo ~]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   5556 MB in  2.00 seconds = 2780.14 MB/sec
 Timing buffered disk reads:  410 MB in  3.01 seconds = 136.14 MB/sec
[root@leopardo ~]#

This show my HDD (a 1TB disk on 3Gbps controller).
If you have small values on second number (Timing buffered disk reads), maybe your computer aren't using DMA to disk access so they will be REALLY SLOW.

Jáder


 

[derekb]

Hi Jader

Thank you for the advice.

i will try but the issue with the HTTP Proxy as explained is only something which has occurred recently.

the server itself have been operational now for +/- 5years. (gone through various hardware upgrades off coarse)

Kind regards

[janet]

derekb

i think a fresh install will do good but for the time being it would be good to know what is actually causing the problem.

You said
"The issue arose after the system was shutdown without a proper halt due to power failures and our UPS failing. "

Charlie has suggested you test ALL your hardware including drives, memory etc (there is test software on sme to do these), or use an external test UBCD etc, have you done that ?

Anything could have become corrupted during that uncontrolled power shutdown & it is probably very hard to find.

Perhaps also delete the squid cache, search forums on
delete squid cache

Unless you have done all things above then there is no use to keep asking why do I have this problem.

Finally you were advised to rebuild the server from known good backups, but ONLY after testing all your hardware.
That is expert advice from one of the worlds leading experts on sme server, so it is probably the best overall approach to resolve your issue.

So get busy !

[derekb]

Hi Janet

Sorry for the late reply.

We have done the hardware testing (even changing of the memory at well as network cards, etc.) but still didn't have improvement and/or results.

We have decided that we will be doing a fresh install on the latest version on new fresh hardware were after we will migrate the configuration.

Also, my question was purely related to and for future reference - something to test, check, etc. I did state that the only failure/fault we are seeing in the logs are squid and the only setting we changed to resolve the lag was the HTTP caching. In future i will refrain from asking question for future reference as i see a person is treated as a retard that doesn't show (or deemed by others of not showing) respect when asking for assistance. Looks like saying thank you is a insult in today's world.

Regards 

[janet]

derekb

No one can tell you exactly what is wrong with your server, only you, as you are the one sitting in front of it.
The onus is on yourself to troubleshoot etc. We can only suggest & advise.
It seemed to me that you kept asking what is wrong, without doing the work required & I just pointed that out to you.

I never referred to you in the way you suggest & never implied any of what you said, so I would ask you to withdraw those less than pleasant remarks, which are really quite uncalled for in a technical public forum like this.

In future i will refrain from asking question for future reference as i see a person is treated as a retard that doesn't show (or deemed by others of not showing) respect when asking for assistance. Looks like saying thank you is a insult in today's world.

You original question was:
"Has anyone ever experienced something similar or has any advise to restore squid?"

Numerous advices were given.

You seem to have a low opinion of yourself, so maybe you best work on improving that attitude.

[derekb]

Maybe you have a slow disk problem.
Try this:

Code: [Select]

[root@leopardo ~]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   5556 MB in  2.00 seconds = 2780.14 MB/sec
 Timing buffered disk reads:  410 MB in  3.01 seconds = 136.14 MB/sec
[root@leopardo ~]#

This show my HDD (a 1TB disk on 3Gbps controller).
If you have small values on second number (Timing buffered disk reads), maybe your computer aren't using DMA to disk access so they will be REALLY SLOW.

Jáder

Hi Jader

Thank you for the recomendation but im sure my HDD is fine:

Code: [Select]

[root@netkeeper ~]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   19080 MB in  2.00 seconds = 9549.91 MB/sec
 Timing buffered disk reads:  424 MB in  3.01 seconds = 141.06 MB/sec
[root@netkeeper ~]# 
As mentioned in post to Janet, we will be redoing the entire system from scratch.

Looks like no other option available.

Kind regards

[janet]

derekb

Looks like no other option available.

Maybe you missed:
Perhaps also delete the squid cache, search forums on
delete squid cache

[derekb]

Janet

I will follow Charlie's advice.

As per his post the squid does not have anything to do with the HTTP cashing.

Code: [Select]

Is there any other way to maybe fully reconfigured (take to default).

Fresh install and restore from backup. You would check your hardware before doing the fresh install.

Quote
Am i right in that squid is controlling that specific feature (HTTP Caching - Server-manager panel)?

squid should not be involved in accesses to the server-manager.
kind regards

[Daniel B.]

Nop, you misunderstood his statement. Squid is controlling HTTP caching, but doesn't interfer with server-manager's access

[janet]

derekb

If you took the time to search forums on
delete squid cache
you will find many examples of issues similar to yours.

I looked already, but it's up to you to follow up & implement.

You possibly have multipe issues to resolve.

While you are at it, make sure you replace the UPS battery & configure Nut to shutdown the server gracefully when the battery starts to get low, ie to avoid uncontrolled power loss hassles in the future.