Topic: Server Locked up - AGAIN????

[Olsen]

I am not sure why my sme server has started locking up.  This is the second time this week that our server has locked up and I cannot even putty to it, and when at the physical server, cannot even cursor up or down....COMPLETELY LOCKED.  I had to hard boot...not good

I am running sme 6.0.1

Now I am getting admin notices that:
Could not open /var/log/httpd/awstats/

Does anyone know why this is happening?
How do I know what is causing these hang ups?  How can I diagnose my problem?

[NickR]

Always suspect hardware first.  IME, 9 times out of 10 it's memory, so swap that out first.

[Olsen]

It does appear to be a memory issue that is occurring since upgrading to the new spam assassin and installing some other security features.

Do you know how I can get the error message from being sent to me

Could not open /var/log/httpd/awstats/
.749

Awstats has a cron job running ever 5 minutes and I keep getting admin emails saying that this file could not be opened.  Do I need to chown it?

[smeghead]

Hmm, the only awstats files I have are as follows:

[root@kryten httpd]# ls -la aw*
-rw-rw-rw-    1 root     admin          63 Jan  8 12:00 awstats_lr
-rw-rw-rw-    1 root     admin          71 Jan  8 03:30 awstats_pos

so check if your awstats files are set to the same ownership/perms.  If not make a note of what they are first and then run:

chown root:admin awstats.*
chmod 666 awstats.*

[raem]

Spamassassin uses a lot of processor power and a lot of memory. You should have at least 512 Mb of RAM, and more if your server is busy with email and other stuff.
Don't just change the RAM, but add more.

[m]

I am not sure why my sme server has started locking up.  This is the second time this week that our server has locked up and I cannot even putty to it, and when at the physical server, cannot even cursor up or down....COMPLETELY LOCKED.  I had to hard boot...not good

Hi,

we had the same on several Dell 1600SC machines under heavy IO load (rsync). The servers have frozen and only responded to ping.

The kernel boot sequence logs a hardware bug concernig APIC timers in the message log. (cannot remember the exact message, just search for apic)
After starting the kernel with the "noapic" option appended, the servers run reliable.

Michael

[Olsen]

Ray - The memory was definitly a main contributing cause to the most recent server lock up.  The strange thing is I saw the swap usage jump above the 256mb (which is all the ram we currently have), definintly odd.  Have you seen a server spike the swap usage like this?  The only reason it should do that is if ALL of the physical memory is being used.  We are going to be adding 1Gb or ram on Monday when we get back in the office.  I am really hoping that this will resolve that issue.  Thanks for your help!

smeghead - Thanks for the help, I will do that first thing on Monday.  Those were the only two files I had in my httpd directory, so I dont know why all of a sudden awstats was looking for an awstats directory within the httpd directory.  I will post the command the cron job was trying to execute when I get back on Monday.  Thanks for your help!

mweinber - Yes, our server is under some pretty heavy I/O requests since we have moved some heavily accessed files onto this server.  I believe we may be needing to do a serious server hardware upgrade.  Currenly we are just running a 2.4 P4 with a 80 gig HDD (IDE)....  We should definitely upgrade to a dual xeon and scsi environment soon.  Thanks for your help!

[smeghead]

I have had to add a second SME server to one site where the mail load was so severe (due to occasional virus/spam floods) that the server was brought to its knees due to lack of RAM.

I upped the RAM to 768MB & added Gordons mailfront contrib to the mailserver and it now only slows for a coupla minutes here and there and memory usage is now significantly under the max.

The dual Xeon system is nice but unless your doin some heavy client/server processing stuff its gonna be a waste (would run folding@home well tho ).  Spend the money on a nice caching RAID setup with 4 or 5 disks & use Gigabit networking to up the disk serving performance.  Perhaps consider a dual server setup!