Topic: High load averages

[DanB35]

I'm running SME 6.0.1 with some updates, and periodically I'm getting very slow response from the server.  Most frequently, it's when I'm trying to access the IMAP server in some way, and it will time out, but it happens at other times as well.  Checking the load averages, they appear very high--I've seen the average go over 10.  However, I can't see any reason for that to be the case--both CPUs are 98+% idle, there's 200+MB of RAM free, hardly any swap used, and no abnormal disk I/O going on from what I can see.

I'd like to try to figure out what's driving up the load, and top, the best thing I know of, isn't helping much.  Any other ideas of tools and/or methods to help me track this down would be appreciated.  Thanks!

[paulfung]

Try to use ther "ps -e" command to tell how many processes are running, see if there is any unneeded one.

Also, use "hdparm /dev/hda" to see if the DMA morde of your HD is set to on or not.....

tell us the result and see what the problem is.....

As for now, there is not enouth info to lets us know what happened.

[DanB35]

ps -e returns 155 processes, none of which look (on initial inspection) to be entirely superfluous.  I can post the list if you like, but I'm not sure it would be useful.  I don't have /dev/hda on my system, as I'm using a hardware SCSI RAID controller.  Here's what I get:

[root@e-smith home]# hdparm /dev/sda

/dev/sda:
 readonly     =  0 (off)
 geometry     = 17683/255/63, sectors = 284082176, start = 0

I'm not even so much asking "what's causing the high load average" as "what tools can I use to find out".  Thanks for the suggestions; are there any other ideas?

[hanscees]

ps -e returns 155 processes, none of which look (on initial inspection) to be entirely superfluous.  I can post the list if you like, but I'm not sure it would be useful.  I don't have /dev/hda on my system, as I'm using a hardware SCSI RAID controller.  Here's what I get:

[root@e-smith home]# hdparm /dev/sda

/dev/sda:
 readonly     =  0 (off)
 geometry     = 17683/255/63, sectors = 284082176, start = 0

I'm not even so much asking "what's causing the high load average" as "what tools can I use to find out".  Thanks for the suggestions; are there any other ideas?

You are not even asking what is causing the high load but what is causing my problems that could be caused by load.

"top" is your friend. uptime is your friend. iptraf is cool.

To see trends you could install collectd.
http://www.hanscees.com/sme7/collectd4sme7howto.html
But that is for sme7. So find some contrib that does rrd-tool.

That might give you a clue what is peaking, memory, or cpu or io or traffic.

also vmstat and iostat can tell you stuff.

Hans-Cees

[raem]

DanB35

>...what tools can I use to find out.

Install the htop rpm from Charlie Brady
ftp://ftp.ibiblio.org/pub/linux/distributions/e-smith/contrib/CharlieBrady/RPMS/i386/

As he said "it's like top on steroids".

Are you running the spamassassin & clam contribs, they can use up resources & clog up the system if there are lot's of incoming /outgoing messages being scanned.
Do you have RBL lists enabled, it will cut down the amount of spam being processed.

These tips may be of use
http://mirror.contribs.org/smeserver/contribs//rmitchell/smeserver/howto/Mail%20system%20tweaks%20HOWTO%20for%20sme%20server.htm

[DanB35]

Are you running the spamassassin & clam contribs, they can use up resources & clog up the system if there are lot's of incoming /outgoing messages being scanned.

I am using those contribs--interestingly, I never had this sort of trouble when I was using RAV.  I'll check out some of those suggestions.  Thanks!

[DanB35]

That might give you a clue what is peaking, memory, or cpu or io or traffic.

I do have the system monitor contrib installed, that that's what has me baffled--CPU, memory, and net traffic are all low.  Looking at top when this is happening can show both CPUs 98% idle, 200+MB RAM free, with a load average of 8 or more.  Watching the disk activity LEDs doesn't reveal anything unusual.

I wasn't familiar with iptraf, vmstat, or iostat; I'll check them out.  Thanks!

[hanscees]

I also know that a time ago there was something about too many imap users. There was some kind of user-limit you could avoid. Do you have many imap users by the way?

Hans-Cees

[DanB35]

Only one IMAP user, which is me, 99% of the time.

[raem]

DanB35

As asked previously, do you have RBL lists enabled, it will cut down the amount of spam being processed ?

[paulfung]

I have more that 10 servers on the data center, 5.5, 5.6 6.01, 6.5.....

all of them are just having 100 (+-20 process)

I guess the extra 50 process may be the problem.

Have you change any setting in MySQL or Apache ?

P.S. : try this one and you can know more on what is happening inside your server. (Watching the HD light will not give you any info you needed.)

http://www.wellsi.com/sme/sysmon/sysmon.html

[DanB35]

As asked previously, do you have RBL lists enabled, it will cut down the amount of spam being processed ?

I didn't have them enabled.  I enabled the conservative lists per the link you gave, and it may have helped a bit, but the load was still pretty high.

[raem]

DanB35

> I didn't have them (RBL's) enabled.

That means your system may be very busy scanning and processing spam messages, and therefore you may also have a lot of mail in the queue.

>....it may have helped a bit, but the load was still pretty high.

Have you checked your mail queue, install the qmHandle contrib. You may have a lot of mail waiting to be delivered and the system is busy retrying.

Also download and install htop, it shows you processes by CPU % and by memory usage, press F6 to reorder the view.
It should tell you what's going on.

[hanscees]

just a plain good old tcpdump running for an hour or so will tell you what is going on too.

outside interface:
tcpdump -nlpi eth1 -s 0 -w /root/outsidedump

inside interface same but then eth0 and insidedump.

after a while do cntr-c. Get the file outsidedump to your workstation and look at it with ethereal.

From the tcpdump on the inside you can see if the imap server does not anwser or if it does not get any reply. Perhaps you have interface/switch problems.

Hans-Cees