Topic: 99% processor usage! Help

[SSBN]

99% processor usage! Help

I have a SME 5.6 server running VoIP software. I can’t upgrade to 6.0 as the VoIP software won’t support it yet.

I am experiencing a problem that has me stumped. My processor is at 99% usage. When I do a top raid1d and raid1syncd are using my entire possessor up. Java is also running about 15 times. The hard drives are also going non stop. They are an ide software raid.
I have tried a killall java to get rid of them but the other two processes can’t be stopped as far as I can tell because they are run by the kernel. I have tried a reboot as well. Here is my top info. Do any of you have an idea what might be causing this.



3:29pm  up  1:24,  1 user,  load average: 10.58, 23.89, 27.59
83 processes: 77 sleeping, 6 running, 0 zombie, 0 stopped
CPU states:  0.1% user, 97.4% system,  0.0% nice,  2.3% idle
Mem:   255768K av,  100180K used,  155588K free,       0K shrd,    4180K buff
Swap:  264952K av,       0K used,  264952K free                   60844K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
   17 root      19   0     0    0     0 RW   41.7  0.0  24:38 raid1d
   18 root      17   0     0    0     0 RW   23.1  0.0   7:28 raid1syncd
 2330 squid     16   0  4500 4500  1200 R    11.4  1.7   2:34 squid
   20 root      16   0     0    0     0 SW    4.7  0.0   2:04 kjournald
  474 root      16   0   308  308   252 S     2.9  0.1   0:17 svscan
    3 root      15   0     0    0     0 SW    0.9  0.0   0:39 kapmd
 1627 ntp       15   0  1876 1876  1680 S     0.9  0.7   0:41 ntpd
 2554 root      15   0  5136 5136  5028 S     0.9  2.0   0:15 httpd
 1630 ntp       15   0  1880 1880  1680 S     0.7  0.7   0:22 ntpd
 1784 ldap      15   0  1932 1928  1512 S     0.3  0.7   0:07 slapd
 6288 root      15   0  1872 1872  1584 R     0.3  0.7   0:06 sshd
 6742 root      15   0  1036 1036   812 R     0.3  0.4   0:02 top
    4 root      34  19     0    0     0 RWN   0.1  0.0   0:00 ksoftirqd_CPU0
    5 root      15   0     0    0     0 SW    0.1  0.0   0:16 kswapd
    1 root      15   0   480  480   420 S     0.0  0.1   0:15 init
    2 root      15   0     0    0     0 SW    0.0  0.0   0:00 keventd
    6 root      25   0     0    0     0 SW    0.0  0.0   0:00 bdflush
    7 root      15   0     0    0     0 SW    0.0  0.0   0:08 kupdated
    8 root      25   0     0    0     0 SW    0.0  0.0   0:00 mdrecoveryd
   16 root      25   0     0    0     0 SW    0.0  0.0   0:00 raid1d
   19 root      25   0     0    0     0 SW    0.0  0.0   0:00 raid1d
  161 root      16   0     0    0     0 SW    0.0  0.0   0:00 khubd
  217 root      15   0     0    0     0 SW    0.0  0.0   0:00 kjournald
  505 root      15   0   296  296   252 S     0.0  0.1   0:00 supervise
  506 root      16   0   296  296   252 S     0.0  0.1   0:00 supervise
  507 root      16   0   296  296   252 S     0.0  0.1   0:00 supervise
  508 root      15   0   296  296   252 S     0.0  0.1   0:00 supervise
  509 root      16   0   296  296   252 S     0.0  0.1   0:00 super

[NickR]

What does running this

# cat /proc/mdstat

a couple of times 5 minutes apart give you?

[cc_skavenger]

use this command:

top

look for the process that is using the most resources.  Use q to quit top and then use this command:

kill {process ID number}

if the process won't die, then use this command:

kill -s 9 {process ID number}

HTH

Marco

[patrick]

NickH has it. It looks like your raid array is rebuilding.

patrick.

[SSBN]

The problem seemed to correct itself. By the time the morning rolled around it was working fine again. I wonder what would have made the drives go out of sink. A bigger question is I didn’t think they would auto rebuild. Last time a hard drive failed I had to initiate it by command line.  But then this time the drive didn't fail.

[NickR]

You'd expect the array to automatically resync after an improper shutdown or some sort of hardware problem with one of the drives.  FWIW, I've seen arrays randomly break when the disks are on the primary & secondary controller and under heavy writing load.  These days, I always make the mirrors on the same (primary) controller and that problem seems to have gone away.

It's well worth installing Darrell May's raidmonitor contrib, which emails you if the array fails.

http://mirror.contribs.org/smeserver/contribs/dmay/mitel/contrib/raidmonitor/