Topic: Raid rebuild every sundag at 4:22 - WHY?

[beast]

Hi All

Have a problem that I do not understand.

I have 2 servers running. One is just an affa backup copy, but both have exactly the same hardware.

The main server starts a raid rebuild event every Sunday at 4:22. Nothing happens on the backup server. In my world this is an indication of the fact that - my problem is not hardware related.

I can not figure out why this rebuild is initiated. I have added some logs and cronjobs running.

Please give me a hint about what may be the problem

BTW: Is there a command to show a list of installed contribs?

Messages:
Jun 23 04:02:04 beastserver squid[3019]: storeDirWriteCleanLogs: Starting...
Jun 23 04:02:05 beastserver squid[3019]:   Finished.  Wrote 6255 entries.
Jun 23 04:02:05 beastserver squid[3019]:   Took 0.0 seconds (213037.7 entries/sec).
Jun 23 04:02:05 beastserver squid[3019]: logfileRotate: /var/log/squid/store.log
Jun 23 04:02:05 beastserver squid[3019]: logfileRotate: /var/log/squid/access.log
Jun 23 04:22:01 beastserver kernel: md: syncing RAID array md1
Jun 23 04:22:01 beastserver kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Jun 23 04:22:01 beastserver kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
Jun 23 04:22:01 beastserver kernel: md: using 128k window, over a total of 104320 blocks.
Jun 23 04:22:01 beastserver kernel: klogd 1.4.1, ---------- state change ----------
Jun 23 04:22:01 beastserver kernel: Inspecting /boot/System.map-2.6.18-348.6.1.el5PAE
Jun 23 04:22:01 beastserver kernel: Loaded 30783 symbols from /boot/System.map-2.6.18-348.6.1.el5PAE.
Jun 23 04:22:01 beastserver kernel: Symbols match kernel version 2.6.18.
Jun 23 04:22:01 beastserver kernel: No module symbols loaded - kernel modules not enabled.
Jun 23 04:22:01 beastserver kernel: md: delaying resync of md2 until md1 has finished resync (they share one or more physical units)
Jun 23 04:22:03 beastserver kernel: md: md1: sync done.
Jun 23 04:22:03 beastserver kernel: md: syncing RAID array md2
Jun 23 04:22:03 beastserver kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Jun 23 04:22:03 beastserver kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reconstruction.
Jun 23 04:22:03 beastserver kernel: md: using 128k window, over a total of 1953407488 blocks.
Jun 23 04:22:03 beastserver kernel: RAID1 conf printout:
Jun 23 04:22:03 beastserver kernel:  --- wd:2 rd:2
Jun 23 04:22:03 beastserver kernel:  disk 0, wo:0, o:1, dev:sda1
Jun 23 04:22:03 beastserver kernel:  disk 1, wo:0, o:1, dev:sdb1


Crontab:
45 1 * * * php -f /home/e-smith/files/ibays/pass_ibay/html/cron.php
15 2 * * * mysqlhotcopy --allowold passwordcrypt /home/e-smith/files/ibays/pass_ibay/html/database/
15 3 1 * * mysqldump passwordcrypt > /home/e-smith/files/ibays/pass_ibay/html/database/passwordcrypt.sql
15 3 10 * * mysqldump phpBB > /home/e-smith/files/ibays/pass_ibay/html/database/phpBB.sql
15 3 20 * * wget -e robots=off -m -k -K -E -rH -P /home/e-smith/files/ibays/pass_ibay/html/help/ -Dsites.google.com http://sites.google.com/site/pcrypthelp/
45 4 * * * rsync -az -e ssh /home/e-smith/files/ibays/pass_ibay/html/* xxxx@xxx.atki.dk:/home/e-smith/files/users/benny/passwordcrypt
30 * * * * php -f /home/e-smith/files/ibays/Primary/html/solarlog/solarlog.php


/var/log/raidmonitor/current:
2013-05-26 04:22:02.291895500 Event: RebuildStarted, Device: /dev/md1, Member:
2013-05-26 04:22:02.924928500 Event: RebuildStarted, Device: /dev/md2, Member:
2013-05-26 04:22:03.017135500 Event: RebuildFinished, Device: /dev/md1, Member:
2013-05-26 11:36:06.614492500 Event: Rebuild20, Device: /dev/md2, Member:
2013-05-26 18:59:09.206838500 Event: Rebuild40, Device: /dev/md2, Member:
2013-05-27 02:35:11.323723500 Event: Rebuild60, Device: /dev/md2, Member:
2013-05-27 05:40:11.962613500 Event: Rebuild80, Device: /dev/md2, Member:
2013-05-27 06:48:30.596167500 Event: RebuildFinished, Device: /dev/md2, Member:
2013-06-02 04:22:02.064475500 Event: RebuildStarted, Device: /dev/md1, Member:
2013-06-02 04:22:04.870988500 Event: RebuildStarted, Device: /dev/md2, Member:
2013-06-02 04:22:04.984008500 Event: RebuildFinished, Device: /dev/md1, Member:
2013-06-02 11:55:07.790171500 Event: Rebuild20, Device: /dev/md2, Member:
2013-06-02 19:51:10.853815500 Event: Rebuild40, Device: /dev/md2, Member:
2013-06-03 03:45:13.296128500 Event: Rebuild60, Device: /dev/md2, Member:
2013-06-03 06:02:14.203354500 Event: Rebuild80, Device: /dev/md2, Member:
2013-06-03 07:10:03.375897500 Event: RebuildFinished, Device: /dev/md2, Member:
2013-06-09 04:22:01.774438500 Event: RebuildStarted, Device: /dev/md1, Member:
2013-06-09 04:22:04.677457500 Event: RebuildStarted, Device: /dev/md2, Member:
2013-06-09 04:22:04.750506500 Event: RebuildFinished, Device: /dev/md1, Member:
2013-06-09 11:45:07.450183500 Event: Rebuild20, Device: /dev/md2, Member:
2013-06-09 19:10:09.098428500 Event: Rebuild40, Device: /dev/md2, Member:
2013-06-10 02:41:11.905893500 Event: Rebuild60, Device: /dev/md2, Member:
2013-06-10 06:03:13.886616500 Event: Rebuild80, Device: /dev/md2, Member:
2013-06-10 07:10:42.661046500 Event: RebuildFinished, Device: /dev/md2, Member:
2013-06-16 04:22:02.175186500 Event: RebuildStarted, Device: /dev/md1, Member:
2013-06-16 04:22:04.581198500 Event: RebuildStarted, Device: /dev/md2, Member:
2013-06-16 04:22:04.818642500 Event: RebuildFinished, Device: /dev/md1, Member:
2013-06-16 11:58:07.050967500 Event: Rebuild20, Device: /dev/md2, Member:
2013-06-16 20:05:11.246946500 Event: Rebuild40, Device: /dev/md2, Member:
2013-06-17 03:55:15.024897500 Event: Rebuild60, Device: /dev/md2, Member:
2013-06-17 06:31:16.258397500 Event: Rebuild80, Device: /dev/md2, Member:
2013-06-17 07:41:15.175167500 Event: RebuildFinished, Device: /dev/md2, Member:
2013-06-23 04:22:01.320781500 Event: RebuildStarted, Device: /dev/md1, Member:
2013-06-23 04:22:03.554614500 Event: RebuildStarted, Device: /dev/md2, Member:
2013-06-23 04:22:03.636060500 Event: RebuildFinished, Device: /dev/md1, Member:

[chris burnat]

Hi All

Have a problem that I do not understand.

I have 2 servers running. One is just an affa backup copy, but both have exactly the same hardware.

The main server starts a raid rebuild event every Sunday at 4:22. Nothing happens on the backup server. In my world this is an indication of the fact that - my problem is not hardware related.

I can not figure out why this rebuild is initiated. I have added some logs and cronjobs running.

Please give me a hint about what may be the problem

Please check : http://bugs.contribs.org/show_bug.cgi?id=6160
No panic, it is not harmful, just a nuisance.

BTW: Is there a command to show a list of installed contribs?

# Test what is installed:
 /sbin/e-smith/audittools/newrpms [will shod you rms]
 /sbin/e-smith/audittools/templates [will show you templates]

best

[beast]

Please check : http://bugs.contribs.org/show_bug.cgi?id=6160
No panic, it is not harmful, just a nuisance.

Thank you Chris for the info - but the rebuild do seam to make a real rebuild (disk activity and slower server) so it will maybe wear down the drives in the long run.

/b

[chris burnat]

Thank you Chris for the info - but the rebuild do seam to make a real rebuild (disk activity and slower server) so it will maybe wear down the drives in the long run.

/b

Yes, all of this is clearly documented in the bug... 
We are doing our best to resolve this issue and hopefully will come up with a fix via updates.
/best.

[elmarconi]

Thank you Chris for the info - but the rebuild do seam to make a real rebuild (disk activity and slower server) so it will maybe wear down the drives in the long run.

I am not worried about additional wear.
If the only disk remaining in a RAID-1 fails during rebuild then you are not happy...

[janet]

elmarconi

Until the real issue is resolved, just follow the workaround in Comment 10
http://bugs.contribs.org/show_bug.cgi?id=6160#c10
Restart crond afterwards
/etc/init.d/crond restart

[CharlieBrady]

Restart crond afterwards
/etc/init.d/crond restart

Not necessary to do that. It's also unwise to disable a useful hardware self-test.

[chris burnat]

Not necessary to do that. It's also unwise to disable a useful hardware self-test.

Charlie's advice is good advice. It is explained in Bug #6160, refer comment #106.

Code: [Select]

Weekly array checks are vital. :-)
See bug 3535 for some background.
Recovery is painful.
We have survived until now, and fixed should soon be released, best is to wait for next release of packages. My personal feeling is that the real issue slowing down system is the weekly antivirus scan which starts around 01:00 local time, and can take hours, overlapping with the disk self-test.