Topic: Degraded array message

[BlueLake]

Hi

Rebooting after a software update I have an email -DegradedArray event on /dev/md1

checking the forums for similar and found several commands to locate the problem ie "cat /proc/mdstat"
which returns  md1: active raid1 sdb1[1] sda1[0]
                           104320 blocks [3/2] [UU_] ...(problem on the second partition of drive one ?)
                    md2: active raid1 sdb2[1] sda2[0]
                           7574528 blocks [2/2] [UU]

which from what I understand reading the same problem on the other posts is - the second partition on drive md1 has failed or failing or been kicked out of the array. So I need to issue the command...
mdadm --manage /dev/md1 --add/dev/sda to rebuild the array (I think)...
which returns - cannot open /dev/md1 device or resource busy.

instead of - mdadm: hot added /dev/md1

when I use...
mdadm --query --detail /dev/md1

  Raid devices 3 (? I only have two drives installed)
  total devices 2
  state: clean, degraded (which again I think is referring to the second partition!)
  number  major  minor  RaidDevices   state
      0         8       1           0              active sync
      1         8      17          1               active sync
      2         0        0          2               removed --- (only two drives installed)

It shows a drive has been removed, so I should be able to add it back to the array. Correct, or have I misunderstood that bit, and 
I don't understand why I have a - 0 / 1 / 2 - when there are only two drives installed...

and to rebuild the array use the command - mdadm -a /dev/md1 /dev/sda0 because its the second partition on the first drive (have I understood that bit right!!!)  or should I just replace the first drive completely ?

[CharlieBrady]

Hi So I need to issue the command...
mdadm --manage /dev/md1 --add/dev/sda to rebuild the array (I think)...

No, not correct. Unfortunately you tried it. Fortunately, it didn't work - likely did no harm.

It looks to me that your RAID arrays are working correctly, but the raid system is confused and thinks that there is a missing third device from your /dev/md1 array.

My advice is borrowed from THGTTG - "Don't panic". Don't do anything in a hurry. Wait until you better understand the problem. You should be able to purge that thrid phantom drive from the configuration and make it happy again.

[BlueLake]

Hi
Thanks for the reply - so you think (as I do) that somehow my server thinks there are three drives installed. This has only happened after the last software update by the way, yesterday (fresh install last week of v8). If I ignore it is there a chance the raid it will correct itself...?

[Daniel B.]

Purging this third "phantom" device from the RAID array is the easy part:

Code: [Select]

mdadm --grow /dev/md1 --raid-devices=2

Should do the trick.

Now, how this third member arrived here is another intersting question. I doubt a simple update could have done that. You should check /var/log/messages when this occurred (check for md, md1 degraded etc...)

[CharlieBrady]

That's an interesting use of the verb "grow", isn't it?

[CharlieBrady]

If I ignore it is there a chance the raid it will correct itself...?

Very unlikely.

[Daniel B.]

yep They choosed a poor word for this option, --grow is used to grow, shrink, or reshape in some way the array

[BlueLake]

Hi
I did a reboot and then checked the messages and found this...md: sdb1 has different UUID to sdb2

so i then googled the web and found this...

http://srvnepal.wordpress.com/tag/md-sda1-has-different-uuid-to-sdb2/

which to my limited knowledge sounds about right - but then a little knowledge is a very dangerous thing - especially where my new server is concerned. From what I can see in my original post is that part of drive 1 (md1) has been removed from the array. So I don't need to do that bit. But it wont allow the next part which is adding it back to the array.

So how big a problem do I have with "md: sdb1 has different UUID to sdb2"

[Daniel B.]

No, sdb1 and sdb2 having different UUID is normal. sdb1 and sda1 are assembled as md1 and sdb2 and sda2 assembled as md2. The only problem is that somehow md1 has been "grown" to have 3 devices, with one missing. None of your RAID are really degraded, as you can see, md1 has two devices (sda1 and sdb1), so to fix it, you just have to tell your system that md1 should only have two devices, with:

mdadm --grow /dev/md1 --raid-devices=2

I'd be interesting to find out what (or who) changed md1 conf to have 3 devices instead of 2. /var/log/messages should contains some info (at least when did this occured)

[BlueLake]

Well Daniel.b that worked 
all I can say is This was a fresh install from a mirror download onto a dvd about two weeks ago...since then I have added some postgresql lib files and expanded to php.ini and httpd.conf to include .htm .html files and rebooted, so far no problems. There was a software update on Sunday morning which gave the first email warning and another this Morning (Tuesday). Both drives are 250gb. no hardware raid installed.

very great-full for you help, if you need me to look at the messages you will need to tell me what to look for - I have just looked through the messages again and nothing leaps out except for a load of MySql errors !!!!

Thanks again 

[idp_qbn]

Just out of interest, about the third drive - is it possible you had a USB drive attached at initial boot time. which got added to the RAID array but which you since removed?
I had a problem (years ago now and I forget the complete details) when I left a USB stick attached when setting up my SME box and it included the USB in the RAID somehow.

Anyway, good to hear the problem was solved.
Cheers
Ian