Topic: Raid5 (4 disks) problem

[engdev]

Hi,
I recently installed 7.5.1 on a new server (HP ML 110 G6) with 4 identical HDD and followed the 'Moving SME to new hardware' howto to transfer everything from our existing server (also with 4 HDD but started out with 7.3 ? and upgraded to 7.5.1).

Last week I received

A DegradedArray event has been detected on md device /dev/md1

so I removed & re-added the sda1 to the array and then received

A SpareActive event has been detected on md device /dev/md1.
Device /dev/sda1 is now an active member of md device /dev/md1

This is the output of mdadm.conf and mdstat:

Code: [Select]

DEVICE partitions
ARRAY /dev/md2 level=raid5 num-devices=3 spares=2 UUID=85f95d35:6ee44c78:8bb85570:19e5f164
ARRAY /dev/md1 level=raid1 num-devices=4 UUID=58368727:fab34221:15b334b3:7232af96

[root@smeserver ~]# cat /proc/mdstat
Personalities : [raid1] [raid5]
md2 : active raid5 sdc2[0] sdd2[2] sdb2[1]
      976558592 blocks level 5, 256k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      104320 blocks [4/4] [UUUU]

unused devices: <none>
However, the output of mdadm.conf and mdstat on the original server is as follows:

Code: [Select]

DEVICE partitions
ARRAY /dev/md2
ARRAY /dev/md1

[root@smeserver3 ~]# cat /proc/mdstat
Personalities : [raid1] [raid5]
md2 : active raid5 sda2[0] sdc2[3] sdd2[2] sdb2[1]
      490014208 blocks level 5, 256k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid1 sda1[0] sdd1[3] sdc1[2] sdb1[1]
      104320 blocks [3/3] [UUU]

unused devices: <none>
So on the old server, sda2 was part of the Raid 5 array but on the new server it is missing. Is this due to sda1 being kicked out of the array on the new server when the array degraded? Do I need to add sda2 back into the md2 array?

Is the Raid 5 array on the new server configured correctly or do I need to take any other action?

How can you tell from the mdstat if the Raid 5 array & spare is functioning properly?

Thanks for any advice (& thanks for the great software)
(Apologies if I am repeating a previous question but I had a search here and on Google and couldn't find anything that matched my installation with 4 disks.)

[idp_qbn]

The RAID 5 is "working properly" but as a 3 disk RAID array. The 4th disk (sda2) is missing.
A quick check is to use the df command (df -h) to show the disk space available.

For RAID5, the space available (in md2) should be approximately 3 x (single disk size) less 10% or so.
So a 4 disk array, using 500 Gb disks should give you  about 1400 Gb

I think when you try it, you will get a value nearer to 2 x (single disk size)

The RAID 5 rule of thumb is : disk space = (Number of disks -1) x (single disk size)

Sorry, but I can't help you with getting the 4th disk into the array since I am not sure how to do that on a "live"system.

Cheers
Ian

[engdev]

Thanks very much for the df -h tip Ian, it's very useful information.

I agree with your calc for RAID 5 nospare but 'I think' the default SME RAID5 (as I have) is to have (n-1) x capacity plus a spare so in my case with 4 x 500 Gb the total is approx 1Tb, which is what is shown by df -h.

I found another mdadm command:

Code: [Select]

mdadm --examine /dev/md2 /dev/sda2which returns these results:

Code: [Select]

mdadm: No super block found on /dev/md2 (Expected magic a92b4efc, got 00000000)
/dev/sda2:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 85f95d35:6ee44c78:8bb85570:19e5f164
  Creation Time : Fri May  6 13:14:17 2011
     Raid Level : raid5
   Raid Devices : 3
  Total Devices : 4
Preferred Minor : 2

    Update Time : Mon Jun  6 21:12:52 2011
          State : clean
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1
       Checksum : fc906df3 - correct
         Events : 0.836279

         Layout : left-symmetric
     Chunk Size : 256K

      Number   Major   Minor   RaidDevice State
this     0       8        2        0      active sync   /dev/sda2

   0     0       8        2        0      active sync   /dev/sda2
   1     1       8       18        1      active sync   /dev/sdb2
   2     2       8       50        2      active sync   /dev/sdd2
   3     3       8       34        3      spare   /dev/sdc2
Which indicates that the array is correct but it contradicts the results from cat /proc/mdstat which still shows sda2 missing from the array. So I am still confused, if you or anyone else can educate me as to why there's a difference between mdstat and the mdadm --examine, I would be very grateful.
regards
John

[idp_qbn]

Yes, the output clearly says it is RAID 5 with 3 devices....with a "hot spare". This means if a disk fails, the "hot spare" can be swapped in to take its place with minimum downtime.

If you want all 4 disks to be used to give extra storage (ie RAID 5 with 4-disk array, no spare) you install using the "sme nospare" option at the beginning of the installation - the very first thing you do after the installation CD had loaded the installer and presented you with an SME installation screen.

Can you re-install? If so, use "sme nospare" at the beginning of the installation. That is the command to make SME use all the available disks in the RAID 5 array.

Search the Wiki and the forums for advice on this.

Cheers
Ian