Topic: RAID 1 Hot spare indicator?

[smiit]

Preparing to migrate from SME 8 to 9 and uncertain about RAID 1 status in the console message.

Default install from DVD to a Dell PowerEdge T310 with three 2TB SATA drives in front hot-swap bay.

After install, updates and RAID resync I see:

Code: [Select]

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[2] sdb1[1] sda1[0]
      255936 blocks super 1.0 [3/3] [UUU]

md1 : active raid1 sdc2[2] sda2[0] sdb2[1]
      1953126208 blocks super 1.1 [3/3] [UUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

And:

Code: [Select]

# mdadm --detail /dev/md1
/dev/md1:
        Version : 1.1
  Creation Time : Thu Dec 18 06:15:49 2014
     Raid Level : raid1
     Array Size : 1953126208 (1862.65 GiB 2000.00 GB)
  Used Dev Size : 1953126208 (1862.65 GiB 2000.00 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue Dec 23 15:29:22 2014
          State : active
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

           Name : localhost.localdomain:1
           UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:xxxxxxxx
         Events : 4689

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2

I assume it loaded as a RAID 1 with a hot spare since the total drive space is 2TB (less overhead) but I would expect this:

Code: [Select]

Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

to appear as:

Code: [Select]

Active Devices : 3
Working Devices : 2
Failed Devices : 0
Spare Devices : 1

Is this the standard output for a clean 3 drive RAID1 setup in the SME 9 environment?

Or should I reinstall with different settings to have the desired two-drive RAID 1 with an available hot-spare?

Thanks

[nicolatiana]

The spare handling for RAID arrays is not implemented.

(From release notes of SME9)
I suppose is not supported in install, but performing install with two drives and then adding the third, with mdadm I suppose you can add it as spare to the array.
Can anyone confirm this ?

Nicola

[stephdl]

for adding a spare drive you can follow this howto http://wiki.contribs.org/Raid#Adding_another_Hard_Drive_Later_.28Raid1_array_only.29

the easy way is to reinstall, but i'm not sure that you need it, probably you can use mdadm to --remove and --set-faulty the drive you want of raid partitions (/dev/md0 & /dev/md1) and after add the correct number of disk in /etc/mdadm.conf

Code: [Select]

mdadm --set-faulty /dev/md0 /dev/sdc1
mdadm --set-faulty /dev/md1 /dev/sdc2
mdadm --remove /dev/md0 /dev/sdc1
mdadm --remove /dev/md1 /dev/sdc2
and

Code: [Select]

# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=5ee074e0:f3fa7c60:27383b37:0663f6f2
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=207ed099:c5abe949:0dc10494:758f4087

then follow up the pointer I gave, of course I assumed that all backups of your data are done.

[smiit]

for adding a spare drive you can follow this howto http://wiki.contribs.org/Raid#Adding_another_Hard_Drive_Later_.28Raid1_array_only.29

the easy way is to reinstall, but i'm not sure that you need it, probably you can use mdadm to --remove and --set-faulty the drive you want of raid partitions (/dev/md0 & /dev/md1) and after add the correct number of disk in /etc/mdadm.conf

then follow up the pointer I gave, of course I assumed that all backups of your data are done.

Thank you, stephdl, for the info.

I followed your suggestions and failed/removed /dev/sdc from the array and added it back in but it still appears exactly the same as an active/working device, not a spare (which is what I keep expecting/desire) when I run the mdadm --detail command.

Code: [Select]

Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0

If I re-install is there a specific RAID installation option I need to specify that I missed before?

Also, I did see Nicola's comment about spare handling not implemented in the release notes .. is there any 'best practices' advice for what to do currently in a production environment?

Should I install 2 drives as a standard RAID1 mirror and just have the spare ready to drop in if necessary?  Or will that 3rd drive effectively act as a hot spare if one fails?

[nicolatiana]

Did you wipe-out with dd the disk before putting in back ?

Nicola

[smiit]

Did you wipe-out with dd the disk before putting in back ?

Nicola

I did not.

Tried the same sequence again but wiped /dev/sdc with the following dd command:

Code: [Select]

dd if=/dev/zero of=/dev/sdc bs=512 count=1

Rebooted and copied partition table:

Code: [Select]

sfdisk -d /dev/sda > sfdisk_sda.output
sfdisk /dev/sdc < sfdisk_sda.output

Rebooted again but now I get this error:

Code: [Select]

mdadm --add /dev/md0 /dev/sdc1
mdadm: Cannot find /dev/sdc1: No such file or directory


[stephdl]

Code: [Select]

# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=5ee074e0:f3fa7c60:27383b37:0663f6f2
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=207ed099:c5abe949:0dc10494:758f4087


do you have modified the /etc/mdadm.conf, ok this was before you wiped the drive

[nicolatiana]

Output of fdsik -l ?

Nicola

[smiit]

Output of fdsik -l ?

Nicola

Code: [Select]

# fdisk -l

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00096293

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          32      256000   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2              32      243202  1953257472   fd  Linux raid autodetect

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006f267

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          32      256000   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2              32      243202  1953257472   fd  Linux raid autodetect

Disk /dev/md1: 2000.0 GB, 2000001236992 bytes
2 heads, 4 sectors/track, 488281552 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/main-root: 1991.6 GB, 1991568785408 bytes
255 heads, 63 sectors/track, 242127 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/main-swap: 8430 MB, 8430551040 bytes
255 heads, 63 sectors/track, 1024 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/md0: 262 MB, 262078464 bytes
2 heads, 4 sectors/track, 63984 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

And

Code: [Select]

cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 level=raid1 num-devices=2 UUID= xxx
ARRAY /dev/md1 level=raid1 num-devices=2 UUID= xxx


[nicolatiana]

No partitions in /dev/sdc

Nicola

[janet]

smiit

Just reinstall with only 2 physical drives attached, clean them with the dd command first.
Then just keep the third drive as a spare that you can connect to a SATA port later & auto/manually add to the array using the Console menu, if & when needed.

You may waste a lot of time troubleshooting the 3 drive array.

[nicolatiana]

Then just keep the third drive as a spare that you can connect to a SATA port later & auto/manually add to the array using the Console menu, if & when needed.
You may waste a lot of time troubleshooting the 3 drive array.

Janet, this means the "hot spare" is not supported at all in SME 9 ? (both in install and in runtime).

Nicola

[stephdl]

You should able to add a drive as hot spare by the CL, but like you noticed there is no partition in /dev/sdc, that could be a start for troubleshoot the issue.
We didn't spend time on the hot spare during the sme9 development mainly because it is the first things that a sysadmin wants to disable...usually.

[janet]

nicolatiana (& smiit et al)

this means the "hot spare" is not supported at all in SME 9 ? (both in install and in runtime).

See the Notes section of SME 9 Release notes at top of this Forum.
http://forums.contribs.org/index.php/topic,51133.msg259082.html#msg259082
IIUIC as per the release notes, the installer does not configure the RAID with a hot standby "spare".

In my earlier post, I just meant that it is easy to add a drive & resync it if one RAID1 drive fails, rather than spend considerable effort working out how to add a hot spare drive.
It would be a very rare occasion that 2 drives would fail at the same time, so there is usually some time available (while running in a single drive degraded array mode), in which to replace a faulty drive.

.....Although I have had both drives of a RAID1 software array fail at the sane time (each failed in a different way) & that was quite distressing at the time, but in that case you need to rely on a good backup, so daily backups are VERY IMPORTANT.

If a admin user physically connects a third drive & somehow tries to configure a RAID1 spare drive during install, then they may have difficulties unravelling the drive configuration.
IMO it is better to install SME as a 2 drive RAID1 array (without a third drive being physically connected) , & after reboot & syncing have completed, you could then follow a generic Linux CentOS Howto to manually add a third hot spare drive to the RAID1 array.
SME Server is a flavour of Linux & you can do just about anything with Linux if you know how.

stephdl's comments also support this ie add a spare using CLI.

[stephdl]

IMHO if you have 3 drives, then you could do a Raid5 and get more drive space.

[smiit]

Thank you all for your insights.

I followed Janet's advice and pulled the 3rd drive and wiped/reinstalled as a standard 2-drive RAID1 default system.

The only clear reference I could find to the 3-drive RAID1 was here:

http://serverfault.com/questions/637410/raid-1-across-3-partitions-what-does-it-mean

Some reports of buffered read speed gains but I can't find any more documentation and don't want to risk any degraded array complications.

[ReetP]

I believe you can run a 3 (or more) drive RAID 1 array quite happily as you note.

If I recall, the correct way to get from 3 drives to 2 drives is to 'grow' the array but from 3 drives to 2 - I asked the same question on the linux RAID dev list a few years back. Slightly illogical, but that's how it works.

e.g.

If your array has 3 drives that you want to reduce to 2 you do something like this :

mdadm --grow --raid-devices=2 /dev/md1


I do run hot spares having added them manually after install, but there can be issues with grub installing correctly (you really need to test it all works as expected before committing data) so have a read around here and the wiki - try this for instance :

http://wiki.contribs.org/Raid#Adding_another_Hard_Drive_Later (this is wrong for v9 and we need to update the wording according to the release notes http://bugs.contribs.org/show_bug.cgi?id=8751)

If you have easy physical access you will probably be best to follow Janets advice and keep the 3rd drive in a box ready to go - I usually do that these days unless the machine is difficult to get to or the person on site can't change it easily. Swap drives, resync, no messing with LVMs et al

If the drive IS a hot spare sitting there idly running, I wonder what the stats are to say the minute one drive fails the hot spare decides it is too old and throws its hand in as well ?

RAID 5 on 3 drives ? I know it is possible (saw an interesting discussion on the RAID list once about technically rebuilding a RAID 5 array from one disk, but I think it has a rebuild time measured in aeons !) but personally I'd want a few more drives in the array, though I am not sure what people would advise is an optimum number.

At a certain point RAID 6 or 10 are probably better options. (6 can suffer two simultaneous drive failures, 10 can as well as long as they are not in the same mirror I think)

Again, trade-offs between security and speed as well. Good backups always required

B. Rgds
John