Koozali.org: home of the SME Server

Adding extra drive to Raid6 (7 x 2TB drives)

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Adding extra drive to Raid6 (7 x 2TB drives)
« on: October 28, 2013, 12:38:43 PM »
Hoping someone maybe able to help I need to add another 2TB drive to my existing Raid 6

Currently I have 7 x 2TB drives in the array (including 1 spare), SME 8 - 64BIT
I have added the new drive and I can see the drive via fdisk and it has no partitions on the drive.
When I look at the admin console "manage disk redundancy" it is not showing or offering to add the drive.

Is there a link to adding a new drive to raid6 as I have checked the forums and can't find any that are relevant.
Also is there a capacity limit (currently it shows 7.2TB) ?

Any help would be appreciated.

Offline Stefano

  • *
  • 10,836
  • +2/-0
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #1 on: October 28, 2013, 12:46:34 PM »
https://raid.wiki.kernel.org/index.php/Growing

HTH

(BTW, I would test on a small virtual machine before working on real server.....)

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #2 on: October 28, 2013, 03:58:59 PM »
When I look at the admin console "manage disk redundancy" it is not showing or offering to add the drive.

That was only ever developed to support RAID1. Feel free to develop an enhancement, or at least offer to test it if somebody else does.

Offline stephdl

  • *
  • 1,519
  • +0/-0
    • Linux et Geekeries
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #3 on: October 28, 2013, 08:37:52 PM »
Any help would be appreciated.

indeed this is not explained too much in our wiki, it could be nice to get a chapter on how growing an array. Maybe after this you should write it.

This is how i do on my debian (RAID and LVM), sorry it is in french. Think before you type.

https://geekeries.de-labrusse.fr/?p=1674
« Last Edit: October 28, 2013, 08:39:53 PM by stephdl »
See http://wiki.contribs.org/Koozali_Foundation
irc : Freenode #sme_server #sme-fr

!!! Please write your knowledge to the Wiki !!!

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #4 on: October 29, 2013, 02:45:16 AM »
https://raid.wiki.kernel.org/index.php/Growing

HTH

(BTW, I would test on a small virtual machine before working on real server.....)

Thank you for the link I will investigate this weekend.
When you say test on a virtual machine that sounds like a good idea and have use virtual machines before but how do I allocate 7 virtual hard disk is there a way to do that.

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #5 on: October 29, 2013, 02:53:42 AM »
That was only ever developed to support RAID1. Feel free to develop an enhancement, or at least offer to test it if somebody else does.

Thanks for your reply.

It seems to work Ok with Raid 5, but needed extra storage so I setup raid6 as need 10 - 20TB.
I did try to install with 8 x 2TB drives but the systems seem to have issues so removed one drive and it installed.
Would like to have used 3TB drives (4TB even better but may be to new) but fdisk has problems with drives over 2TB.

Has anyone setup raid (SME) with 10TB or more?

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #6 on: October 29, 2013, 03:02:37 AM »
indeed this is not explained too much in our wiki, it could be nice to get a chapter on how growing an array. Maybe after this you should write it.

This is how i do on my debian (RAID and LVM), sorry it is in french. Think before you type.

https://geekeries.de-labrusse.fr/?p=1674

Thank you for link I will check it out.

I have a test server so may be I will try it with 1TB drives first to check the adding another drive to raid6.

Wiki - yes if I find a way then will write it up for others as have searched but have not found much info.
 

Offline stephdl

  • *
  • 1,519
  • +0/-0
    • Linux et Geekeries
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #7 on: October 29, 2013, 01:43:55 PM »
When you say test on a virtual machine that sounds like a good idea and have use virtual machines before but how do I allocate 7 virtual hard disk is there a way to do that.
simply use virtualbox with dynamic virtual disks, set the number you want, install a smeserver on this, at the reboot add another virtual hard disk, and play with this.
If i remember well, virtualbox can manages 30 hd sata!
See http://wiki.contribs.org/Koozali_Foundation
irc : Freenode #sme_server #sme-fr

!!! Please write your knowledge to the Wiki !!!

Offline stephdl

  • *
  • 1,519
  • +0/-0
    • Linux et Geekeries
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #8 on: October 29, 2013, 01:51:23 PM »
Is there a link to adding a new drive to raid6 as I have checked the forums and can't find any that are relevant.
Also is there a capacity limit (currently it shows 7.2TB) ?

it seems good for me

(2G*7)-(2*2G)-2G        =8G
          raid6   spare
See http://wiki.contribs.org/Koozali_Foundation
irc : Freenode #sme_server #sme-fr

!!! Please write your knowledge to the Wiki !!!

Offline stephdl

  • *
  • 1,519
  • +0/-0
    • Linux et Geekeries
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #9 on: October 29, 2013, 02:03:17 PM »
It seems to work Ok with Raid 5, but needed extra storage so I setup raid6 as need 10 - 20TB.
I did try to install with 8 x 2TB drives but the systems seem to have issues so removed one drive and it installed.
Would like to have used 3TB drives (4TB even better but may be to new) but fdisk has problems with drives over 2TB.

Has anyone setup raid (SME) with 10TB or more?
-This version of fdisk doesn't support gpt partition, therefore you can not support HD more than 2G, two solutions, wait about sme9 or virtualise your sme for mail, web on other OS.
-it seems that you want to be bigger than google, so think about backups, because the raid is not a backup solution. However i never tested with as much HD like you on a sme server, so i do not know how the sme is working with 10 HD...never tried
A friend of mine aims to get a large array of HD, many more you can imagine, but it gets a second one to backup the first.
See http://wiki.contribs.org/Koozali_Foundation
irc : Freenode #sme_server #sme-fr

!!! Please write your knowledge to the Wiki !!!

Offline stephdl

  • *
  • 1,519
  • +0/-0
    • Linux et Geekeries
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #10 on: October 29, 2013, 07:43:05 PM »
I come with bad news, your issue is interesting and i googled a bit on a bug i found when i tried to add a new hd to a raid6 of 9hd (of course in a virtual environment).

i have had this error when i try to  grow the array

[root@smeraid ~]# mdadm --grow --raid-devices=10 /dev/md2
mdadm: Need to backup 6144K of critical section..
mdadm: Cannot set device size/shape for /dev/md2: Invalid argument

after googling i have seen that the kernel 2.6.18 (actual kernel of sme server) has a bug and you can not resize a raid6, our cousin (clearos) has the same problem, has suggested this post

http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,26/func,view/id,12396/

you can see that from Centos
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=24121

as you can see the solution found is dangerous on a production server : upgrade the kernel, boot on the new kernel, shaping the array, reboot on sme kernel, remove the other kernel...no really a health walk.

i think really that all your command need to be tested in a virtual environment.
« Last Edit: October 29, 2013, 07:44:43 PM by stephdl »
See http://wiki.contribs.org/Koozali_Foundation
irc : Freenode #sme_server #sme-fr

!!! Please write your knowledge to the Wiki !!!

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #11 on: October 30, 2013, 03:08:31 AM »
it seems good for me

(2G*7)-(2*2G)-2G        =8G
          raid6   spare

Thanks for your help Stephdl

My reason for asking about the limit was because I originally tried to install SME 8 with 8 x 2TB drives but kept failing at the format stage.
When I removed 1 drive 7x 2TB all ran as expected.

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #12 on: October 30, 2013, 03:12:09 AM »
simply use virtualbox with dynamic virtual disks, set the number you want, install a smeserver on this, at the reboot add another virtual hard disk, and play with this.
If i remember well, virtualbox can manages 30 hd sata!

Thanks Stephdl

Useful to know not sure I will use this time as seem you have answer the question about adding drives in raid6 IE not easy with out messing with the kernal.
Still the dynamic virtual disks could be useful in future testing etc.

Offline tdbsoft

  • *
  • 81
  • +0/-0
    • http://www.tdb.com.au
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #13 on: October 30, 2013, 03:25:39 AM »
I come with bad news, your issue is interesting and i googled a bit on a bug i found when i tried to add a new hd to a raid6 of 9hd (of course in a virtual environment).

i have had this error when i try to  grow the array

[root@smeraid ~]# mdadm --grow --raid-devices=10 /dev/md2
mdadm: Need to backup 6144K of critical section..
mdadm: Cannot set device size/shape for /dev/md2: Invalid argument

after googling i have seen that the kernel 2.6.18 (actual kernel of sme server) has a bug and you can not resize a raid6, our cousin (clearos) has the same problem, has suggested this post

http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,26/func,view/id,12396/

you can see that from Centos
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=24121

as you can see the solution found is dangerous on a production server : upgrade the kernel, boot on the new kernel, shaping the array, reboot on sme kernel, remove the other kernel...no really a health walk.

i think really that all your command need to be tested in a virtual environment.

Thanks for your help Stephdl

I have decided for easy of rebuild for the moment to try another method to overcome the problem
I have already built a new test server using 1 x 1TB to hold the SME 8 system and 4 x 3TB drives hardware raid10 (need to add 2 or 3 more drives)
I think final build my be 1 x 1TB drive and 6 x 3TB drives using hardware raid (at the moment can only do 4 as need a bigger raid card).

Anyway have tested and SME and formatted and setup the drives as raid 1 = 12TB and then tried raid10 = 6TB + 2stripes both seem to be working.  Would prefer to use SME software raid as I have found it to be very reliable but for the time being will use either just the hardware raid or maybe a combination of software and hardware raid.

Anyway thanks for your help I appreciate the time you have taken to answer my questions and make suggestions etc, cheers.

Offline stephdl

  • *
  • 1,519
  • +0/-0
    • Linux et Geekeries
Re: Adding extra drive to Raid6 (7 x 2TB drives)
« Reply #14 on: October 30, 2013, 07:35:01 AM »
My reason for asking about the limit was because I originally tried to install SME 8 with 8 x 2TB drives but kept failing at the format stage.
When I removed 1 drive 7x 2TB all ran as expected.

Very strange i have played a bit yesterday night on raid in virtualbox, with 9 drives (10G each) initially and adding one more after the installation, without problem. But of course by default it is a raid6, you have to start the installer with "sme raid=5" if you do not  want a raid6 array. See http://wiki.contribs.org/Booting for all options.

i wrote a page on the wiki on how grow an array of raid5  http://wiki.contribs.org/Raid:Growing you could read this as example.

I have decided for easy of rebuild for the moment to try another method to overcome the problem
I have already built a new test server using 1 x 1TB to hold the SME 8 system and 4 x 3TB drives hardware raid10 (need to add 2 or 3 more drives)
I think final build my be 1 x 1TB drive and 6 x 3TB drives using hardware raid (at the moment can only do 4 as need a bigger raid card).

I do not believe that SME can handle Hard drives more than 2TB, i can be wrong, but you should verify before to buy it (look after CENTOS5 for more feedbacks)
moreover you should have two disks in raid 1 for the operating system, even not very large, it can be useful if one of them crash.

See http://wiki.contribs.org/Koozali_Foundation
irc : Freenode #sme_server #sme-fr

!!! Please write your knowledge to the Wiki !!!