Topic: Adding extra drive to Raid6 (7 x 2TB drives)

[tdbsoft]

Hoping someone maybe able to help I need to add another 2TB drive to my existing Raid 6

Currently I have 7 x 2TB drives in the array (including 1 spare), SME 8 - 64BIT
I have added the new drive and I can see the drive via fdisk and it has no partitions on the drive.
When I look at the admin console "manage disk redundancy" it is not showing or offering to add the drive.

Is there a link to adding a new drive to raid6 as I have checked the forums and can't find any that are relevant.
Also is there a capacity limit (currently it shows 7.2TB) ?

Any help would be appreciated.

[Stefano]

https://raid.wiki.kernel.org/index.php/Growing

HTH

(BTW, I would test on a small virtual machine before working on real server.....)

[CharlieBrady]

When I look at the admin console "manage disk redundancy" it is not showing or offering to add the drive.

That was only ever developed to support RAID1. Feel free to develop an enhancement, or at least offer to test it if somebody else does.

[stephdl]

Any help would be appreciated.

indeed this is not explained too much in our wiki, it could be nice to get a chapter on how growing an array. Maybe after this you should write it.

This is how i do on my debian (RAID and LVM), sorry it is in french. Think before you type.

https://geekeries.de-labrusse.fr/?p=1674

[tdbsoft]

https://raid.wiki.kernel.org/index.php/Growing

HTH

(BTW, I would test on a small virtual machine before working on real server.....)

Thank you for the link I will investigate this weekend.
When you say test on a virtual machine that sounds like a good idea and have use virtual machines before but how do I allocate 7 virtual hard disk is there a way to do that.

[tdbsoft]

That was only ever developed to support RAID1. Feel free to develop an enhancement, or at least offer to test it if somebody else does.

Thanks for your reply.

It seems to work Ok with Raid 5, but needed extra storage so I setup raid6 as need 10 - 20TB.
I did try to install with 8 x 2TB drives but the systems seem to have issues so removed one drive and it installed.
Would like to have used 3TB drives (4TB even better but may be to new) but fdisk has problems with drives over 2TB.

Has anyone setup raid (SME) with 10TB or more?

[tdbsoft]

indeed this is not explained too much in our wiki, it could be nice to get a chapter on how growing an array. Maybe after this you should write it.

This is how i do on my debian (RAID and LVM), sorry it is in french. Think before you type.

https://geekeries.de-labrusse.fr/?p=1674

Thank you for link I will check it out.

I have a test server so may be I will try it with 1TB drives first to check the adding another drive to raid6.

Wiki - yes if I find a way then will write it up for others as have searched but have not found much info.
 

[stephdl]

When you say test on a virtual machine that sounds like a good idea and have use virtual machines before but how do I allocate 7 virtual hard disk is there a way to do that.

simply use virtualbox with dynamic virtual disks, set the number you want, install a smeserver on this, at the reboot add another virtual hard disk, and play with this.
If i remember well, virtualbox can manages 30 hd sata!

[stephdl]

Is there a link to adding a new drive to raid6 as I have checked the forums and can't find any that are relevant.
Also is there a capacity limit (currently it shows 7.2TB) ?

it seems good for me

(2G*7)-(2*2G)-2G        =8G
          raid6   spare

[stephdl]

It seems to work Ok with Raid 5, but needed extra storage so I setup raid6 as need 10 - 20TB.
I did try to install with 8 x 2TB drives but the systems seem to have issues so removed one drive and it installed.
Would like to have used 3TB drives (4TB even better but may be to new) but fdisk has problems with drives over 2TB.

Has anyone setup raid (SME) with 10TB or more?

-This version of fdisk doesn't support gpt partition, therefore you can not support HD more than 2G, two solutions, wait about sme9 or virtualise your sme for mail, web on other OS.
-it seems that you want to be bigger than google, so think about backups, because the raid is not a backup solution. However i never tested with as much HD like you on a sme server, so i do not know how the sme is working with 10 HD...never tried
A friend of mine aims to get a large array of HD, many more you can imagine, but it gets a second one to backup the first.

[stephdl]

I come with bad news, your issue is interesting and i googled a bit on a bug i found when i tried to add a new hd to a raid6 of 9hd (of course in a virtual environment).

i have had this error when i try to  grow the array

[root@smeraid ~]# mdadm --grow --raid-devices=10 /dev/md2
mdadm: Need to backup 6144K of critical section..
mdadm: Cannot set device size/shape for /dev/md2: Invalid argument

after googling i have seen that the kernel 2.6.18 (actual kernel of sme server) has a bug and you can not resize a raid6, our cousin (clearos) has the same problem, has suggested this post

http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,26/func,view/id,12396/

you can see that from Centos
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=24121

as you can see the solution found is dangerous on a production server : upgrade the kernel, boot on the new kernel, shaping the array, reboot on sme kernel, remove the other kernel...no really a health walk.

i think really that all your command need to be tested in a virtual environment.

[tdbsoft]

it seems good for me

(2G*7)-(2*2G)-2G        =8G
          raid6   spare

Thanks for your help Stephdl

My reason for asking about the limit was because I originally tried to install SME 8 with 8 x 2TB drives but kept failing at the format stage.
When I removed 1 drive 7x 2TB all ran as expected.

[tdbsoft]

simply use virtualbox with dynamic virtual disks, set the number you want, install a smeserver on this, at the reboot add another virtual hard disk, and play with this.
If i remember well, virtualbox can manages 30 hd sata!

Thanks Stephdl

Useful to know not sure I will use this time as seem you have answer the question about adding drives in raid6 IE not easy with out messing with the kernal.
Still the dynamic virtual disks could be useful in future testing etc.

[tdbsoft]

I come with bad news, your issue is interesting and i googled a bit on a bug i found when i tried to add a new hd to a raid6 of 9hd (of course in a virtual environment).

i have had this error when i try to  grow the array

[root@smeraid ~]# mdadm --grow --raid-devices=10 /dev/md2
mdadm: Need to backup 6144K of critical section..
mdadm: Cannot set device size/shape for /dev/md2: Invalid argument

after googling i have seen that the kernel 2.6.18 (actual kernel of sme server) has a bug and you can not resize a raid6, our cousin (clearos) has the same problem, has suggested this post

http://www.clearfoundation.com/component/option,com_kunena/Itemid,232/catid,26/func,view/id,12396/

you can see that from Centos
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=24121

as you can see the solution found is dangerous on a production server : upgrade the kernel, boot on the new kernel, shaping the array, reboot on sme kernel, remove the other kernel...no really a health walk.

i think really that all your command need to be tested in a virtual environment.

Thanks for your help Stephdl

I have decided for easy of rebuild for the moment to try another method to overcome the problem
I have already built a new test server using 1 x 1TB to hold the SME 8 system and 4 x 3TB drives hardware raid10 (need to add 2 or 3 more drives)
I think final build my be 1 x 1TB drive and 6 x 3TB drives using hardware raid (at the moment can only do 4 as need a bigger raid card).

Anyway have tested and SME and formatted and setup the drives as raid 1 = 12TB and then tried raid10 = 6TB + 2stripes both seem to be working.  Would prefer to use SME software raid as I have found it to be very reliable but for the time being will use either just the hardware raid or maybe a combination of software and hardware raid.

Anyway thanks for your help I appreciate the time you have taken to answer my questions and make suggestions etc, cheers.

[stephdl]

My reason for asking about the limit was because I originally tried to install SME 8 with 8 x 2TB drives but kept failing at the format stage.
When I removed 1 drive 7x 2TB all ran as expected.

Very strange i have played a bit yesterday night on raid in virtualbox, with 9 drives (10G each) initially and adding one more after the installation, without problem. But of course by default it is a raid6, you have to start the installer with "sme raid=5" if you do not  want a raid6 array. See http://wiki.contribs.org/Booting for all options.

i wrote a page on the wiki on how grow an array of raid5  http://wiki.contribs.org/Raid:Growing you could read this as example.

I have decided for easy of rebuild for the moment to try another method to overcome the problem
I have already built a new test server using 1 x 1TB to hold the SME 8 system and 4 x 3TB drives hardware raid10 (need to add 2 or 3 more drives)
I think final build my be 1 x 1TB drive and 6 x 3TB drives using hardware raid (at the moment can only do 4 as need a bigger raid card).

I do not believe that SME can handle Hard drives more than 2TB, i can be wrong, but you should verify before to buy it (look after CENTOS5 for more feedbacks)
moreover you should have two disks in raid 1 for the operating system, even not very large, it can be useful if one of them crash.

[tdbsoft]

Very strange i have played a bit yesterday night on raid in virtualbox, with 9 drives (10G each) initially and adding one more after the installation, without problem. But of course by default it is a raid6, you have to start the installer with "sme raid=5" if you do not  want a raid6 array. See http://wiki.contribs.org/Booting for all options.

i wrote a page on the wiki on how grow an array of raid5  http://wiki.contribs.org/Raid:Growing you could read this as example.

I do not believe that SME can handle Hard drives more than 2TB, i can be wrong, but you should verify before to buy it (look after CENTOS5 for more feedbacks)
moreover you should have two disks in raid 1 for the operating system, even not very large, it can be useful if one of them crash.

Hi Stephdl,

Using the hardware raid with the 3TB drives and PARTED to format etc, SME can see the hardware raided drives and in hardware raid0 4drives = 12TB I then changed to raid10 and 4drives = 6TB so it seems to work Ok in SME, I just need a raid card that supports 6 SATA ports.

Are you saying that with raid5 you can add 7 or more 2TB drives using your wiki instructions if so that would be perfect.
I thought that I needed raid6 to allow me to use 7 drives or more see wiki
http://wiki.contribs.org/Raid maybe I miss-read as it shows raid 5 as 5drives + 1spare (SME auto configuration) but dose not say how many drives can be added in raid5.

Can you confirm if it can handle more that 7 x 2TB drives in raid 5 by adding after install I need maybe 10 drives 2TB in total.
If it dose then I will use the test server to backup the production server and take over the role while if redo the server in raid 5 and then add the drives etc as I am fairly sure you cannot downgrade from raid6 to raid5.

Thanks again for your help.

[stephdl]

Using the hardware raid with the 3TB drives and PARTED to format etc, SME can see the hardware raided drives and in hardware raid0 4drives = 12TB I then changed to raid10 and 4drives = 6TB so it seems to work Ok in SME, I just need a raid card that supports 6 SATA ports.

I agree, the raid is handled by a raid hardware card, so the sme see only one disk. I was speaking about software raid and partition by fdisk. it is a bit different in this case.

Are you saying that with raid5 you can add 7 or more 2TB drives using your wiki instructions if so that would be perfect.
I thought that I needed raid6 to allow me to use 7 drives or more see wiki
http://wiki.contribs.org/Raid maybe I miss-read as it shows raid 5 as 5drives + 1spare (SME auto configuration) but dose not say how many drives can be added in raid5.
Can you confirm if it can handle more that 7 x 2TB drives in raid 5 by adding after install I need maybe 10 drives 2TB in total.

How many, i do not know, but for a purpose of test i did an installation of 9 drives (8 in raid5 array and one spare) and after reboot i have added one more without problem in raid5. Of course it is in a virtual environment, but if you send the real drives i would test for you
If you want raid5 and no spare (very dangerous) at boot prompt type "sme raid=5 nospare"
You can see all booting option http://wiki.contribs.org/Booting

If it dose then I will use the test server to backup the production server and take over the role while if redo the server in raid 5 and then add the drives etc as I am fairly sure you cannot downgrade from raid6 to raid5.

i do not know, maybe you should reinstall the server.

[stephdl]

If you have some problem to add a HD to an array, you can erase the whole or a part of the disk with the command dd

to erase the partition

Code: [Select]

dd if=/dev/zero of=/dev/sdX bs=512 count=1to erase the drive

Code: [Select]

dd if=/dev/zero of=/dev/sdX
THINK BEFORE YOU TYPE

[tdbsoft]

I agree, the raid is handled by a raid hardware card, so the sme see only one disk. I was speaking about software raid and partition by fdisk. it is a bit different in this case.How many, i do not know, but for a purpose of test i did an installation of 9 drives (8 in raid5 array and one spare) and after reboot i have added one more without problem in raid5. Of course it is in a virtual environment, but if you send the real drives i would test for you
If you want raid5 and no spare (very dangerous) at boot prompt type "sme raid=5 nospare"
i do not know, maybe you should reinstall the server.

Thanks for your help

We have decided to backup the main server then re-install with raid5 then add the extra drives as per your wiki, it will likely take a week by the time we get it back up.
Failing that we will setup with two raid arrays one software raid using 7x 2TB drives and the other hardware raid5 (or raid10) with 4x (or possible more) 3TB drives.
That should give plenty of storage for while, hopefully by the time we need more storage GPT will be supported.

NB. We did try the Virtual server with 8x drives and it did work as you said so hopefully it will work on the physical drives   

Will let you know how it goes next week. Cheers TDB

[stephdl]

do no hesitate to add comments or feedbacks on the relevant  wiki page

[stephdl]

Thanks for your help

We have decided to backup the main server then re-install with raid5 then add the extra drives as per your wiki, it will likely take a week by the time we get it back up.

sorry if i do not understand,i'm french, but why do you want to add another drive if you do a fresh install, put simply all your disks and install sme server

[tdbsoft]

sorry if i do not understand,i'm french, but why do you want to add another drive if you do a fresh install, put simply all your disks and install sme server

Sorry you are correct will add all drives and use the command to -raid5 to install, but still will take along time as 4TB in use that need to be transfer to the Temp server also Sync of drives took 3 days last time.

But thanks that will save some time.

[stephdl]

Thanks for your help

It is the Koozali SME Server project you need to thank, me i'm just an actor of this world human foundation, try to give some time,  help in buzilla, write an article of documentation, give some money to support the development, and why not become an actor you too of this marvellous project.

[tdbsoft]

It is the Koozali SME Server project you need to thank, me i'm just an actor of this world human foundation, try to give some time,  help in buzilla, write an article of documentation, give some money to support the development, and why not become an actor you too of this marvellous project.

Yes agreed the SME project is a worthy cause and in this time-poor world it is too easy forget to give a little.
Will write up a the basic details of what we have found about creating a server > 10TB and about the hardware raid for 3TB drives as it may help someone avoid the same issues.

[tdbsoft]

Update

I have tested installing SME with the following parameter of "sme raid=5"
9 x 2TB drives

This gives an error of
An error occurred trying to format the main/root. This problem is serious, and the
Install cannot continue.
Press Ok to reboot...

Booting into UBCD and checked the drives with fdisk it shows

                   Start             End             Block
sdc1                  1              13           104391
Partition1 does not start on physical sector boundry
sdc2                 14        243201    1953407610
Partition2 does not start on physical sector boundry

These errors/messages showed on all the drives

I then tried with
8 x 2TB drives - same result
7 x 2TB drives - same result
6 x 2TB drives - Format started Ok

As I had 6 x drives attached to the mainboard and 3x on the Rocket PCI-Express card
I thought it might have been due to the last 3 x drives on the add-in raid card not working so to test
I tried 5 drives on the main board and one on the add-in card - this tested Ok.

Decided to try installing with 6 x 2TB drives using "sme raid=5" once if sync's all the drives I will try to add a seventh drive.
Will let you know how it goes..

[stephdl]

you can test to add directly these three drives by the mdadm command as i wrote in the wiki (after the installation on 6 drives), except when you set the number of disk in the raid, keep in mind to declare one disk less if you want to have a spare disk.
it is possible that
-there is a bug in the automatic partition
-the additional raid card is not really compatible with centos5
-the automatic partition was not thought for drives of 2T

[tdbsoft]

Results from the test
Installed SME with 6 x 2TB drives (use raid=5 option) all OK.
Updated SME with the latest patches all OK
Shutdown and install another 1 x 2TB drive = now have total of 7 drives

Followed the instructions on Growing Raid
http://wiki.contribs.org/Raid:Growing

Waited 3 days for the raid to be rebuilt
Checked the size of the drive but no extra space and raid details look correct
/dev/md1 size displayed is 7.1TB should be 10TB
My Calcs are 7 drive minus one for spare minus one for strip leaves 5 drives * 2TB ~ expected size to be about 9 - 10TB

you can download a text file of the output from pvdisplay and other commands here http://www.tdb.com.au/dl/raid.txt

Any suggestions, should I try add another drive or have I missed some important step?

[stephdl]

you can download a text file of the output from pvdisplay and other commands here http://www.tdb.com.au/dl/raid.txt

Any suggestions, should I try add another drive or have I missed some important step?

Nope the partition seems good as i see, only md2 is concerned by the spare drive however it is what i expected. No i think that you can add all other drives.
If you have time you can add them one after one, but it could work if you add them all in one time, except that you may have a bigger risk to loose your data , yes adding a drive to a raid is not without risks. Eventually you can make a backup of drive as explained in documentation, but you need a large array.

 

Code: [Select]

The process can take many hours or even days. There is a critical section at start, which cannot be backed up. To allow recovery after unexpected power failure, an additional option --backup-file= can be specified. Make sure this file is on a different disk or it defeats the purpose.

mdadm --grow --raid-devices=5 --backup-file=/root/grow_md1.bak /dev/md1
mdadm --grow --raid-devices=4 --backup-file=/root/grow_md2.bak /dev/md2

If you have some troubles to add a drive to an array, you should try to erase the mbr, or the whole drive with the dd command

#dd if=/dev/zero of=/dev/sdX bs=512 count=1

If i say that it is because in the first attempt you were not be able to reshape the array with 9 drives and i suspect that one of them could make some noise when you want to use it.

[tdbsoft]

Think re-size is now working I missed one line of the Wiki -
>> Once the construction is complete, we have to set the LVM to use the whole space
Need to run commands below this note AFTER the syncing of the new drive is complete

Once I re-ran the instructions below this note it took a couple of hours to run the last one but after it was complete
I ran "df -h" and now it is reporting 8.9TB as the size.

So I am going to add 2 x more drives in one go if it all works then I will report back (will like take about 4 days).

[tdbsoft]

Hi all,

For all that are interested here is the wiki link to a howto build at 19TB raid array
http://wiki.contribs.org/Creating_large_raid5_array

[stephdl]

Hi all,

For all that are interested here is the wiki link to a howto build at 19TB raid array
http://wiki.contribs.org/Creating_large_raid5_array

for the record what tdbsoft has written is a howto on how use hard drive of 4TB with SME Server, of course you cannot install the system but you can use them for a data partition. Thanks tdbsoft