Topic: Adding extra drive to Raid6 (7 x 2TB drives)

[tdbsoft]

Very strange i have played a bit yesterday night on raid in virtualbox, with 9 drives (10G each) initially and adding one more after the installation, without problem. But of course by default it is a raid6, you have to start the installer with "sme raid=5" if you do not  want a raid6 array. See http://wiki.contribs.org/Booting for all options.

i wrote a page on the wiki on how grow an array of raid5  http://wiki.contribs.org/Raid:Growing you could read this as example.

I do not believe that SME can handle Hard drives more than 2TB, i can be wrong, but you should verify before to buy it (look after CENTOS5 for more feedbacks)
moreover you should have two disks in raid 1 for the operating system, even not very large, it can be useful if one of them crash.

Hi Stephdl,

Using the hardware raid with the 3TB drives and PARTED to format etc, SME can see the hardware raided drives and in hardware raid0 4drives = 12TB I then changed to raid10 and 4drives = 6TB so it seems to work Ok in SME, I just need a raid card that supports 6 SATA ports.

Are you saying that with raid5 you can add 7 or more 2TB drives using your wiki instructions if so that would be perfect.
I thought that I needed raid6 to allow me to use 7 drives or more see wiki
http://wiki.contribs.org/Raid maybe I miss-read as it shows raid 5 as 5drives + 1spare (SME auto configuration) but dose not say how many drives can be added in raid5.

Can you confirm if it can handle more that 7 x 2TB drives in raid 5 by adding after install I need maybe 10 drives 2TB in total.
If it dose then I will use the test server to backup the production server and take over the role while if redo the server in raid 5 and then add the drives etc as I am fairly sure you cannot downgrade from raid6 to raid5.

Thanks again for your help.

[stephdl]

Using the hardware raid with the 3TB drives and PARTED to format etc, SME can see the hardware raided drives and in hardware raid0 4drives = 12TB I then changed to raid10 and 4drives = 6TB so it seems to work Ok in SME, I just need a raid card that supports 6 SATA ports.

I agree, the raid is handled by a raid hardware card, so the sme see only one disk. I was speaking about software raid and partition by fdisk. it is a bit different in this case.

Are you saying that with raid5 you can add 7 or more 2TB drives using your wiki instructions if so that would be perfect.
I thought that I needed raid6 to allow me to use 7 drives or more see wiki
http://wiki.contribs.org/Raid maybe I miss-read as it shows raid 5 as 5drives + 1spare (SME auto configuration) but dose not say how many drives can be added in raid5.
Can you confirm if it can handle more that 7 x 2TB drives in raid 5 by adding after install I need maybe 10 drives 2TB in total.

How many, i do not know, but for a purpose of test i did an installation of 9 drives (8 in raid5 array and one spare) and after reboot i have added one more without problem in raid5. Of course it is in a virtual environment, but if you send the real drives i would test for you
If you want raid5 and no spare (very dangerous) at boot prompt type "sme raid=5 nospare"
You can see all booting option http://wiki.contribs.org/Booting

If it dose then I will use the test server to backup the production server and take over the role while if redo the server in raid 5 and then add the drives etc as I am fairly sure you cannot downgrade from raid6 to raid5.

i do not know, maybe you should reinstall the server.

[stephdl]

If you have some problem to add a HD to an array, you can erase the whole or a part of the disk with the command dd

to erase the partition

Code: [Select]

dd if=/dev/zero of=/dev/sdX bs=512 count=1to erase the drive

Code: [Select]

dd if=/dev/zero of=/dev/sdX
THINK BEFORE YOU TYPE

[tdbsoft]

I agree, the raid is handled by a raid hardware card, so the sme see only one disk. I was speaking about software raid and partition by fdisk. it is a bit different in this case.How many, i do not know, but for a purpose of test i did an installation of 9 drives (8 in raid5 array and one spare) and after reboot i have added one more without problem in raid5. Of course it is in a virtual environment, but if you send the real drives i would test for you
If you want raid5 and no spare (very dangerous) at boot prompt type "sme raid=5 nospare"
i do not know, maybe you should reinstall the server.

Thanks for your help

We have decided to backup the main server then re-install with raid5 then add the extra drives as per your wiki, it will likely take a week by the time we get it back up.
Failing that we will setup with two raid arrays one software raid using 7x 2TB drives and the other hardware raid5 (or raid10) with 4x (or possible more) 3TB drives.
That should give plenty of storage for while, hopefully by the time we need more storage GPT will be supported.

NB. We did try the Virtual server with 8x drives and it did work as you said so hopefully it will work on the physical drives   

Will let you know how it goes next week. Cheers TDB

[stephdl]

do no hesitate to add comments or feedbacks on the relevant  wiki page

[stephdl]

Thanks for your help

We have decided to backup the main server then re-install with raid5 then add the extra drives as per your wiki, it will likely take a week by the time we get it back up.

sorry if i do not understand,i'm french, but why do you want to add another drive if you do a fresh install, put simply all your disks and install sme server

[tdbsoft]

sorry if i do not understand,i'm french, but why do you want to add another drive if you do a fresh install, put simply all your disks and install sme server

Sorry you are correct will add all drives and use the command to -raid5 to install, but still will take along time as 4TB in use that need to be transfer to the Temp server also Sync of drives took 3 days last time.

But thanks that will save some time.

[stephdl]

Thanks for your help

It is the Koozali SME Server project you need to thank, me i'm just an actor of this world human foundation, try to give some time,  help in buzilla, write an article of documentation, give some money to support the development, and why not become an actor you too of this marvellous project.

[tdbsoft]

It is the Koozali SME Server project you need to thank, me i'm just an actor of this world human foundation, try to give some time,  help in buzilla, write an article of documentation, give some money to support the development, and why not become an actor you too of this marvellous project.

Yes agreed the SME project is a worthy cause and in this time-poor world it is too easy forget to give a little.
Will write up a the basic details of what we have found about creating a server > 10TB and about the hardware raid for 3TB drives as it may help someone avoid the same issues.

[tdbsoft]

Update

I have tested installing SME with the following parameter of "sme raid=5"
9 x 2TB drives

This gives an error of
An error occurred trying to format the main/root. This problem is serious, and the
Install cannot continue.
Press Ok to reboot...

Booting into UBCD and checked the drives with fdisk it shows

                   Start             End             Block
sdc1                  1              13           104391
Partition1 does not start on physical sector boundry
sdc2                 14        243201    1953407610
Partition2 does not start on physical sector boundry

These errors/messages showed on all the drives

I then tried with
8 x 2TB drives - same result
7 x 2TB drives - same result
6 x 2TB drives - Format started Ok

As I had 6 x drives attached to the mainboard and 3x on the Rocket PCI-Express card
I thought it might have been due to the last 3 x drives on the add-in raid card not working so to test
I tried 5 drives on the main board and one on the add-in card - this tested Ok.

Decided to try installing with 6 x 2TB drives using "sme raid=5" once if sync's all the drives I will try to add a seventh drive.
Will let you know how it goes..

[stephdl]

you can test to add directly these three drives by the mdadm command as i wrote in the wiki (after the installation on 6 drives), except when you set the number of disk in the raid, keep in mind to declare one disk less if you want to have a spare disk.
it is possible that
-there is a bug in the automatic partition
-the additional raid card is not really compatible with centos5
-the automatic partition was not thought for drives of 2T

[tdbsoft]

Results from the test
Installed SME with 6 x 2TB drives (use raid=5 option) all OK.
Updated SME with the latest patches all OK
Shutdown and install another 1 x 2TB drive = now have total of 7 drives

Followed the instructions on Growing Raid
http://wiki.contribs.org/Raid:Growing

Waited 3 days for the raid to be rebuilt
Checked the size of the drive but no extra space and raid details look correct
/dev/md1 size displayed is 7.1TB should be 10TB
My Calcs are 7 drive minus one for spare minus one for strip leaves 5 drives * 2TB ~ expected size to be about 9 - 10TB

you can download a text file of the output from pvdisplay and other commands here http://www.tdb.com.au/dl/raid.txt

Any suggestions, should I try add another drive or have I missed some important step?

[stephdl]

you can download a text file of the output from pvdisplay and other commands here http://www.tdb.com.au/dl/raid.txt

Any suggestions, should I try add another drive or have I missed some important step?

Nope the partition seems good as i see, only md2 is concerned by the spare drive however it is what i expected. No i think that you can add all other drives.
If you have time you can add them one after one, but it could work if you add them all in one time, except that you may have a bigger risk to loose your data , yes adding a drive to a raid is not without risks. Eventually you can make a backup of drive as explained in documentation, but you need a large array.

 

Code: [Select]

The process can take many hours or even days. There is a critical section at start, which cannot be backed up. To allow recovery after unexpected power failure, an additional option --backup-file= can be specified. Make sure this file is on a different disk or it defeats the purpose.

mdadm --grow --raid-devices=5 --backup-file=/root/grow_md1.bak /dev/md1
mdadm --grow --raid-devices=4 --backup-file=/root/grow_md2.bak /dev/md2

If you have some troubles to add a drive to an array, you should try to erase the mbr, or the whole drive with the dd command

#dd if=/dev/zero of=/dev/sdX bs=512 count=1

If i say that it is because in the first attempt you were not be able to reshape the array with 9 drives and i suspect that one of them could make some noise when you want to use it.

[tdbsoft]

Think re-size is now working I missed one line of the Wiki -
>> Once the construction is complete, we have to set the LVM to use the whole space
Need to run commands below this note AFTER the syncing of the new drive is complete

Once I re-ran the instructions below this note it took a couple of hours to run the last one but after it was complete
I ran "df -h" and now it is reporting 8.9TB as the size.

So I am going to add 2 x more drives in one go if it all works then I will report back (will like take about 4 days).

[tdbsoft]

Hi all,

For all that are interested here is the wiki link to a howto build at 19TB raid array
http://wiki.contribs.org/Creating_large_raid5_array