Topic: [ANNOUNCE] Add Extra Disk to LVM on Raid 5 HOWTO

[mudtoe]

Here it is:


[root@srw-smeserver ~]# fdisk -l

Disk /dev/sda: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14        2610    20860402+  fd  Linux raid autodetect

Disk /dev/md1: 106 MB, 106823680 bytes
2 heads, 4 sectors/track, 26080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md2: 21.3 GB, 21360934912 bytes
2 heads, 4 sectors/track, 5215072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md2 doesn't contain a valid partition table
[root@srw-smeserver ~]#

[ldkeen]

OK, follow the steps below:
Partition the free space:

Code: [Select]

#fdisk /dev/sda
type "n" for new
type "p" for primary
type "3" for the third partition
accept the default starting cylinder of 2611
accept the default ending cylinder of 13054
now type "t" for type
enter "3" for the third partiton
then "fd" for Linux Raid
now type "w" to write out the partition table
Initialise the new Physical Volume for use with LVM

Code: [Select]

#pvcreate /dev/sda3
Extend the existing Volume Group onto the new PV

Code: [Select]

#vgextend main /dev/sda3
Find how much free space you have

Code: [Select]

#vgdisplay
Resize the Logical Volume using the free space

Code: [Select]

#lvresize -L +80GB /dev/main/root
Resize the filesystem

Code: [Select]

#ext2online -d -v /dev/main/root

This is all untested - so please make sure you have a backup. Let me know how you go.
Lloyd

[mudtoe]

Your procedure worked.  The only thing that had to be changed was that a reboot needed to be peformed after the "fdisk" and before the "pvcreate".  The fdisk put out a message saying that the kernel wouldn't use the new partition table until after a reboot, and even though "fdisk -l" showed the new partition at that point, the pvcreate said that the partition didn't exist.  After a reboot the pvcreate and the rest of the sequence worked just fine.

I do have a couple of remaining questions concerning the resulting configuration.  First, when I do a "fdisk -l" I get some messages about partition tables being invalid.  I'm not sure if it means anything or not, as I'm not sure if what the fdisk is talking about is a real partition or some artifact of the raid setup.  Also the "pvdisplay" shows something a little odd in that one volume is /dev/md2 and the other is /dev/sda3 (I understand what sd type partitions are (SCSI), but not the md ones).    I'm including a list of what things look like now:


login as: root
root@172.26.199.252's password:
Last login: Thu Oct 19 00:28:25 2006



[root@srw-smeserver ~]# fdisk -l

Disk /dev/sda: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14        2610    20860402+  fd  Linux raid autodetect
/dev/sda3            2611       13054    83891430   fd  Linux raid autodetect

Disk /dev/md1: 106 MB, 106823680 bytes
2 heads, 4 sectors/track, 26080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md2: 21.3 GB, 21360934912 bytes
2 heads, 4 sectors/track, 5215072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md2 doesn't contain a valid partition table




[root@srw-smeserver ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/main-root
                       98G  3.1G   90G   4% /
/dev/md1               99M   19M   75M  21% /boot
none                  157M     0  157M   0% /dev/shm





[root@srw-smeserver ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               main
  PV Size               19.88 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              636
  Free PE               0
  Allocated PE          636
  PV UUID               1qQ6Tk-pC59-L661-FkeI-B8Pi-rH6D-GeLzTz

  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               main
  PV Size               80.00 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              2560
  Free PE               0
  Allocated PE          2560
  PV UUID               ZeWoKc-Ld32-t2Nf-QNEq-NbRi-5BTU-o2Bma0






[root@srw-smeserver ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/main/root
  VG Name                main
  LV UUID                kFiXCV-23jG-6RJ6-PzTv-Q1rv-GutR-eJZosL
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                99.38 GB
  Current LE             3180
  Segments               3
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/main/swap
  VG Name                main
  LV UUID                DbEivz-MPkK-RIJO-7OMg-4k6b-St64-Ke6yp3
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                512.00 MB
  Current LE             16
  Segments               1
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:1

[root@srw-smeserver ~]#






It might be nice to package up the final procedure from this effort and publish it along with your other procedure for adding volumes to the raid configuration.   I'd say this thread just became the definitive source for doing disk reorganization on an SME Server installation.

Once again,  thanks much for your assistance.  


mudtoe

[ldkeen]

Mudtoe

First, when I do a "fdisk -l" I get some messages about partition tables being invalid

Are you talking about this message:

Disk /dev/md1 doesn't contain a valid partition table

If so, that is a normal message with software raid. I not exactly sure but I think only physical disks contain a partition table or the partiton table for the metadevice is stored in software. Either way it's nothing to worry about.

"pvdisplay" shows something a little odd in that one volume is /dev/md2 and the other is /dev/sda3

What would have been the best way to go would have been to simply resize /dev/md2, as it stands we have created a new partition and extended the logical volume onto this. I'm in the process of setting up a vm to have a bit more of a play with it. Over the next week or so I'll post some more info. That's the beauty of LVM, there's a few ways to achieve the end result.
Lloyd

[mudtoe]

Yes, those were the messages I was referring to.  What exactly is the /dev/mdx device type?   I know that /dev/hdx refers to ide hard drives, and /dev/sdx are SCSI drives, but the mdx type is new to me, although I'm not a linux expert by any means.   It also seems a bit odd to see the types mixed like they show on the pvdisplay.


I'm assuming that there really isn't any problem (e.g. performance, stablilty, etc.) with the way I have it configured (binding two partitions together in the configuration), versus having expanded the md2 partition, other than just general neatness of the configuration.  As I'm the only person fooling with this, it won't be an issue down the road with regard to someone else trying to figure out why the partition configuration is the way it is.


mudtoe

[ldkeen]

What exactly is the /dev/mdx device type?

md is short for metadevice and is the terminolgy that is used when combining one or more physical drives to create a virtual drive (or metadevice). If you combine 2 x 20Gb disks (which could be /dev/sda1 and /dev/sdc1) to form a raid 1 mirror, that virtual disk would be referred to as /dev/md1. The next raid device would be md2 and so on.

I'm assuming that there really isn't any problem...other than just general neatness

That's about it in a nutshell.
Lloyd

[ldkeen]

Mudtoe,

the best way to go would have been to simply resize /dev/md2

After some further testing I've found out that the metadevice can be resized using the following:

Code: [Select]

#mdadm --grow --size=max /dev/md2
So the more elegant way of doing it would have been:

Code: [Select]

Resize the vm (already done)
Boot from the Finnix CD or sme rescue but don't mount the filesystem, delete partition sda2 and recreate using free space (this doesn't destroy the data, it just updates the partition table with info about the new free space)
Reboot into smeserver
Resize the metadevice
#mdadm -G -z max /dev/md2
#pvresize -v /dev/md2
#vgdisplay (to find free space)
#lvresize -L +80GB /dev/main/root (where 80GB is the free space listed above)
#ext2online -d -v /dev/main/root
Note, I've only partially tested this. I'll fully test this soon and report back.
Regards, Lloyd

[meanpenguin]

What exactly is the /dev/mdx device type?

md is short for metadevice and is the terminolgy that is used when combining one or more physical drives to create a virtual drive (or metadevice). If you combine 2 x 20Gb disks (which could be /dev/sda1 and /dev/sdc1) to form a raid 1 mirror, that virtual disk would be referred to as /dev/md1. The next raid device would be md2 and so on.

I'm assuming that there really isn't any problem...other than just general neatness

That's about it in a nutshell.
Lloyd

I could be wrong but, as a learning exercise....

1.  In general /dev/mdx is a meta device.  Specifically in the stock SME7, the /dev/mdx are raid drives.
2.  SME7 builds LVM drives on top of the raid drives (Even if there is only one HD)

So to look at the SME7 setup - single HD

1.  The installer builds 2, RAID 1 partitions   /dev/md1, and /dev/md2 in degraded mode since there is only 1 hd.
2.  /dev/md1 is the boot partition and is mounted on /boot
3.  /dev/md2 is used as a pv for the LVM
4.  Two logical volumes are built on the pv (/dev/main/swap and /dev/main/root)

Therefore, showing /dev/md2 and /dev/sda3 as the physical volumes in the pvdisplay says that the original is a raid device (/dev/md2) and the new drive (/dev/sda3) is not a raid device.

Please feel free to correct anything in this post...

ed

[Rob Thonus]

Mudtoe,

the best way to go would have been to simply resize /dev/md2

After some further testing I've found out that the metadevice can be resized using the following:

Code: [Select]

#mdadm --grow --size=max /dev/md2
So the more elegant way of doing it would have been:

Code: [Select]

Resize the vm (already done)
Boot from the Finnix CD or sme rescue but don't mount the filesystem, delete partition sda2 and recreate using free space (this doesn't destroy the data, it just updates the partition table with info about the new free space)
Reboot into smeserver
Resize the metadevice
#mdadm -G -z max /dev/md2
#pvresize -v /dev/md2
#vgdisplay (to find free space)
#lvresize -L +80GB /dev/main/root (where 80GB is the free space listed above)
#ext2online -d -v /dev/main/root
Note, I've only partially tested this. I'll fully test this soon and report back.
Regards, Lloyd

I've tested this, since I wanted to place a larger harddisk instead of the old one.
I use a single disk.

I did:

Boot from SME server CD, skip media test, leave it there, do nothing else here.

Goto other console using: CTRL ALT F2
#fdisk /dev/hda
d to delete partition, then 2 for /dev/hda2
n for new partition, 1 for primary partition (not sure of number), partition number 2.
c to change partition type to fd for linux raid.

Then reboot into SME server.
#pvresize -v /dev/hda2
#vgdisplay (to find free space)
#lvresize -L +80GB /dev/main/root (where 80GB is the free space listed above)
#ext2online -d -v /dev/main/root

This worked fine on my machine, thanks everyone for the help this topic provided for me, it solved my problem I was trying to solve for a couple of days.

[tandum]

Thanks for the finnux link. That fixed me.

For those that have a raid 5 array hanging off ibays, here is a method to grow that array with a new disk.

1. boot from the finnux disk you have already made.

2. cat /proc/mdstat    check that the target array is stable.  If it's rebuilding etc, wait untill it's done.

3. mdadm /dev/mdx -a /dev/sdx1  (Add the new disk as a spare raid member)

4. mdadm --grow /dev/mdx --raid-disks=x  (This grows the number of disks to x )

5. Wait several hours ... cat /proc/mdstat for an update

6. e2fsck -f /dev/mdx

7. resize2fs -p /dev/mdx

Woo hoo   /dev/mdx is now bigger.