Topic: Big trouble with raid array

[Sijhtam]

Hello everyone,
I’m having a big fight with my sme 7.2 server.  When I had a power failure I want to restart my server. But after power up it gives a message; unclean stop.
Than I repaired the data by using the fscd. After a new reboot I get the following message:

Error:  /bin/lvm exited abnormally (pid 461)
Creating root device
Mounting root filesystem
Mount error 6 mounting ext3
Mount error 2 mounting none
Switching to new root
Switchingroot: mount failed: 22
Unmount /initrd/dev failed:2
Kernel panic – not syncing: attempted to kill init!


I did not make any backup  but the date contains pictures of my doughter getting born.
So please help me get some data back

Mathijs

[gilsaa]

What if you try to boot on one of the disks only (I assume the disks are mirrored)

[Sijhtam]

I have 4 disks. I found out that one of the disks failed i removed that one and replaced the one for a new one.

This is the message i get:

md: md1 stoped
md: bind<hde1>
raid1: raidset md1 active with 3 out of 4 mirrors
md: md2 stoped
mdadm: /dev/md2 assambled from 2 drives - not enough to start the array. Making divice mapper contole node.
scanning logical volumes
reading all physical volumes. This may take a while...
cdrom: open failed
volume group "main" not found
error: bin/lvm exited abnormaly (pid 459)
Creating root device
Mounting root filesystem
Mount error 6 mounting ext3
Mount error 2 mounting none
Switching to new root
Switchingroot: mount failed: 22
Unmount /initrd/dev failed:2
Kernel panic – not syncing: attempted to kill init!

I have tried to strat up on one of the disks (all seperated) and did it with 2 and 3. Nothing works. Is there a way to recover data by putting some disks in a windows machine and use some kind of tool?
Some of the data i cant lose!

Mathijs

[byte]

Is there a way to recover data by putting some disks in a windows machine and use some kind of tool?
Some of the data i cant lose

I normally have a burn copy of knoppix as a last resort which you can boot as a Live-cd and recover (if possible) any data.

[Sijhtam]

I will download a copy of the knopix cd and try to get some data back.
Hope it workes
Thanx

[Sijhtam]

Just logged on with knoppix, but cant get to the disks. This is dmesg | tail i get.
dmesg |tail
Buffer I/O error on device hdg, logical block 6
end_request: I/O error, dev hdg, sector 56
Buffer I/O error on device hdg, logical block 7
end_request: I/O error, dev hdg, sector 0
end_request: I/O error, dev hdg, sector 0
EFS: cannot read volume header
EXT3-fs error (device hde1): ext3_check_descriptors: Inode bitmap for group 7 not in group (block 1073799428)!
EXT3-fs: group descriptors corrupted!
EXT3-fs error (device hde1): ext3_check_descriptors: Inode bitmap for group 7 not in group (block 1073799428)!
EXT3-fs: group descriptors corrupted!



I dont get the error. is there some one else?
Mathijs

[Stefano]

I have tried to strat up on one of the disks (all seperated) and did it with 2 and 3. Nothing works. Is there a way to recover data by putting some disks in a windows machine and use some kind of tool?
Some of the data i cant lose!

try to boot your server (with all 3 good hd onboard) with sme's cd and use

Code: [Select]

sme rescue
at boot prompt

HTH

Stefano

[Sijhtam]

try to boot your server (with all 3 good hd onboard) with sme's cd and use

Code: [Select]

sme rescue
at boot prompt

HTH

Stefano

I already try that one. It gives back an error. (kernel panic) and than the server needs to reboot.
When i try sme update. i get an error. It says it is a bug. and i have to report it on bugzilla. but when i try to save, it fails?

Isnt there a way to look at the raw data? I have tryed to put the disk in my windows machine en load te ext2ifs drivers. than i see the bootloader (grub) and some boot img files.

Still nothing works!
Mathijs

[Sijhtam]

I have tried some an other thing:
I only put in the disfunctioning disk and this is de servers message:
ide: faild
upcode was unknown
end request: 1/0 error def hda, sector 160071501
/def/ hda2: read faild after 0 of 512 at 8184967976 input/output error

To me this don't make sense.
Mathijs

[Gaston94]

Hi,
I am not sure this can help, but it might help you understanding where you are and how you can you further.

let me try describing your configuration :
 You built your server with 4 disks.
 The installation went in a Raid5 configuration on 3 disks + 1 hot spare
 You are using a LVM layout.

Accessing data for such configuration, out of the normal operation mode, requires  you can 
 - start the raid array (automatically or manually)
 - activate your LVM layout
only once these steps are completed, you will be able to mount the FS and access the datas (and as far as I know this can't be done on a non Linux system). Any access to your lvm part out of these prerequisites will failed (your last error messages ...)

You should not have get errors because, you add a Hot spare disk, unless you choose "no spare" at installation stage (did you ? )
Some informations looks strange.

mdadm: /dev/md2 assambled from 2 drives - not enough to start the array.

like if you were not using the correct initrd image
Are you sure you change the correct faulting drive ?
I do not see neither good reason for the SME not booting in rescue mode 
I would have try the following :
 - have your good disks in your box (let's say hda and hdb)
 - boot on a live CD supporting LVM2 and raid device (Linux CD rescue for instance)
 - try starting the raid array : mdadm -AR /dev/md5 /dev/hda2 /dev/hdb2
try any combinaison of your disks - I am naming disk1 as the former hda, disk2 as the former hdb, ... - disk1+disk2 / disk1+disk3 / disk1+disk4 / disk2+disk3 / disk2+disk4 / disk3+disk4
then  go on with activating you lvm and mounting your FS :

Code: [Select]

#vgscan
#vgchange -a y main
#mkdir /mnt/tmp
#mount /dev/main/root /mnt/tmpand backup your data
should you can't restart a raid array with your existing disk, I have no further idea

Gaston

[Sijhtam]

Thanx for your sugestions, i try it today. Let you al know the answer.
Mathijs

[Sijhtam]

You should not have get errors because, you add a Hot spare disk, unless you choose "no spare" at installation stage (did you ? )

No i dit not choose "no spare"

Are you sure you change the correct faulting drive ?

Do i need to put a good drive in the server to replace the faulting drive?


After i take out the faulting drive and dit not put in a new one, this is the message given by the server:

md: Kicking non-fresh hde2 from array
md: md5 raidarry is not clean --starting background reconstruction
raid5: not enough operational devices for md5 (3/4 failed)
raid5 conf printout:
--- rd:4 wd:1
disk 2, 0:1 dev:hdh2
raid5: failed to run raid set md5
md: press -> run() failed
mdadm: failed to RUN_ARRAY /dev/md5: input/output error
MDADM not enough divices to start array

What does it say? and What do i have to do to make it work?

Mathijs

[Gaston94]

Hi,
things are quite bad,

md: Kicking non-fresh hde2 from array
md: md5 raidarry is not clean --starting background reconstruction
raid5: not enough operational devices for md5 (3/4 failed)
raid5 conf printout:
--- rd:4 wd:1
disk 2, 0:1 dev:hdh2
raid5: failed to run raid set md5
md: press -> run() failed
mdadm: failed to RUN_ARRAY /dev/md5: input/output error
MDADM not enough devices to start array

this mean that your hdh disk is OK and that the hde drive is the one which is faulty !
and that you originaly have an array of 4 disksn but only one is recognized ok to go on, but the Raid mechanism cannot restart a raid5 on a single disk : you need at least two goode one (but not the spare one ).
Either you make a mistake and move out one of the good disks, either you had more than one disk faulty and there is nothing I know to help you.

Considering you were having a single disk faulty, let's try this last :Put all your original disks in the box and use
          mdadm -AR /dev/md5 /dev/hde2 /dev/hdf2 /dev/hdg2 /dev/hdh2
(yes I know trying to add hde2 will not success, but we can give it a chance ...)

you might need to use the --force option and to star manually the array (mdadm -R /dev/md5)

Yous should see  something like this on the console:
raid5 conf printout:
--- rd:4 wd:3 fd:1
disk 1, o:1, dev:hdf2
disk 2, o:1, dev:hdg2
disk 3, o:1 dev:hdh2
...

I don't think I can help you further 
G.

[Confucius]

Mathijs,

Silly question maybe but I didn't see anything about "Linux raid autodetect" being set to the newly attached drive. Might that be the problem ?

Harro

[Sijhtam]

Mathijs,

Silly question maybe but I didn't see anything about "Linux raid autodetect" being set to the newly attached drive. Might that be the problem ?

Harro

@Harro, when do i have to see the linux raid autodetect? at start up with a new drive?

Considering you were having a single disk faulty, let's try this last :Put all your original disks in the box and use
          mdadm -AR /dev/md5 /dev/hde2 /dev/hdf2 /dev/hdg2 /dev/hdh2
(yes I know trying to add hde2 will not success, but we can give it a chance ...)

you might need to use the --force option and to star manually the array (mdadm -R /dev/md5)

Yous should see  something like this on the console:
raid5 conf printout:
--- rd:4 wd:3 fd:1
disk 1, o:1, dev:hdf2
disk 2, o:1, dev:hdg2
disk 3, o:1 dev:hdh2
...

@ Gaston,

I put in all my disks, when booting from the rescue cd this is the message:

ide: failed opcode was: unknown end_request: 1/0 error, dev/hdf, sector 160071506
hdf: dma_intr: status = 0x51 {DriveReady Seek Complete Error}
hdf: dma_intr: error = 0x40 {uncorrectable error}, LBAsect = 160071507, sector = 160071507

I get this error on multiple sectors 22 times exactly.

Than after

Code: [Select]

mdadm -AR /dev/md5 /dev/hde2 /dev/hdf2 /dev/hdg2 /dev/hdh2

I get the same error 22 times on different sectors and it ends with:
mdadm: no RAID superblock on /dev/hdf2
mdadm: /dev/hdf2 has no superblock - assembley abborted

Hope you someone have an other suggestion, or else i'm gonne build a new server and loosing my data!

Mathijs