Topic: [SOLVED] Failed single disk; any way to mirror to 2nd disk?

[Michail Pappas]

Hello all,

on a 7.5.1 system with a single 320Gb drive and a non-existent system admin, the disk produced errors. smartctl -t short shows that the drive fails even the small test.

I installed a second 640Gb disk, in the hope that a sync can be performed. md1 (boot) syncs ok, but the rest filesystem does not and restarts obviously due to read error (hdc is the old/failed 320Gb disk):

Code: [Select]

Jan 28 05:04:18 sme-server kernel: hdc: read_intr: status=0x51 { DriveReady SeekComplete Error }
Jan 28 05:04:18 sme-server kernel: hdc: read_intr: error=0x40 { UncorrectableError }, LBAsect=121288638, high=7, low=3848126, sector=121288637
Jan 28 05:04:18 sme-server kernel: ide: failed opcode was: unknown
Jan 28 05:04:18 sme-server kernel: end_request: I/O error, dev hdc, sector 121288637
Jan 28 05:04:18 sme-server kernel: raid1: hdc: unrecoverable I/O read error for block 121079680
Jan 28 05:04:18 sme-server kernel: md: md2: sync done.
Jan 28 05:04:18 sme-server kernel: md: syncing RAID array md2
Jan 28 05:04:18 sme-server kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Jan 28 05:04:18 sme-server kernel: md: using maximum available idle IO bandwith (but not more than 200000 KB/sec) for reconstruction.
Jan 28 05:04:18 sme-server kernel: md: using 128k window, over a total of 312464128 blocks.

# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 hdc2[0] hda2[2]
      312464128 blocks [2/1] [U_]
      [==>..................]  recovery = 14.9% (46712320/312464128) finish=2440.7min speed=1814K/sec
md1 : active raid1 hdc1[0] hda1[1]
      104320 blocks [2/2] [UU]

unused devices: <none>

Any way I can somehow sync the two drives, in order to completely remove the old one? The only thing that comes to mind is finding the sysadmin and shooting him right in the eye. But it definitely would not help here :/

[byte]

I installed a second 640Gb disk, in the hope that a sync can be performed. md1 (boot) syncs ok, but the rest filesystem does not and restarts obviously due to read error (hdc is the old/failed 320Gb disk):

I do hope you have a good backup?  He can't of been a very good sysadmin as why would he have used the secondary master rather than primary master (hda) and cost of hard disk these days are cheap so why he didn't setup with 2 drives running RAID 1

Code: [Select]

Jan 28 05:04:18 sme-server kernel: hdc: read_intr: status=0x51 { DriveReady SeekComplete Error }
Jan 28 05:04:18 sme-server kernel: hdc: read_intr: error=0x40 { UncorrectableError }, LBAsect=121288638, high=7, low=3848126, sector=121288637
Jan 28 05:04:18 sme-server kernel: ide: failed opcode was: unknown
Jan 28 05:04:18 sme-server kernel: end_request: I/O error, dev hdc, sector 121288637
Jan 28 05:04:18 sme-server kernel: raid1: hdc: unrecoverable I/O read error for block 121079680
Jan 28 05:04:18 sme-server kernel: md: md2: sync done.
Jan 28 05:04:18 sme-server kernel: md: syncing RAID array md2
Jan 28 05:04:18 sme-server kernel: md: minimum _guaranteed_ reconstruction speed: 1000 KB/sec/disc.
Jan 28 05:04:18 sme-server kernel: md: using maximum available idle IO bandwith (but not more than 200000 KB/sec) for reconstruction.
Jan 28 05:04:18 sme-server kernel: md: using 128k window, over a total of 312464128 blocks.

# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 hdc2[0] hda2[2]
      312464128 blocks [2/1] [U_]
      [==>..................]  recovery = 14.9% (46712320/312464128) finish=2440.7min speed=1814K/sec
md1 : active raid1 hdc1[0] hda1[1]
      104320 blocks [2/2] [UU]

unused devices: <none>


Unfortunately, when you see "DriveReady SeekComplete Error" your disk is pretty much useless.

Any way I can somehow sync the two drives, in order to completely remove the old one?

Your going to need your backup here as the old drive will keep looping and never pass that stage.

The only thing that comes to mind is finding the sysadmin and shooting him right in the eye. But it definitely would not help here :/

I'd sack him for failing do to his job properly as any company data is valuable.

Sorry I can't bring better news.

[Michail Pappas]

I do hope you have a good backup?  He can't of been a very good sysadmin as why would he have used the secondary master rather than primary master (hda) and cost of hard disk these days are cheap so why he didn't setup with 2 drives running RAID 1

Actually, the failed disk was primary master and I moved it to secondary master, leaving the new 640Gb as primary master. I also can't understand why in the love of God, one would install a system like this, with no UPS protection, no data backup plan and not disk redundancy...

Unfortunately, when you see "DriveReady SeekComplete Error" your disk is pretty much useless.

Had the hope that I can do a disk clone, not stopping on drive errors at all. If this was an NTFS basic (not dynamic that is) disk, that would be the way I'd go; Ghost the disk, ignoring any errors... Would ghosting help on lvm volumes? That is, could I do something to clone the 320 to the 640, ignoring error in the process?

Your going to need your backup here as the old drive will keep looping and never pass that stage.

No backup here either, AFAIK!

[byte]

Would ghosting help on lvm volumes? That is, could I do something to clone the 320 to the 640, ignoring error in the process?

You could use dd to ignore the bad blocks or you could try this tool

http://www.device-image.de/

I've downloaded this but haven't had a bad disk to try it on yet so if you do try I'd be interested in your results

[Stefano]

You could use dd to ignore the bad blocks or you could try this tool

http://www.device-image.de/

I've downloaded this but haven't had a bad disk to try it on yet so if you do try I'd be interested in your results

I would add dd_rescue

@OP: good luck

[Michail Pappas]

Thank you for your help. Being a newbie, had to find out how to disable md2, so I removed the good partition hda2 from the raid array and have started a copy from the failed partition hdc2 to the good one:

Code: [Select]

#  dd conv=noerror if=/dev/hdc2 of=/dev/hda2 bs=32256
Hope this command does the trick. I can see that it has encountered an error, but the dd process just went on. Keeping my fingers crossed here.

If it finishes, I'd like to remove the faulty disk altogether and see if it starts ok with only the good one present. Do you think this has any chances? md1 was mirrored alright.

[byte]

If it finishes, I'd like to remove the faulty disk altogether and see if it starts ok with only the good one present. Do you think this has any chances?

It really depends what data was on the bad blocks, if it was any system files you might find it boots up with a kernel panic.  Good Luck!

[Michail Pappas]

Still struggling to be honest. Booted up with SystemRescueCD to try and clone using dd_rescue...

<OT>One fast question: this is a system that takes days to sync on SME; on SysRescueCD the sync operation is 50x in speed (from 2Mbytes/sec on SME, 104Mbytes on SystemRescueCD). This is a system that does not support AHCI. Disks in SME look like /dev/hdX, whereas on SystemRescueCD look like sdX devices. Any way I can have the same speed on SME?</OT>

[Michail Pappas]

<OT>One fast question: this is a system that takes days to sync on SME; on SysRescueCD the sync operation is 50x in speed (from 2Mbytes/sec on SME, 104Mbytes on SystemRescueCD). This is a system that does not support AHCI. Disks in SME look like /dev/hdX, whereas on SystemRescueCD look like sdX devices. Any way I can have the same speed on SME?</OT>

Nvm, found this excellent post here: http://forums.contribs.org/index.php/topic,44743.msg216478.html#msg216478

[Michail Pappas]

A product is as good as its supporters. Must say, my deepest respects to the community here!

Let me elaborate a bit: dd_rescue on a partition level did not work; received panics. But on a drive level it worked like a charm!!!!

I've removed the faulty disk and added a new second in order to create a mirror. Will increase the size. Tomorrow I'll create a daily back plan and I've asked the owner to buy a solid ups...

Hats off to your community and a zillion thanks!

[byte]

Excellent News Glad we could help.

Take a look here before you buy a UPS to make sure its support

http://www.networkupstools.org/stable-hcl.html

[Michail Pappas]

Thanks again for the info! I know "nut", been using it on my FreeBSD server alongside some MGE Pulsar Ellipse models. Awesome!

Once more, thanks for an awesome forum, really friendly place you've created here! Keep it up!!!!