Koozali.org: home of the SME Server

Obsolete Releases => SME Server 7.x => Topic started by: SchulzStefan on November 17, 2010, 05:38:49 PM

Title: SME RESCUE - Boot question
Post by: SchulzStefan on November 17, 2010, 05:38:49 PM
I'm running a 7.5.1 installation, up-to-date.

The server crashed in the last days a few times - so I took a closer look at the RAID 1. There are two identical hard drives, running without any problems for the last 2 years. Well, a

[root@sme]# cat /proc/mdstat shows something like this:

md2 : active raid1 sda2[2] sdb2[1]
      1048704 blocks [2/1] [UU]
      [=>...................]  recovery =  6.4% (67712/1048704) finish=1.2min speed=13542K/sec
md1 : active raid1 sda1[0] sdb1[1]
      255936 blocks [2/2] [UU]

I did:

[root@ ~]# fdisk -l /dev/sdb; fdisk -lu /dev/sdb

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104384+  fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2              13      121601   976655647   fd  Linux raid autodetect

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1      208769      104384+  fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2          208770  1953520063   976655647   fd  Linux raid autodetect

and:

smartctl -i /dev/sda
smartctl -i /dev/sdb

and:

smartctl -t short /dev/sda
smartctl -t short /dev/sdb

both showed no errors at all.

and:

[root@ ~]# mdadm --manage /dev/md2 --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md2
[root@ ~]# mdadm --manage /dev/md2 --remove /dev/sdb2
mdadm: hot removed /dev/sdb2
[root@ ~]# mdadm --manage /dev/md1 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md1
[root@ ~]# mdadm --manage /dev/md1 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1

Booting in the server causes in a crash while a file check is forced. The machine crashes during the check. All fans are o.k., the BIOS shows nothing special for any temperature.

Now I wanted to boot from the sme server CD (is still a 7.4 ver) in rescue mode. The machine is booting from the CD, but then something goes wrong. It seems, it looses the CD-Drive. After choosing the CD-ROM as installation medium, I have to choose a driver? Tried a few ones, but no luck. The same thing happens using the TRK-CD. The machine is booting and looses the drive. Same with any other live CD. I checked the BIOS Settings and it seems to me, that erverything is normal, means conservative for a server. Now I'm stuck.

Thank's for any help in advance.
stefan
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 17, 2010, 06:35:42 PM
Meanwhile I did some more tests with the hardware. I changed the hard-disk(s), memory, LAN-adapter, graphic-adapter, cd-rom and now I think, it's the mainboard. The machine crashes, no matter what I change. Or it is the processor. But how to eliminate this? As I mentioned, the fans are o.k.

Question: if I buy a new (probably another type) mainbord, is there a chance to get the two hard-disks back on track?

Thank's in advance for any reply.
stefan
Title: Re: SME RESCUE - Boot question
Post by: idp_qbn on November 17, 2010, 09:38:52 PM
Stefan,
I have twice in the past removed HDDs from an older server and successfully mounted them in a newer server. Both times, the system started OK but I needed to logon at the new server, and run"console", then choose option 2, "Configure this server". This allowed the server to make adjustments for the new N/W adapters, specifically the on-board ones. Then all was well.

I think this depended on the newer servers not being too cutting-edge with lots of new hardware to contend with.

I believe that if necessary, I could have run the install disk in Rescue mode to pick up the hardware changes - in fact this is probably the best way of handling the situation.

I take it there is user data at stake here as well as configuration settings. If you have a backup (done from SMEserver "server-manager" panel, you could do a clean installation on a new server and restore from your backup. This should pick up user data and system settings BUT NOT any contribs you have installed on the old system.

From you first post, it looked like the system was in the process of recovering from the RAID failure when you ran mdadm. That may complicate things. Perhaps you could try mounting ONE disk in a new server and see if it boots. If successful, you can add the second disk - look in the wiki for advice on how to add a second disk: you may have to zero it out first (wipe it, in other words).

Good luck
Ian
Title: Re: SME RESCUE - Boot question
Post by: CharlieBrady on November 17, 2010, 10:04:42 PM
I believe that if necessary, I could have run the install disk in Rescue mode to pick up the hardware changes - in fact this is probably the best way of handling the situation.

No, please do not spread misinformation. Running the install disk in Rescue mode does not change any system configuration items. The only way to adjust to new network hardware changes is to run through the Configuration option in the console.
Title: Re: SME RESCUE - Boot question
Post by: CharlieBrady on November 17, 2010, 10:06:41 PM
Question: if I buy a new (probably another type) mainbord, is there a chance to get the two hard-disks back on track?

Yes, there is a chance. I would guess there is a good chance.
Title: Re: SME RESCUE - Boot question
Post by: idp_qbn on November 17, 2010, 10:34:35 PM
Yes, Charlie, Sorry.
I have never used the Rescue Disk mode - as I said, just transplanting the disk to a new server and running "console" to reconfigure n/w adapters worked just fine for me.

Ian
Title: Re: SME RESCUE - Boot question
Post by: janet on November 18, 2010, 02:17:12 AM
SchulzStefan

Quote
....I changed the hard-disk(s), memory, LAN-adapter, graphic-adapter, cd-rom and now I think, it's the mainboard. The machine crashes, no matter what I change. Or it is the processor. But how to eliminate this? As I mentioned, the fans are o.k.

You did not mention the power supply. I have seen systems fail temporarily & behave quiet erratically & in a seemingly unrepeatable or identifiable fashion when the CD/DVD drive spins up, where the power supply is faulty or weak. The extra drain on the power supply pulls the voltage rails down and the whole system becomes erratic under the low voltage situation.

Your myriad of problems could all be answered by a faulty (weak output) power supply.
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 08:56:11 AM
Ian, Charlie and Mary,

thank you for your replies. At this place I want to mention, that the network was only for 15 minutes down. A BIG THANK TO MICHAEL WEINBERGER and his AFFA.

@ ian: I bought a new board - the machine is back on track.
@ charlie: thank you for your hint. In this case it seems, that I do not have to re-configure the machine.
@ mary: I checked the power supply also, it's o.k.

BUT, there are still a few problems with the disks. Well, I'm able to boot in the server, all data seem to be o.k. But a fdisk -lu /dev/sda; fdisk -lu /dev/sdb shows up, that on BOTH disks is an incorrecty partitioned first partition. "Partition 1 does not end on cylinder boundary". I think it may cause from the blackouts during several resyncs. Well, seems strange to me anyway, both disks were brand new (same vendor, type, size and model), when installed.

O.k., the idea is to get it fixed on one disk, and resync the other one after clearing. Is there a way to do it? Any idea?

Thank's for any reply.
stefan
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 10:21:26 AM
I read bug 2542 - that's allright so far. Could have done a search earlier, sorry.

Anyway, smartd reports on sda "1 Offline uncorrectable sectors". Interesting, because the BIOS reports no errors at all. What counts: BIOS or smartctl? No errors on sdb, either BIOS nor smartctl. This one I wiped meanwhile.

Does it make sense build the array again under those circumstances?
stefan
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 11:03:35 AM
Stuck - syncing aborts and restarts. No luck. Does it make sense to try a backup via USB?

stefan
Title: Re: SME RESCUE - Boot question
Post by: janet on November 19, 2010, 12:34:59 PM
SchulzStefan

Quote
No errors on sdb, either BIOS nor smartctl. This one I wiped meanwhile.

Why did you do that, you said your machine was booting up OK. That partition/cylinder boundary error is not a problem AFAIK.

How did you wipe the drive, ie what commands or method, if you did not do it correctly that could be the reason for your syncing abort ???

Quote
Does it make sense to try a backup via USB?

If you wish, it's your machine & how you resolve this issue is your choice, but jumping quickly from one "fix" to another does not allow time to diagnose. If all else has seemingly failed then a restore is your only option.

Quote
Does it make sense build the array again under those circumstances?

Probably depends on answers given to above questions.
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 01:45:39 PM
@mary:
Quote
Why did you do that, you said your machine was booting up OK.
The machine is still booting o.k. on one disk. The machine did not boot on the other. Therefore I cleaned the other one.

Unluckily on the boot disk is the "1 Offline uncorrectable sectors". And trying to sync ends up in an endless loop.

Quote
That partition/cylinder boundary error is not a problem AFAIK.
Yep, I noticed that already. Nevertheless is the problem that the disks have been out of sync. And the disk which boots, has an error.

Quote
but jumping quickly from one "fix" to another does not allow time to diagnose
Sorry, you're right. I'm too fast in trying and eliminating the things. After all, it's not possible to do it online...

O.k., I decided to make a backup with dar. I think, I'll see in the logs, when it breaks. Maybe I can eliminate the files and then exclude. I'll give this a try. Right now it's more to figure out what one can do, if there's no backup...

The target is to install on the wiped disk an new smesever, and then move the backup on to it. We'll see, how it works.
stefan
Title: Re: SME RESCUE - Boot question
Post by: janet on November 19, 2010, 02:00:12 PM
SchulzStefan

Quote
The machine is still booting o.k. on one disk....
And the disk which boots, has an error.

Do you mean it boots OK and does not give any errors, but when you run a smartctl test it gives the error
"1 Offline uncorrectable sectors".

You should test both disks thoroughly ie full test, with a disk manufacturers utility. You can download the UBCD and run many manufacturers disk diagnostic tests by booting to the CD. I'd also run the smartctl "long" test on both disks.

You did not answer how you prepared (wiped) the sdb disk before trying to resync.
You must delete the MBR
You can use the dd command or use delpart.exe or similar downloadable partition deleting tools.


Quote
The target is to install on the wiped disk an new smesever, and then move the backup on to it.

I would consider that disk still suspect until you have done thorough testing on it and the other disk too.
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 02:14:01 PM
@mary:

snip -

Nov 19 13:15:45 real-saturn smartd[4269]: smartd version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Nov 19 13:15:45 real-saturn smartd[4269]: Home page is http://smartmontools.sourceforge.net/ 
Nov 19 13:15:45 real-saturn smartd[4269]: Opened configuration file /etc/smartd.conf
Nov 19 13:15:45 real-saturn smartd[4269]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Nov 19 13:15:45 real-saturn smartd[4269]: Problem creating device name scan list
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sda, opened
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sda, not found in smartd database.
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sda, is SMART capable. Adding to "monitor" list.
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sdb, opened
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sdb, not found in smartd database.
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list.
Nov 19 13:15:45 real-saturn smartd[4269]: Monitoring 2 ATA and 0 SCSI devices
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sda, 1 Currently unreadable (pending) sectors
Nov 19 13:15:45 real-saturn smartd[4269]: Sending warning via mail to admin ...
Nov 19 13:15:45 real-saturn smartd[4269]: Warning via mail to admin: successful
Nov 19 13:15:45 real-saturn smartd[4269]: Device: /dev/sda, 1 Offline uncorrectable sectors
Nov 19 13:15:45 real-saturn smartd[4269]: Sending warning via mail to admin ...
Nov 19 13:15:45 real-saturn smartd[4269]: Warning via mail to admin: successful

- snip
Never got any email to the admin account from smartd before.

- snip

Nov 19 13:45:46 real-saturn smartd[4320]: Device: /dev/sda, 1 Currently unreadable (pending) sectors
Nov 19 13:45:46 real-saturn smartd[4320]: Device: /dev/sda, 1 Offline uncorrectable sectors
Nov 19 13:45:46 real-saturn smartd[4320]: Device: /dev/sda, SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 62 to 63
Nov 19 13:45:46 real-saturn smartd[4320]: Device: /dev/sdb, SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 109 to 108
Nov 19 13:45:46 real-saturn smartd[4320]: Device: /dev/sdb, SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 62 to 61

- snip

Quote
You should test both disks thoroughly ie full test, with a disk manufacturers utility. You can download the UBCD and run many manufacturers disk diagnostic tests by booting to the CD. I'd also run the smartctl "long" test on both disks.

I'll do that.

Quote
You did not answer how you prepared (wiped) the sdb disk before trying to resync.
You must delete the MBR
You can use the dd command or use delpart.exe or similar downloadable partition deleting tools.

Pardon me, I did not delete the MBR. I formatted the disk with mkfs.ext3. IMHO I didn't believe, that the MBR could be damaged. I just added the clean disk via install menu. Before I re-install, I'll test the disks and will also delete the MBR.

stefan
Title: Re: SME RESCUE - Boot question
Post by: CharlieBrady on November 19, 2010, 03:09:27 PM
Anyway, smartd reports on sda "1 Offline uncorrectable sectors". Interesting, because the BIOS reports no errors at all. What counts: BIOS or smartctl?

I've never heard of BIOS being a reliable source of information about disk errors.

Title: Re: SME RESCUE - Boot question
Post by: CharlieBrady on November 19, 2010, 03:12:06 PM
You should test both disks thoroughly ie full test, with a disk manufacturers utility.

Some manufacturer's test utilities do destructive tests - overwriting data. Use with care.

Checking smart status with smartctl is usually adequate.
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 07:07:51 PM
@ charlie:

The motherboard is an asus p5qd turbo. It reports the health status of the disks. Obviously not correct.

The disks are seagate barracudas 7200.10, 500 GB. I downloaded the seagate disk-tool and run the long test. It reported an error with the option to correct it. I choosed this option and booted in the server. The machine booted normal and is up. Quite interesting.

Well now I' ll blank the other one (I'll delete the MBR) again, and put it back in the machine. Let's see, what happens.
stefan
Title: Re: SME RESCUE - Boot question
Post by: SchulzStefan on November 19, 2010, 11:01:29 PM
Syncing went through. And here we are:

[root@real-saturn ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdb2[0] sda2[1]
      488279488 blocks [2/2] [UU]

md1 : active raid1 sdb1[0] sda1[1]
      104320 blocks [2/2] [UU]

unused devices: <none>

and:

This email was generated by the smartd daemon running on:

   host name: real-saturn
  DNS domain: real.ivb.org
  NIS domain: (none)

The following warning/error was logged by the smartd daemon:

Device: /dev/sda, ATA error count increased from 3814 to 3815

For details see host's SYSLOG (default: /var/log/messages).

You can also use the smartctl utility for further investigation.
Another email message will be sent in 1 days if the problem persists

and:

[root@real-saturn ~]# fdisk -lu /dev/sda; fdisk -lu /dev/sdb

Platte /dev/sda: 500.1 GByte, 500107862016 Byte
255 Köpfe, 63 Sektoren/Spuren, 60801 Zylinder, total 976773168 sectors
Einheiten = Sektoren von 1 * 512 = 512 Bytes

    Gerät Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1      208769      104384+  fd  Linux raid autodetect
Partition 1 endet nicht an einer Zylindergrenze.
/dev/sda2          208770   976768063   488279647   fd  Linux raid autodetect

Platte /dev/sdb: 500.1 GByte, 500107862016 Byte
255 Köpfe, 63 Sektoren/Spuren, 60801 Zylinder, total 976773168 sectors
Einheiten = Sektoren von 1 * 512 = 512 Bytes

    Gerät Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1      208769      104384+  fd  Linux raid autodetect
Partition 1 endet nicht an einer Zylindergrenze.
/dev/sdb2          208770   976768063   488279647   fd  Linux raid autodetect

and:

[root@real-saturn ~]# smartctl -a /dev/sda
smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST3500630AS
Serial Number:    6QG3A7B0
Firmware Version: 3.AAK
User Capacity:    500,107,862,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Fri Nov 19 22:46:50 2010 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (  24) The self-test routine was aborted by
                                        the host.
Total time to complete Offline
data collection:                 ( 430) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 163) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   115   100   006    Pre-fail  Always       -       90420120
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       86
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   082   060   030    Pre-fail  Always       -       185476578
  9 Power_On_Hours          0x0032   078   078   000    Old_age   Always       -       19876
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       86
187 Unknown_Attribute       0x0032   001   001   000    Old_age   Always       -       3488
189 Unknown_Attribute       0x003a   100   100   000    Old_age   Always       -       0
190 Unknown_Attribute       0x0022   062   055   045    Old_age   Always       -       707133478
194 Temperature_Celsius     0x0022   038   045   000    Old_age   Always       -       38 (Lifetime Min/Max 0/24)
195 Hardware_ECC_Recovered  0x001a   068   057   000    Old_age   Always       -       107020588
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
202 TA_Increase_Count       0x0032   100   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
ATA Error Count: 3815 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 3815 occurred at disk power-on lifetime: 19875 hours (828 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 af 53 bb 0c e0  Error: ICRC, ABRT 175 sectors at LBA = 0x000cbb53 = 834387

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 02 b8 0c e0 00      01:31:58.989  READ DMA EXT
  25 00 00 02 b4 0c e0 00      01:31:58.980  READ DMA EXT
  25 00 00 02 b0 0c e0 00      01:31:58.972  READ DMA EXT
  25 00 80 82 ac 0c e0 00      01:31:58.963  READ DMA EXT
  25 00 00 82 a8 0c e0 00      01:31:59.076  READ DMA EXT

Error 3814 occurred at disk power-on lifetime: 19872 hours (828 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 01 98 b9 94 e0  Error: UNC at LBA = 0x0094b998 = 9746840

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  42 00 00 00 b8 94 e0 00      00:14:33.231  READ VERIFY SECTOR(S) EXT
  42 00 00 00 b0 94 e0 00      00:14:33.220  READ VERIFY SECTOR(S) EXT
  42 00 00 00 a8 94 e0 00      00:14:33.203  READ VERIFY SECTOR(S) EXT
  42 00 00 00 a0 94 e0 00      00:14:33.191  READ VERIFY SECTOR(S) EXT
  42 00 00 00 98 94 e0 00      00:14:33.174  READ VERIFY SECTOR(S) EXT

Error 3813 occurred at disk power-on lifetime: 19867 hours (827 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 98 b9 94 e7  Error: UNC at LBA = 0x0794b998 = 127187352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 70 92 b9 94 e7 00      01:30:05.464  READ DMA
  ec 03 46 00 00 00 a0 00      01:30:05.461  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      01:30:05.461  SET FEATURES [Set transfer mode]
  ec 00 00 98 b9 94 a0 00      01:30:05.459  IDENTIFY DEVICE
  c8 00 78 8a b9 94 e7 00      01:30:03.419  READ DMA

Error 3812 occurred at disk power-on lifetime: 19867 hours (827 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 98 b9 94 e7  Error: UNC at LBA = 0x0794b998 = 127187352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 78 8a b9 94 e7 00      01:30:05.464  READ DMA
  ec 03 46 00 00 00 a0 00      01:30:05.461  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      01:30:05.461  SET FEATURES [Set transfer mode]
  ec 00 00 98 b9 94 a0 00      01:30:05.459  IDENTIFY DEVICE
  c8 00 80 82 b9 94 e7 00      01:30:03.419  READ DMA

Error 3811 occurred at disk power-on lifetime: 19867 hours (827 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 98 b9 94 e7  Error: UNC at LBA = 0x0794b998 = 127187352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 82 b9 94 e7 00      01:29:59.327  READ DMA
  ec 03 46 00 00 00 a0 00      01:29:57.288  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 00      01:29:57.285  SET FEATURES [Set transfer mode]
  ec 00 00 98 b9 94 a0 00      01:29:57.285  IDENTIFY DEVICE
  c8 00 88 7a b9 94 e7 00      01:30:03.419  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Aborted by host               80%     19873         -
# 2  Short offline       Completed without error       00%     19873         -
# 3  Short offline       Aborted by host               90%     19873         -
# 4  Short offline       Completed: read failure       90%     19873         127187352
# 5  Short offline       Completed: read failure       90%     19871         127187352
# 6  Short offline       Interrupted (host reset)      90%     19846         -
# 7  Short offline       Aborted by host               80%     19873         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

and:

[root@real-saturn ~]# smartctl -a /dev/sdb
smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST3500630AS
Serial Number:    9QG8S58F
Firmware Version: 3.AAK
User Capacity:    500,107,862,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Fri Nov 19 22:48:56 2010 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                 ( 430) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 163) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   113   094   006    Pre-fail  Always       -       57276716
  3 Spin_Up_Time            0x0003   093   093   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       90
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       14
  7 Seek_Error_Rate         0x000f   082   060   030    Pre-fail  Always       -       181467908
  9 Power_On_Hours          0x0032   078   078   000    Old_age   Always       -       19901
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       90
187 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
189 Unknown_Attribute       0x003a   100   100   000    Old_age   Always       -       0
190 Unknown_Attribute       0x0022   066   056   045    Old_age   Always       -       656539682
194 Temperature_Celsius     0x0022   034   044   000    Old_age   Always       -       34 (Lifetime Min/Max 0/22)
195 Hardware_ECC_Recovered  0x001a   063   059   000    Old_age   Always       -       18356524
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
202 TA_Increase_Count       0x0032   100   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19898         -
# 2  Short offline       Completed without error       00%     19897         -
# 3  Short offline       Interrupted (host reset)      90%     19871         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Am I right to replace the sda-disk? Can I just plug it off the machine?

stefan
Title: Re: SME RESCUE - Boot question
Post by: janet on November 20, 2010, 12:48:56 AM
SchulzStefan

It would seem that sda has been problematic.
smartctl was monitoring it, and errors were detected. It appears those errors may have been severe enough to stop your machine working correctly.
Those errors have been "fixed" now, so is the drive OK or safe to use ?

This could be a sign the whole drive is starting to fail or it might just be a one off issue.

You could continue to use the drive and monitor it for ongoing errors. Otherwise if you consider the server is doing mission critical tasks then it may be wise to totally replace the drive.

I think you should have a good read about smartctl & what it does, use google.
Title: Re: SME RESCUE - Boot question
Post by: CharlieBrady on November 20, 2010, 01:01:57 AM
Am I right to replace the sda-disk?

Yes.

Quote
Can I just plug it off the machine?

Shut down the OS and power down the machine first.