Topic: minimal BASH-like line editing is supported POST USB HDD backup & Reboot

[mileshc]

Hello all, this is my first post on this version of the forum.  May I say what a fantastic job all have done over the years, more than a decade now?

Apologies 1st if appearing dense.  I have some years of IT experience but mostly on the hardware build, install & cabling side, programming makes my head very fuzzy. Hammers & spanners.

To task then. System based in the UK.

SME 7.4 patched, on factory outlet (scratched) dell optiplex N series Intel dual core 2.3GHz, twin SATA 160Gb mirrored with SME, 2Gb RAM, twin Nic's one onboard. Set server only.  Running with occasional re-boots & long power outages for about a year.  Used currently to remove spam & scan for viri on incoming email from POP3 server.  An SBS2003 server then picks up from the SME system. Long term would like rid of SBS.

I connected a Seagate Expansion 1Tb USB 2.0http://direct.tesco.com/q/R.207-2219.aspx to a rear port, logged in as admin & started a backup. this was on the 23rd Dec 2009.

Today 29th Dec 2009 it was noticed that no email was being collected (don't ask why it's taken so long to notice).

I got someone (father) to check the systems, the SME server had completed it's backup & requested the removal of the USB drive, which he did.

What exactly happened next is a little fuzzy. The return key was pressed to inform the console that the drive had been removed. (should this action have stopped the server working & collecting email? I assume certain services/daemons are paused or stopped?)

A re-boot was instigated to refresh & get back (I thought) to a working system. As a "test internet access" failed.

Which has now resulted in the (for me at least, especially being 200 miles away) alarming lack of a correct re-start & the following:-

gnu grub version 0.95
[minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions.
anywhere else TAB lists the possible completions of a device / file name]
grub>

I am at a complete loss as to what to do, especially being so distant. If I had the system here (at home) I'd likely try a fresh install & use the backup to restore it.  I have googled & the issue is not unique, but I do not know how to proceed with regard to the SME/centos system. Also my father is not fully IT literate.

Any pointers would be greatly appreciated.  And again thank you very much.

Miles HC

[paradigm]

Hey Miles , this looks weird :

"I got someone (father) to check the systems, the SME server had completed it's backup & requested the removal of the USB drive, which he did."

How exactly are you doing a backup , using Dar ? it looks like you are using the backup in the console which is not vary usable..

If you do not have internet access i don't think you can do anything remotely...

[mileshc]

Hello,

the backup process was entirely done at the console, NOT via the web admin interface.  I beleive this is not possible other than via directly on the keyboard (ssh, putty & the like excepted of course)

No remote access of course, as it's not booted up. In fact turned off just now. No point in wasting energy & generating CO2 etc.

Cheers.

Miles HC

[paradigm]

Well i believe you have 3 options :

1.a fresh install

2.a repair install

3. use the rescue console - i recommend you wait for some one with more experience than me for this option.

If using one the two options please remember to unplug your usb disk during install so it doesn't destroy your backup

[philbrearley]

Miles,

Before taking any further action please confirm that all external drives have been removed and then reboot.
If you have an external bootable drive attached the boot will stall (depending on BIOS settings).

Phil Brearley

[mileshc]

Happy New Year!

Thank you for the replies!

I will try to confirm a definite removal of the Ext. USB drive next week.  I was informed that it was literally in hand & removed.  But being at distance I cannot be certain 100% until next week.

Take it easy, take care.  Have a modicum of fun.

Miles HC

[mileshc]

Hello, I have finally got hold of the system.

I've checked & removed-reseated all sata leads, both ends & power, checked bios which sees both drives. No ext. drives, no cd-rom etc.

It boots as stated into grub minimal, hmm.. no where might I go please.

Cheers.

Miles HC

[janet]

mileshc

A starting point will be to boot up to the install CD in rescue mode
http://wiki.contribs.org/Booting
Get to a command prompt,
then you can try to fix the boot problem which may be to do with
/boot/grub/grub.conf

Follow up here when you get to that stage or even search here on the topic
Perhaps post the contents of that file to see if it looks right.

[mileshc]

Hello & thank you Mary,  apologies for not getting back sooner, life has got in the way somewhat.

I've booted via sme 7.4 cd-rom into the sme rescue & chroot /mnt/sysimage I can use mc & see the files in the ibays are there & ok?

I've navigated to /boot/grub/ & viewed grub.conf it's a zero byte file!

My home system on PIII HW is 1183 bytes.

I have spent an hour or so trying to find posts relating to rebuilding the grub.conf but cannot really see anything I feel confident is a solution. i.e grub-install /dev/sd?? ?

I have run sfdisk -l  (L) & this reveals both disks etc.

/dev/sda1  fd  104391  linux raid
/dev/sda2  fd  156143761+ linux raid
/dev/sdb1  (As above)
/dev/sdb2  (As above)

/dev/md2 39035920 cyl  2hds  4sct
/dev/md1 2608 cyl   "  "

I could manually type & copy grub.conf from my working system I guess? as I am not sure how to copy the text onto a usb drive & then mount & access it on the broken system.

I am guessing there is a more sensible or robust recovery procedure for this event, that can aid others & well as me?

I will widen my search off contribs & gather more info. in parallel.

Thank you very much,

Miles HC

[mileshc]

A further note.  One of the posts I was looking at is here.

http://forums.contribs.org/index.php?topic=41745.0

Cheers.

Miles HC

[janet]

mileshc

I've not had to do this, so unsure if this is the most appropriate approach:

http://bugs.contribs.org/show_bug.cgi?id=589#c17

For more info
man grub

Try running:
grub
run the commands in the link above
exit

[mileshc]

Mary, thank you for that.

I have now tried this a few ways, but it does not get back to a bootable system.  Done sofar.

Ran grub from cd-rom boot & F5 sme rescue.

grub> root (hd0,0)
        Filesystem type is ext2fs partition type 0xfd

grub> setup (hd0,0)

checking if "/boot/grub/stage1" exists ... no
checking if "/grub/stage1" exists ... yes
checking if "/grub/stage2" exists ... yes
checking if "/grub/e2fs_stage1_5(hd0)" ... 16 sectors are embedded. succeeded
Running "install /grub/stage1 (hd0) (hd0) 1=16p (hd0,0)/grub/stage2 grub/grub.conf" ... succeeded
Done.

I rebooted still lands at grub! so ran same again with same results & rebooted still lands at grub.

Tried variant.

grub>root (hd0,0)
       (results as above)
grub>setup (hd0)
       (results as above)

Rebooted, status quo. hmm....

I will do some more grubbing about & try later tomorrow.  System to noisy & near bedtime baby, ho hum.

Cheers again for your guidance.

Miles HC

[janet]

mileshc

Do you as yet have a seemingly correct
/boot/grub/grub.conf    ?
I assume not.

You could do a upgrade install from CD  and see if that rebuilds grub.conf
ie normal boot from sme7.4 CD and select the upgrade option rather than new install option.
I'm not sure if this will replace grub.conf. Don't know that much about grub
I see Charlie advises this approach here:
http://forums.contribs.org/index.php/topic,39104.msg178434.html#msg178434

If that doesn't work, then you could manually restore it from your USB  backup rather than copying from a different system (so the boot parameters will be correct).
Let us know if a CD upgrade fixes the problem.
If it does not, then we can go through the selective restore process.

 

[mileshc]

Hello Mary,  I had not spotted that thread.

I tried an upgrade, this was NOT successful unfortunately.  CD-ROM was checked & ok. it did roughly the following.

Error occurred transferring the install image to your hard disk. you are probably out of disk space.

The system has twin 160Gb sata which are in no way full. Next.

The file systems of the Linux installation you have chosen to upgrade has already been mounted. You cannot go back past this point.

A dependency check occurs & then.

An unhandled exception has occurred.
  Traceback
File "/usr/bin/anaconda", line 1190, in ? intf.run(id, dispatch, configFileData)
File "/tmp/updates/text.py", line 171, in gotoNext self.moveStep()
File "/tmp/updates/dispatch.py", line 240, in moveStep rc - apply(func self.bindArgs(args))
File "/tmp/updates/packages.py", line 792, in doPreInstall f = open(instPath + "/etc/mtab", "w+")
IOError: [Errno 2] No such file or directory: '/mnt/sysimage/etc/mtab'

It then exits & states it is safe to reboot.

I will investigate the thread on the 0 byte grub.conf.  I do not have a usb based backup of this system, the 1Tb disk was checked & had no files on it!  I believe this is due to it being incorrectly formatted.  The keyboard based admin, backup to usb disk, was used & made no complaint when run & I understand when finished.

I know you mentioned this but, might a quick & dirty copy of a "good" grub.conf work I wonder.  If I use a usb pendrive I can copy the text over. BUT I do not know how to go about mounting it etc.

Cheers.

Miles HC

[janet]

mileshc

Work with errors & responses as they give clues as to what's up.

Error occurred transferring the install image to your hard disk. you are probably out of disk space.

To check, boot up in rescue mode and do
df -h

I do not have a usb based backup of this system, the 1Tb disk was checked & had no files on it!

If the USB drive did not mount properly, then your backup may have been made to the local drives, and that may explain why your disk is full. I'm sure that scenario has been reported here before. For starters look in /media

You can just edit grub.conf and enter the correct parameters. Refer to your good system or just do an advanced search of forums on grub.conf to find an example (click the magnifying glass).

Do
man mount

The commands needed will be something like
mount /media/usbdisk
copy files etc
umount /media/usbdisk

depending on your mount point and name

[mileshc]

Mary, Thank you very much yet again.  I feel like a complete cretin.  Assumption is a dangerous thing! I could have sworn there was plenty of space.  Drat those end users. Ahh that may be me.

I checked with df -h which indeed revealed.

df -h
                       size   used   avail   use%   mounted
/dev/main/root   145G  145G   0       100%    /
/dev/md1           99M   45M   50M    48%     /boot

I have the deleted some old archives off the system & re-run the upgrade.

post deletions, boot is the same & "/" is 118G used & 21G available.

It now completes the upgrade & reboots, BUT right back to the minimal Grub.

I will now try your further suggestions etc.

Cheers,

Miles (CLI & more, challenged) HC

[mileshc]

Hello Mary & all.

The previous post may have inferred that the problem was resolved & caused by a full HDD. It MAY have been caused by it but deleting files & re-running an upgrade did not work.

I tried to connect a USB pen drive but could not get, due to lack of knowledge, find it or mount it from the rescue console, chroot /mnt/sysimage

I did find in /etc/fstab  dev/sdc1  /media/Expansion_Drive ntfs  etc....  This was the Segate device I connected that caused the problem.  I also found later under /media/Expansion_Drive a 36Gb smeserver.tgz file the aborted backup to USB disk file!

I finally fixed, at least sofar by booting from the 7.4 CD-ROM (thank you Mary) & sme rescue  chroot /mnt/sysimage

From here I used mc & navigated to /boot/grub & manually edited the 0 byte grub.conf file to read :-

default=0
timeout-5

title SME Server (2.6.9-89.0.16.EL)
      root (hd0,0)
      kernel /vmlinuz-2.6.9-89.0.16.EL ro root=/dev/main/root
      initrd /initrd-2.6.9-89.0.16.EL.img

This did not boot as the image file did not exist, good grief, so I altered (back to rescue console etc.) & amended to the last latest image in /boot which was 2.6.9-78.0.22.EL.img

This now boot up just fine.  I will I guess temporarily alter the NIC settings & see what an update & upgrade does & report back.

So connecting a new Ext. USB HDD out of the box (NTFS?) & running "backup to USB disk" is not recommended & not in the manual (RTFM!)

I am not to sure if it is worth testing this again to see if the fault is reproducible or not.  many Other things to do here really.

Cheers,

Miles HC

[janet]

mileshc

I suggest you format any USB drives you use on sme to ext3, see
http://wiki.contribs.org/USBDisks

[mileshc]

Arrghh!!

So I booted a few times & all seemed OK.

I amended /etc/fstab & removed the line relating to the non existent /dev/sdc /media/Expansion_Drive  etc.
I ran from root yum update & signal-event post-upgrade; signal-event reboot
I now have a system that boots so far and Kernel panics!
I have tried the previous image & current one in EL & smp versions it still panics.
I reverted to the previous hand made grub & it still panics (not sure why I did this)

I note the lvm & raid are not loading, error /bin/lvm no devices found.
unmount /initrd/dev  ?
Kernel panic non specific

I will have a shufty around to look for others with the same or similar.  I'd like to know why this has occurred, me fiddling with the fstab? updating? I have not had any such problems before in many years.  A little knowledge can be fatal?

Miles HC

[MSmith]

I'd also run a full hardware diagnostic including but not limited to RAM testing and using the manufacturer's diagnostic CD to test the hard drives and Dell's diagnostic CD to test the other system hardware.