Koozali.org: home of the SME Server

server did automatic system restore!! HELP!!!

larry007

server did automatic system restore!! HELP!!!
« on: September 27, 2006, 01:45:33 PM »
My sme7.0 server did a system restore by hook and by crook - all by itself during a reboot.  It reverted back to settings/user data from about 6 weeks ago!!  I did not know that this was a feature SME posessed   :oops:

It lost the NICs yesterday which I eventually got up and running - thought it was weird.  System settings that was changed are obviously lost i.e. quota settings etc, but with that I can live.  Emails simply came through again as unread - interesting feature as well....

Work files and data, including financials of more than a month is simply missing which is totally unacceptable, not even for Windows! What can I do about the data that simply dissapeared?  THIS IS URGENT!!!!!

Mirror drive refuse to boot up completely, was a loooong shot as the console stated the mirror was unlcean, but hey...  I am desparate.

Can you help me?  Please post a solution before the boss fires me!!!

Offline byte

  • *
  • 2,183
  • +2/-0
Re: server did automatic system restore!! HELP!!!
« Reply #1 on: September 27, 2006, 02:04:46 PM »
Quote from: "larry007"

Can you help me?


Please report any problems to the Bug Tracker, and explain your setup
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline chris burnat

  • *****
  • 1,135
  • +2/-0
    • http://www.burnat.com
server did automatic system restore!! HELP!!!
« Reply #2 on: September 27, 2006, 02:10:11 PM »
Larry, I do not think that what you are experiencing is a standard feature of sme7 or for that matter of sme.  Please describe fully the sequence of events to the best of your recollection.  How did you "loose: your network cards"  What happend to the mirror?  Take it in steps, telling how all this started, what you believe was wrong and what you did about it.  Then someone may be able to help you.
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

Offline byte

  • *
  • 2,183
  • +2/-0
server did automatic system restore!! HELP!!!
« Reply #3 on: September 27, 2006, 03:00:00 PM »
Quote from: "burnat"
Larry, I do not think that what you are experiencing is a standard feature of sme7 or for that matter of sme.  


Chris,

I think he may have been using mirror drives so if one drive went down the other would take over, but it seems one drive went down and the data wasn't sync'ing correctly between the 2 drives.
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline chris burnat

  • *****
  • 1,135
  • +2/-0
    • http://www.burnat.com
server did automatic system restore!! HELP!!!
« Reply #4 on: September 27, 2006, 03:18:40 PM »
I
Code: [Select]
think he may have been using mirror drives so if one drive went down the other would take over, but it seems one drive went down and the data wasn't sync'ing correctly between the 2 drives.

Make sense, thanks Byte.  If the case, its going to be a long way up the proverbial creek without a paddle....   Meaning that for the past month, Larry should have received warnings about his array being degraded. (':(')
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

larry007

update
« Reply #5 on: September 27, 2006, 03:46:03 PM »
The server had a mirror set which was synch'ed just fine - up until yesterday after the first reboot.

Connectivity was lost after 18days uptime.  The server was rebooted in my absence, presumably with the reset button....  

Upon the reboot there was still no connectivity, althought the server responded fine.  I tried kudzu, as well as re-configuring the server form the console without success.  I noticed the network wasn't detected using ethtool eth0/1.  Performing vi /etc/sysconfig/hwconf showed the hardware existed fine.  In /etc/sysconfig/network-scripts/ I noticed that there were no ifcfg-eth0/1 files.  I promptly created these manually, performed driver update command (which I cant' remember from bugtracker), did the signal-event console-save and rebooted.  Networking was restored, but with no routing.  From the console I re-configured the server and all was now working as it should.  Expect that I now had this mysterious data loss, system settings loss etc as described previously.

Logs are BIG, but yesterday's reveal it trying to use newer kernel of some kind - with an additional 0.2 at the end.  Yum update reveals that this new kernel exists, but I haven't performed any updates in the last month, so I have no idea where this setting came from.  I can only guess that this triggered a whole lot of actions and failures, as can be rightly seen from the logs.  But how it is possible to manipulate the system in such a way that data is merely lost, is a mystery.

Point being that I have no idea what brought this phenomena about, even less how to restore the data.  I still can't believe it, but it really seems as if someone took a snapshop of the system a month ago and merely restored it.  

The MD didn't think USB drives to be important at the time for backups, so I guess we're are screwed now...

larry007

about the mirror
« Reply #6 on: September 27, 2006, 03:52:29 PM »
Just want to complete the topic.  Software mirrorset was fully functional, but is currently broken.  In desperation I tried to boot from the slave, hoping that the data wasn't overwritten because of the broken mirrorset.  

Unfortunately the slave doesn't boot the system and is thus useless.  If the mirrorset was in tact, it would have been a useless exercise, as the problem would only have been duplicated.

Sorry for the confusion caused

Offline byte

  • *
  • 2,183
  • +2/-0
Re: update
« Reply #7 on: September 27, 2006, 07:26:36 PM »
Quote from: "larry007"
but yesterday's reveal it trying to use newer kernel of some kind - with an additional 0.2 at the end.  Yum update reveals that this new kernel exists


It looks like your using the centos boot kernel which is why your mirror seems not to be working as the SME version is...

SME Server 2.6.9-34

Where you are using...

CentOS 2.6.9-34.0.2

If you reboot machine go to the bootloader change to the SME Server kernel all my well be working
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

larry007

solution?
« Reply #8 on: September 28, 2006, 10:09:07 AM »
So do you wreckon that if I boot with the correct kernel that all data would be restored?  THIS IS MY LIFE.....

If for some reason the data was lost because of the wrong kernel that was loaded, where can I find the data.  IF no restoration thereof can be done within the system, what can I do next?  Can I send the drive to forrensic company to do a data recovery?

Since I suspect that the kernel does not exist, would I need to perform a yum update first?

Offline byte

  • *
  • 2,183
  • +2/-0
server did automatic system restore!! HELP!!!
« Reply #9 on: September 28, 2006, 10:31:39 AM »
Yes it's a possibility that all will be well.

No the kernel will already be there you will just have to reboot and select the right SME Server kernel NOT a Centos kernel.
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

larry007

Mirror available again
« Reply #10 on: September 28, 2006, 11:02:26 AM »
Hi Byte,

Mirror option is available again, but data is still missing.  Do I start a search for a data recovery company?

If for some reason the data was lost because of the wrong kernel that was loaded, where can I find the data. IF no restoration thereof can be done within the system, what can I do next? Can I send the drive to forrensic company to do a data recovery?

Offline william_syd

  • *****
  • 1,608
  • +0/-0
  • Nothing to see here.
    • http://www.magicwilly.info
Re: update
« Reply #11 on: September 28, 2006, 12:37:10 PM »
Quote from: "byte"
Quote from: "larry007"
but yesterday's reveal it trying to use newer kernel of some kind - with an additional 0.2 at the end.  Yum update reveals that this new kernel exists


It looks like your using the centos boot kernel which is why your mirror seems not to be working as the SME version is...

SME Server 2.6.9-34

Where you are using...

CentOS 2.6.9-34.0.2

If you reboot machine go to the bootloader change to the SME Server kernel all my well be working


I didn't think SMEServer developers repackaged kernels.
Regards,
William

IF I give advise.. It's only if it was me....

Offline raem

  • *
  • 3,972
  • +4/-0
Re: update
« Reply #12 on: September 28, 2006, 12:55:56 PM »
larry007

The first thing I would do is STOP playing with the system and making changes etc etc etc.

Immediately disconnect both drives and secure them safely for later analysis. If there was data to be recovered then due to your actions carried out already, it may be too late to recover.

Ideally you would clone each drive and put the originals aside for later data retrieval.

Using the cloned drives I would try to boot up to ONLY one drive at a time in degraded software RAID mode.

That way you will see what configuration and data is on each drive. Don't make any changes, just observe.

You mentioned a problem booting to the slave, you should be able to do this IF THE DRIVE IS OK, you may need to configure the BIOS to boot from hdc ( D: ) instead of hda ( C: ). I was just doing this the other day and that's what I had to do. The BIOS setting was stopping the Secondary channel drive from booting alone in degraded mode.

Perhaps due to the wrong boot kernel, your data was only being written to one drive, and that drive has failed, and the system reverted to the other drive which had old data on it (as it was hda and the system was only booting to that drive).
Perhaps a user swapped a drive, and replaced it with a drive that had old data on it and the system is now booting up to that drive, is that possible ?
I think drives should be clean when added to an array, or you risk booting to the OS and/or config on the newly inserted drive, rather than the current data.

If both drives are now sync'd to old data, then there will be nothing to retrieve from the second drive in the array. If the array is not sync'd then you stand a chance of getting something.

All your incorrect config issues could be explained by the above suggested scenarios.

It is also possible that a user inadvertantly did a system restore, I don't think sme is capable of automatically doing that. Is it possible for users to do that in the situation the server is in.

> Emails simply came through again as unread - interesting feature....

To be expected if the message files are restored to the Maildir cur folder


> Work files and data, including financials of more than a month is simply > missing which is totally unacceptable, not even for Windows! What can I > do about the data that simply dissapeared?

Clearly your system is reading old data, as that is all that is on the array  disk(s). There is no current information on the disk or disks being read.


> The server was rebooted in my absence, presumably with the reset button....  

So you don't really know what someone did, and they probably won't tell you either. Therefore anything is possible.


>  From the console I re-configured the server and all was now working as it should.  

A sign that old config existed on the server.


> But how it is possible .... that data is merely lost.

Lost is probably not how it happened, replaced may be a better word to use.

Hopefully if you can boot to the "other" drive in the array, it may have the more recent data on it.


Unfortunately if any of the above suggestions do not work out, then your ONLY answer is to restore from recent backups. You said something about no USB disks, so are you saying that this company does not do backups at all, for a whole month too ?

If so, I bet they will (do daily backups) after this !
...

Offline raem

  • *
  • 3,972
  • +4/-0
Re: Mirror available again
« Reply #13 on: September 28, 2006, 01:03:33 PM »
larry007

> Can I send the drive to forensic company to do a data recovery?

It's an option, but whether they will be successful is questionable.
It's hard to say as you cannot even tell us what happened with this server, so we are left to guess.

As suggested in my earlier post, STOP using the drives immediately & clone them if doing further data recovery testing.

You could build another Linux or sme server, and try mounting the drives (one at at time, and see if you can recover usable (recent) data from them.

Forensic recovery is really for broken drives, and my limited understanding of ext3 file systems is that you are unlikely to be able to retrieve deleted files anyway.

It doesn't sound like deleted files is your problem either.
...

Offline JonB

  • *
  • 351
  • +0/-0
server did automatic system restore!! HELP!!!
« Reply #14 on: September 28, 2006, 01:23:20 PM »
Larry007,

I agree with Ray. I suspect that your mirror hasn't been synchronising for the past 6 weeks. The drive that was working has failed/corrupted and the server booted off the out of date drive.

Take a look at the logs. Is there a 6 week gap?

I suspect that the chances of recovering data off the drive now that you have re-synched the mirror are minimal.

Jon
...