Topic: RAID1 failure; data missing

[Brenno]

Ok, so let's set aside for a minute that I'm still using a 6.x release in one of my installations... it's been running just peachy as an email server (until now) and we were actually just about to replace the machine entirely with new hardware and the latest version of SME.  Anyway...

I have 4 SCSI drives in this box, 0 and 1 were in RAID1, 2 and 3 were connected but not used.  1 failed, and then 4 shortly thereafter.  So, I followed a howto on here to partition 3 to match 0 and then rebuild the raid.  Seemed to work, but upon reboot, 3 is not a member of the raid!

More perplexingly, when I was able to boot into degraded mode, I mysteriously lost almost 2 months of data... nothing between September 24th and today exists in user folders.  I thought RAID1 was supposed to mirror the drives, so upon failure of one, the other had a complete copy... this appears to not be the case!  I was able to do a 911 backup, but suspect that this, too, will be missing the data.

I really just need to do two things: 1) have this box run for another 10 days until the new hardware is in place, and 2) recover the missing data.

Hoping the forum experts can provide some assistance (and go easy on the chastising for the old version still in service issue...)

Appreciated in advance.

[Brenno]

Now I am even more confused as it appears that this server has gone back in time to September 24th:

- user accounts added since then have disappeared
- user accounts removed since then have reappeared with data intact
- domains added since then have disappeared
- passwords changed since then have reverted back to what they were
- deleted emails are restored

Is it possible that the server was storing all new/changed data since 09/24 on the failed drive?

My game plan now is to continue to use the server as-is in degraded mode until it can be replaced next week.  If anyone has any better ideas, I'm excited to hear them.

Thanks.

[David Harper]

Sounds like rip-n-replace is the best option you have right now. Were you using a tape backup that might have the missing data on it?

[Brenno]

I do have a valid and complete backup thanks to dmay's backup2ws contrib.

As for the time warp, turns out the mirroring of the drives must have broken on 09/24 without us realizing it.  When the mirroring broke, the server picked one of the two drives to continue on with, and sadly it was the drive it chose that ultimately failed.  The other drive simply wasn't being updated, so it's "stuck in time" back on 09/24.

I've curious as I know I have rebooted that server several times between 09/24 and now, but never once did I see any indication that there was an issue with the RAID (and I religiously watch the messages stream by on startup).

[David Harper]

I'm not sure you'd see much at startup, especially if the array is hardware-based. In SME 7.3, there's a console panel that lets you manage RAID arrays.

[Brenno]

Here's my thoughts on data recovery for this... going to set up a fresh install of 7.3 (with all updates) on a new box.  I have three data sets, all containing various stages that I'd ideally like to combine.  Note that these backups are user files only (ie /home/e-smith/files/users and below):

Set1 = all files up to 9/24 and after 11/17 (current 6.x box, crippled as it may be)
Set2 = all files up to 11/16 (partial backup; not all users included)
Set3 = all files up to 11/9 (last complete backup)

So, I'm thinking that I could use rsync to first "prime" the 7.3 box with Set3. Then, I'll run rsync against Set2 to get what exists there but not in Set3, and finally, I run rsync on Set1 to get what exists there but is not in Set3 or Set2.  When all this is done, I will need to correct the permissions and whatnot.

I believe that if I run rsync with the correct options, I should be able to end up with nearly all the data (save for a few users whose 11/9 to 11/16 data was were not included in Set2).

Does anyone here see any flaws in this logic?  Am I missing something?

[pfloor]

There ia a lot more than just home directories to consider when trying to restore a system. Going from 6.x to 7.x is complicated and the upgrade process is quite extensive under the covers.  It's not just "load and copy".

-How many users are on the old system and how do you intend to create them on the new one?
-Is there a lot of email?

[Brenno]

We have <60 users on this system, all running in IMAP accounts, with about 12GB of data in total.  We aren't storing anything in the home folders, just the Maildir at this point.  My plan is to recreate the users manually as we will do a password change as the new accounts are created.

I realize that, due to the way the mail files are altered and suffixed by doing things like marking for deletion and forwarding, I am likely to get some duplication of emails in different states, but I think the users can purge these duplicates and would be happier to have them then no email at all.

My plan B is just to use rsync to prime the data with Set3 and have users move/copy any post 11/17 emails from Set1 manually to their new account (running both servers concurrently for a short period).

[David Harper]

You can use lat-users (Lazy Admin Tools) to batch add your 60 users from a CSV file, which should save you some heartache.

[janet]

Brenno

It will save you a good bit of manual work if you upgrade the sme6 server to sme7 and then manually move Maildir's etc to the new sme7 server.
Quite a few routines get run during the 6>7 upgrade process to rename folders etc to suit sme 7.

[Brenno]

My level of confidence in the 6.x box surviving the upgrade to 7.x is not high.  Perhaps I should go the other way around and start the new hardware on 6.x, migrate the users and data, then perform an upgrade to 7.x?

I suppose I could restore the good backup (Set3) using the backup2ws contrib, then take care of the rest of the files manually.

[janet]

Brenno

My level of confidence in the 6.x box surviving the upgrade to 7.x is not high.

If you follow the instructions here (exactly too)
http://wiki.contribs.org/UpgradeDisk

then you retain your fully working sme6 disk, and can easily go back to sme6, in the event that you run into unresolvable problems during the upgrade to sme7.

sme6 to sme 7 upgrades are supported, so it should go OK as long as you remove incompatible contribs and custom templates and user custom templates (as per instructions).

The only area I'm aware of that seemed to be a problem in some cases was workstation machine accounts on sme6 not being recognised under sme7, necessitating the removal of a Windows workstation from a (sme) domain (controller) and the rejoining of it to the domain.

[Brenno]

My confidence (or lack thereof) is more indicative of the current state of the hardware, not the software...  two failed drives and a bad power supply fan means that I've been nursing this box for the past week and don't want to introduce any more "stress" than I have to

I think I'll run with my latest plan (start with 6.x, restore, upgrade) as this will be easier given that I won't have any extra contribs or custom templates present when I start with the clean install of 6.x

[Brenno]

You can use lat-users (Lazy Admin Tools) to batch add your 60 users from a CSV file, which should save you some heartache.

Of course, everything so far has gone wrong, so I shouldn't be surprised that my both my 6.0b3 and 6.0 Final install disks failed to load on the new hardware.  Had to install 7.3 directly, so my plan has been scrapped and is now being written on the fly

I did use the lat-dump to get info from the 6.x box I was running and manually modified the files to add the additional users which had been lost in the failed mirror.  Now, can I use these scripts to run the lat-restore on the 7.3 box, or are the differences in the two systems too great for this?

[janet]

Brenno

Had to install 7.3 directly, so my plan has been scrapped and is now being written on the fly

So just restore directly from the mounted sme6 disk to a fresh install of sme7 (as per previous link provided) and make life easy for yourself.