Topic: Migration strategy

[DanB35]

As mentioned in a couple of other threads, I'm looking to migrate my SME 9 installation to a VM on a Proxmox host.  I have a number of contribs installed, and I'm wanting to make sure I have a good plan for the migration.  Here's what I'm thinking:

• /sbin/e-smith/audittools/newrpms > /root/newrpms.txt
• Disconnect server from network
• Back up to USB device from the console menu
• Install SME 9 on VM
• Restore from backup, on first boot wizard
• Go back through "configure this server" to make sure NICs are set properly
• yum update
• Reinstall RPMs listed in newrpms.txt
• post-upgrade && reboot
• Profit?

I'm thinking the backup/restore will get any custom template fragments, any additional repos, and any config database entries for the contribs.  Anything I'm missing with this list?

[Fumetto]

A good prayer to your God would be all well and good...

[DanB35]

Well, that goes without saying... (-:  I'm not planning on overwriting the existing disks, though, so I figure in the worst case I'll just plug those back in and I should be back in business with the old install.  I won't have gained anything in that event, but I won't have lost anything either, other than several hours of downtime.

[holck]

Why don't you create a new VM with a SME 9 installation first? And then install the missing rpms, do the yum update and make sure everything is fine, before you disconnect the old server from the network and do the backup?

[DanB35]

Two reasons:
1.  The VM host will be on the same hardware as the current SME installation
2.  The restore from USB backup is only available on the first boot of the system.

I could mitigate reason #2 by using a different backup/restore method (I'm already backing up my server across my network to my FreeNAS server, for example), but restoring from any other backup type involves completely configuring the server first, only to have that configuration wiped out by restoring from the backup.

Edit:  In any event, I think the server needs to be off the net from the time when the backup begins until the restore ends, to avoid missing things like incoming emails and such.

[janet]

holck  (& Dan FYI)

That approach is OK for test purposes where you totally wipe that test install later, & then reinstall the new OS to blanked drives.
Restore should only be made to a fresh install of SME OS, with only basic network setup where needed. If you add users & ibays & install contribs before doing the Restore, then you can end up with orphaned entries, folders & so on. The restore does not overwrite all entries/folders on the new server, only those that exist in the backup (per inclusions standards).

Contribs should be reinstalled after the Restore is done, they will use the data &  configuration from the restore, where the old & new contrib versions are compatible.
SME9 has a lot of path changes & almost all contribs needed code amendments, check for any code/functionality differences.

It is probably a good idea to make a test SME9 server, perform a restore  & install current versions of contribs, to be sure everything works as expected. Better to be safe with your data & functionality, than sorry.

Also you can run a command to reset the switch to be asked if you want to restore on first boot, refer Backup server config Howto. Also that Howto details the correct sequence ie Install fresh OS to clean drives, restore backup, install contribs.

One other comment, it is better to move all custom templates either before the Backup (on old server) or after Restore (on the new server).
Contribs will install necessary templates, & you can then recreate any custom templates as required & to suit the new OS version & package requirements/limitations/switches/standard templates etc.

You can disable the mail system (eg qpsmtpd status disabled & so on) & even disconnect LAN cable if there are LAN users who may change files etc, to prevent data chaging while backing up & restoring.

[DanB35]

Janet,

With respect to your notes on changed paths and versions of contribs, I'm already running SME 9, so I wouldn't think they would apply here, right?

[ReetP]

Janet,

With respect to your notes on changed paths and versions of contribs, I'm already running SME 9, so I wouldn't think they would apply here, right?

If they are pulled from the SME v9 contribs they should be fine, but if you have hacked about with any custom templates and amended paths then please check them.

B. Rgds
John

[DanB35]

Well, the migration is running--restore is 20% finished.  I just saw another round of 20 or so software updates the other day; I didn't bother installing them since there will be 300 or so to install once the restore finishes anyway.

One thing I didn't think about is that cron jobs aren't part of the backup.  Since I have one for owncloud maintenance functions, I'll need to recreate that once the system's back up.

[DanB35]

I left to run some errands, and when I came back, the restore had apparently completed, and the server had rebooted itself.  All the data seems to have made it across, and all the accounts, config data, etc. were preserved as well, as expected.  Something that was not preserved, though, were the SQL databases.

I ran the "backup to removable media" from the console menu, which completed successfully (and should have run a pre-backup event before tarring up everything), and I used the restore option on the new installation.  I don't know why they weren't preserved, but they most definitely weren't.

Fortunately, since I hadn't overwritten the old drives, I was able to boot from one of them (on a spare computer), run signal-event pre-backup, and copy /home/e-smith/db/mysql/*.dump over to the new server, and then do mysql < whatever.dump for the databases I needed.  I also needed to manually add the database users back to mysql (these would probably have come across if I'd imported the mysql.dump file, but I was concerned that it might bring in other data that could hose my system).

[janet]

DanB35

Well that is a bug, please lodge a bug report at bugzilla.
All mysql databases should be restored.
The triage devs will want to look at the logs on both servers to see what went wrong. This could affect many people if it really is a code bug, so therefore needs to be resolved.
Can you replicate it on test servers ?

[DanB35]

I'll see if I can reproduce it.  What did we do before VMs?

[janet]

DanB35

I had a couple of old Celeron PCs with 2 drives in RAID1 & I pulled out one drive, tested in degraded mode, then rebuilt the array to original when finished. Much faster & easier with VMs.

[DanB35]

Well, after a bit of a go-round trying to get the backup to work on the VM (and finding courtesy of bug 9126 that the backup won't work if there's a CD/DVD in the (virtual) drive, whether or not it's mounted), I've found two things:
1.  The unprompted reboot after the restore is normal--as soon as the restore finishes, the machine reboots automatically, without prompting or waiting.
2.  On testing, the databases were preserved.

If I can't duplicate the database issue, I don't know how much help a bug report would be.

[janet]

DanB35

2.  On testing, the databases were preserved.
If I can't duplicate the database issue, I don't know how much help a bug report would be.

The original backup & restore process should have "just worked".
Looking at logs on your original server or examing the contents of the backup, should determine if the mysql dbs dump file was created.
If ithe dump was not created then it is understandable that the mysql dbs were not restored.
If the dump was created then why was it not restored when other data was restored ?

These issues would not seem to be related to issues of your USB not being detected or read properly as your backup did run & complete but "failed" in a vital respect.

Replicating your issue on a test server is useful and/or diagnostic, but it does not really troubleshoot your original issue.

A bug report on the failed restore of mysql dbs should still be lodged, & please do it asap before you loose the
forensic diagnostic trail that is still available in your log files etc.

Let the bug triage devs determine whether it is a bug or not.
Remember you are lodging a suspected bug, you do not need to prove it is a bug before lodging a "bug" report.
Your issue could be a code bug, a wiki documentation bug, or just a user bug, which is yet to be determined.

Backup & restore should seemlessly just work without fail.

[DanB35]

After thinking a bit more about it, I came to the same conclusion and reported it earlier this afternoon.

[TerryF]

See Bug 9130 for details..upgrading from a SME8.2 to SME9.0

The differences between the SME8.2 dar and libdar versions compared to SME9.0 appears to be the cause of this issue, smeupdates-testing currently has new rpms waiting for release.

SME9.1RC1 has the latest versions so is not susceptible

So if you have a sme8.1 system updating to sme8.2 prior to doing a upgrade will be problematic if going to sme9.0 using the backup/Restore functions from the server-manager. Please NOTE this only affects the soon to be obsoleted SME9.0....

Upgrade to a SME9.1RC1 system is unaffected

Backups from the console are different and would appear to be OK from my testing to date

[TerryF]

Latest dar and libdar packages have been made avail for SME9.0 from smeupdates, these are defaults with SME9.1rc1

[root@sme90x64 ~]# yum update dar libdar

both dar and libdar need to be updated, just doing dar is not sufficient

See bug 9130 comment 32 for verification http://bugs.contribs.org/show_bug.cgi?id=9130#c32