Topic: Alternate methods to migrate from SME 8 to 9

[stabilys]

Sorry for longish post - preamble here to set the scene for my questions at end:
           
I have to do two functions on a SME8 server:

• upgrade to SME9
  
• move it to new hardware

The new hardware consists of

a. a virtual instance of SME-server 9

b. an iSCSI-mounted data store to be mounted over /home/e-smith/files

The data consists of over 3 TB of not very compressible material. A full backup and restore of this takes a Very Long Time indeed, and in 18 months, when the RAID 10 array will need to be rebuild again for more space (as it it will have doubled by then) it will be 6 TB and will take even longer.

I do not trust any USB drive to reliably backup and restore much more than 1 TB of data - I have had over the years various USB drives of good makes fail in these circumstances as they are not intended for continuous duty and in any case error rate is a problem with large data sets. Moreover most modern drives have auto power on/off which works well with Windows systems but not so good on Linux. I build my own but still have issues. Moreover, when restoring to a virtualised machine there is the issue of USB passthrough, which always makes me nervous.

I therefore intended to do this via backup to a server set up as a workstation; unfortunately the backups were corrupted as electricians in the building turned power on and off to half the premises. Since time was pressing I looked for another way.

I am sure many SME users have bigger data sets than these so a general solution for moving and migrating large data sets on SME-server would be useful to more than just me.


Proposed solution involves splitting data out, upgrading, then reattaching data:

1. Mount iscsi drive under a temporary mount point
2. Stop all services that will change the filesystem
3. Rsync (-a) the files under the path /home/e-smith/files to the storage server

The above part has been done, and the working server put back into use for now. Steps 2 and 3 can be updated just before the final migration.


4. Following the procedures here: https://wiki.contribs.org/Backup_server_config

Temporarily remount files/users and /files/ibays

mkdir -p /temp
mv /home/e-smith/files/users /temp
mv /home/e-smith/files/ibays /temp



5. Run backup to workstation of the diminished size system

6. Install new minimal SME 9 instance on virtualisation host

7. Restore the minimal SME 8 to the new instance to migrate all custom settings

8. On the old machine remount the moved directories:

mv /temp/users /home/e-smith/files/
mv /temp/ibays /home/e-smith/files/

9. Final temporary remount of storage server and rsync after stopping services etc.

10. Then shutdown old server, mount the storage server filesystem on the new virtual server with the rsync'd contents of /home/e-smith/files over  /home/e-smith/files.

(I have successfully done this part to move internal data store to SAN on other SME servers, but without the upgrade issue)


So to my questions:

A. Will this work? Is there any problem that I am overlooking that would block this working?

B. Are there changes in other system files under /home/e-smith/files that change drastically between SME8 and 9? (I can't see any: server-resources is different, at least here, but that’s easy)

C. Is there any variation of these procedures that would fix that?

Thanks for any input and your time, I know developers are all busy.

[Daniel B.]

I also had to manage upgrade on installations where the amount of data made it impractical to use the standard backup/restore procedure. What I always do is to use a separated block device for /home/e-smith/files. What I do is quite simple:

• I install SME9 on another box/VM/whatever
• I rsync /home/e-smith/files/ (with -a flag) from the old to the new one, this can be done while the old system is still running
• I manually create a tgz backup just like the standard console feature would do, but including some more pathes (where contribs I use store their data), and excluding /home/e-smith/files. This is usually a ~1GB tgz, so quite quick to create. Here's some notes on how I create this tgz file: https://wikit.firewall-services.com/doku.php/tuto/ipasserelle/divers/mise_a_jour_sme8_vers_sme9 (sorry, it's french, it's not a complete how-to, just some quick notes I took)
• I use the standard restore feature on the new system to inject this smeserver.tgz file (you can just copy it on an usb HDD and plug it into the new system)
• I now just have to stop all the services of the old system, run a final rsync to sync the delta of data (and using --delete), and that's about it

[stabilys]

Thank you Daniel for your reply. Je parle 'Francais a l'ecole'

I'll check your notes. I thought the general scheme would work   

MeJ

[stabilys]

All moved over and upgrade completed without issues. I'll write up what I did at some point when I have time so that others can find it.

Thanks Daniel.