Topic: SME 6 => SME 7 migration.

[silasp]

Hello.
Here is my (somewhat) successful SME 6 => SME 7 migration guide. Mind you, it's still a work in progress.

[1] Backup with rsync

Code: [Select]


# Make sure you check
# /usr/lib/perl5/site_perl/esmith/Backup.pm
# for the correct backup file list for your SME version
# it begins with "sub restore_list" (near line 70)

# Replace this IP with your remote host IP
RMH=192.168.1.1

cd /
# All these are directories - note trailing slashes
rsync -vPa -e ssh $RMH:/home/e-smith/ /home/e-smith/
rsync -vPa -e ssh $RMH:/etc/e-smith/templates-custom/ /etc/e-smith/templates-custom/
rsync -vPa -e ssh $RMH:/etc/e-smith/templates-user-custom/ /etc/e-smith/templates-user-custom/

# All these are files - note no trailing slashes
rsync -vPa -e ssh $RMH:/etc/group /etc/group
rsync -vPa -e ssh $RMH:/etc/gshadow /etc/gshadow
rsync -vPa -e ssh $RMH:/etc/passwd /etc/passwd
rsync -vPa -e ssh $RMH:/etc/shadow /etc/shadow
rsync -vPa -e ssh $RMH:/etc/samba/smbpasswd /etc/samba/smbpasswd
rsync -vPa -e ssh $RMH:/etc/samba/secrets.tdb /etc/samba/secrets.tdb
rsync -vPa -e ssh $RMH:/etc/smbpasswd /etc/smbpasswd
rsync -vPa -e ssh $RMH:/etc/sudoers /etc/sudoers


Remember to also backup any mysql databases you have running on the server.

[2] New install SME 7 from CD

[3] On the SME 7 server, backup the following files:
cp /etc/group /etc/group.bu
cp /etc/gshadow /etc/gshadow.bu
cp /etc/passwd /etc/passwd.bu
cp /etc/shadow /etc/shadow.bu

[4] Restore the files you backed up in [1] by running the same script on the new server and changing the IP to the server you backed the files up to.

signal-event post-upgrade
signal event reboot

Most things should work after this.

[5] Fix broken bits by copying any users and passwords missing from the group, gshadow, passwd and shadow files from the backups you made in step [3] - I'm not sure if cat group.bu >> group (etc) will do this - or whether it is better to use text editors and cut / paste.

[6] Log on as administrator to the "local machine" of all your windoze machines and rejoin the domain (you might have to temporarily join workgroup "nothing" to do this)

[7] Set up printers, etc

[8] Install contribs (eg Phpmyadmin, SME7Admin)

This worked for me. I recently did a YUM update and have run into some minor problems, but other than that, everything was pain-free.

Cheers,
Silas

[gordonr]

Hello.
Here is my (somewhat) successful SME 6 => SME 7 migration guide. Mind you, it's still a work in progress.

The correct procedure is:

Code: [Select]

signal-event pre-restore
  ...copy the files and directories listed in the restore_list...
signal-event post-upgrade
signal-event reboot


There should be no manual editing of /etc files required. You needed to because you didn't run pre-restore prior to doing what amounts to a restore.

[silasp]

Hello, thank you so much for posting this. Wow, that's great!
I've cross-posted a similar guide elsewhere, and will edit it accordingly, thankyou for helping out.

So that pre-restore thing takes care of all the problems associated with similar entries in the "new" and "old" user and group files? (Eg there's a clamd user entry in both the sme 6 and sme 7 passwd files, but they have different UID, PID and default shell.) That's really cool!

Cheers,
Silas.

[timlitw]

this is what I came up with based on your script - I'm testing it now
you will have to have the old sme root password and enter it for each new rsync instance that runs.

Code: [Select]

#!/bin/sh

#####################################################################################
# This script modified from contribs.org forums - can be run from new server only.
# it will run pre-restore to prepare the new sme server to accept the files and
# setting from the old server.
# then will use rsync to get needed files from the olde server to the temp directory
# then it will use rsync to merge them with the current files
# finally it will run post-upgrade and reboot to get the server back to the correct
# operating state.
#
# After I trust it more - the first rsync will be altered to modify the original file
# on the first rsync - then it won't need the temp folder.
#
#####################################################################################

signal-event pre-restore

# Replace this IP with your old sme server ip address
RMH=192.168.2.18
# Local Temp Directory
LTD=/root/tmp/


# make temporary storage place
cd $LTD
mkdir -p etc/samba
mkdir -p home/e-smith


# backup original users and groups to userbackup.tar.gz
tar -czvf /root/userbackup.tar.gz /etc/group /etc/gshadow /etc/passwd /etc/shadow

# prepare to merge current users and users from backup
rsync -vPa /etc/group $LTD/etc/group
rsync -vPa /etc/gshadow $LTD/etc/gshadow
rsync -vPa /etc/passwd $LTD/etc/passwd
rsync -vPa /etc/shadow $LTD/etc/shadow
rsync -vPa /etc/samba/smbpasswd $LTD/etc/samba/smbpasswd
rsync -vPa /etc/samba/secrets.tdb $LTD/etc/samba/secrets.tdb
rsync -vPa /etc/smbpasswd $LTD/etc/smbpasswd
rsync -vPa /etc/sudoers $LTD/etc/sudoers



# Make sure you check
# /usr/lib/perl5/site_perl/esmith/Backup.pm
# for the correct backup file list for your SME version
# it begins with "sub restore_list" (near line 70)


cd $LTD
# All these are directories - note trailing slashes
rsync -vPa -e ssh $RMH:/home/e-smith/ $LTD/home/e-smith/
rsync -vPa -e ssh $RMH:/etc/e-smith/templates-custom/ $LTD/etc/e-smith/templates-custom/
rsync -vPa -e ssh $RMH:/etc/e-smith/templates-user-custom/ $LTD/etc/e-smith/templates-user-custom/


# All these are files - note no trailing slashes
rsync -vPa -e ssh $RMH:/etc/group $LTD/etc/group
rsync -vPa -e ssh $RMH:/etc/gshadow $LTD/etc/gshadow
rsync -vPa -e ssh $RMH:/etc/passwd $LTD/etc/passwd
rsync -vPa -e ssh $RMH:/etc/shadow $LTD/etc/shadow
rsync -vPa -e ssh $RMH:/etc/samba/smbpasswd $LTD/etc/samba/smbpasswd
rsync -vPa -e ssh $RMH:/etc/samba/secrets.tdb $LTD/etc/samba/secrets.tdb
rsync -vPa -e ssh $RMH:/etc/smbpasswd $LTD/etc/smbpasswd
rsync -vPa -e ssh $RMH:/etc/sudoers $LTD/etc/sudoers


cd /
# All these are directories - note trailing slashes
rsync -vPa $LTD/home/e-smith/ /home/e-smith/
rsync -vPa $LTD/etc/e-smith/templates-custom/ /etc/e-smith/templates-custom/
rsync -vPa $LTD/etc/e-smith/templates-user-custom/ /etc/e-smith/templates-user-custom/


# All these are files - note no trailing slashes
rsync -vPa $LTD/etc/group /etc/group
rsync -vPa $LTD/etc/gshadow /etc/gshadow
rsync -vPa $LTD/etc/passwd /etc/passwd
rsync -vPa $LTD/etc/shadow /etc/shadow
rsync -vPa $LTD/etc/samba/smbpasswd /etc/samba/smbpasswd
rsync -vPa $LTD/etc/samba/secrets.tdb /etc/samba/secrets.tdb
rsync -vPa $LTD/etc/smbpasswd /etc/smbpasswd
rsync -vPa $LTD/etc/sudoers /etc/sudoers


signal-event post-upgrade
signal-event reboot

[brianr]

How do you get rsync NOT to ask for a password for each call?

[timlitw]

If you want to do that here are instructions

http://new.linuxjournal.com/article/8600

ro scroll down to simplefying ssh on this page
http://nerdvittles.com/index.php?p=123

[silasp]

Avoid typing passwords into the rsync script:
[Sorry for the cross-posting]

Log in as root to the local (new) server, enter
cd /root/.ssh
ssh-keygen -t rsa
(choose a randomish name eg "keyname" and press enter twice when prompted for a passphrase)

chmod 600 keyname*
scp keyname.pub [ip of remote server]:/root/.ssh/
(enter root password)

- Now connect to the Old server (replace 192.168.1.1 with actual ip)
ssh 192.168.1.1
(log on as root)
cd /root/.ssh

if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi
(above "if" command should all be on one line)
cat keyname.pub >> authorized_keys
rm -f keyname.pub
exit

Now you should be able to execute the script above without password prompts.

Cheers,
Silas.

[brianr]

Ok, this is the code I used, some corrections to avoid double "/" on some file paths, and also to initially create the /root/tmp directory.

I have used it sucessfully (with the keygen from above), and the new server seems to have all the data, emails, passwords etc that it needs.  Am running in test mode for a couple of days before committing to the new one.

This script looks as though it could also form the basis for an overnight sync of one server to another.

Code: [Select]

#!/bin/sh

#####################################################################################
# This script modified from contribs.org forums - can be run from new server only.
# it will run pre-restore to prepare the new sme server to accept the files and
# setting from the old server.
# then will use rsync to get needed files from the olde server to the temp directory
# then it will use rsync to merge them with the current files
# finally it will run post-upgrade and reboot to get the server back to the correct
# operating state.
#
# After I trust it more - the first rsync will be altered to modify the original file
# on the first rsync - then it won't need the temp folder.
#
#####################################################################################

signal-event pre-restore

# Replace this IP with your old sme server ip address
RMH=192.168.100.2
# Local Temp Directory
mkdir -p /root/tmp
LTD=/root/tmp


# make temporary storage place
cd $LTD
mkdir -p etc/samba
mkdir -p home/e-smith


# backup original users and groups to userbackup.tar.gz
tar -czvf /root/userbackup.tar.gz /etc/group /etc/gshadow /etc/passwd /etc/shadow

# prepare to merge current users and users from backup
rsync -vPa /etc/group $LTD/etc/group
rsync -vPa /etc/gshadow $LTD/etc/gshadow
rsync -vPa /etc/passwd $LTD/etc/passwd
rsync -vPa /etc/shadow $LTD/etc/shadow
rsync -vPa /etc/samba/smbpasswd $LTD/etc/samba/smbpasswd
rsync -vPa /etc/samba/secrets.tdb $LTD/etc/samba/secrets.tdb
rsync -vPa /etc/smbpasswd $LTD/etc/smbpasswd
rsync -vPa /etc/sudoers $LTD/etc/sudoers



# Make sure you check
# /usr/lib/perl5/site_perl/esmith/Backup.pm
# for the correct backup file list for your SME version
# it begins with "sub restore_list" (near line 70)


cd $LTD
# All these are directories - note trailing slashes
rsync -vPa -e ssh $RMH:/home/e-smith/ $LTD/home/e-smith/
rsync -vPa -e ssh $RMH:/etc/e-smith/templates-custom/ $LTD/etc/e-smith/templates-custom/
rsync -vPa -e ssh $RMH:/etc/e-smith/templates-user-custom/ $LTD/etc/e-smith/templates-user-custom/


# All these are files - note no trailing slashes
rsync -vPa -e ssh $RMH:/etc/group $LTD/etc/group
rsync -vPa -e ssh $RMH:/etc/gshadow $LTD/etc/gshadow
rsync -vPa -e ssh $RMH:/etc/passwd $LTD/etc/passwd
rsync -vPa -e ssh $RMH:/etc/shadow $LTD/etc/shadow
rsync -vPa -e ssh $RMH:/etc/samba/smbpasswd $LTD/etc/samba/smbpasswd
rsync -vPa -e ssh $RMH:/etc/samba/secrets.tdb $LTD/etc/samba/secrets.tdb
rsync -vPa -e ssh $RMH:/etc/smbpasswd $LTD/etc/smbpasswd
rsync -vPa -e ssh $RMH:/etc/sudoers $LTD/etc/sudoers


cd /
# All these are directories - note trailing slashes
rsync -vPa $LTD/home/e-smith/ /home/e-smith/
rsync -vPa $LTD/etc/e-smith/templates-custom/ /etc/e-smith/templates-custom/
rsync -vPa $LTD/etc/e-smith/templates-user-custom/ /etc/e-smith/templates-user-custom/


# All these are files - note no trailing slashes
rsync -vPa $LTD/etc/group /etc/group
rsync -vPa $LTD/etc/gshadow /etc/gshadow
rsync -vPa $LTD/etc/passwd /etc/passwd
rsync -vPa $LTD/etc/shadow /etc/shadow
rsync -vPa $LTD/etc/samba/smbpasswd /etc/samba/smbpasswd
rsync -vPa $LTD/etc/samba/secrets.tdb /etc/samba/secrets.tdb
rsync -vPa $LTD/etc/smbpasswd /etc/smbpasswd
rsync -vPa $LTD/etc/sudoers /etc/sudoers


signal-event post-upgrade
signal-event reboot


[vdeg]

Hi
I used this script to migrate to SME 7 from  SME 6.0.1 and everything worked except for webmail this is what I get when I go to https://myserver/webmail/
A fatal error has occurred
DB Error: connect failed Details have been logged for the administrator.
here is the error from message log.
Apr 24 14:05:38 newstupig HORDE[17134]: [horde] DB Error: connect failed:  [nativecode=Access denied for user 'horde'@'localhost' (using password: YES)] ** Array [on line 1329 of "/home/httpd/html/horde/lib/Horde/DataTree/sql.php"]
any idea how to fix this?
thanks

[timlitw]

yeh, run  the post-upgrade event again and the reboot. It apparently didn't finish and mysql isn't running

[netspirit]

Hi all!

I'am facing problem using this script to migrate. After runing it and redo a signal-event post-upgrade
signal event reboot
several times, I still do not have the MySQL db. So I copied them by hand (stopping MySQL during copy) but still getting dabase connection errors on all web site and Horde.
Is there anybody who has a sure migration howto?
Thanks in advance.

[timlitw]

Open another console and tail /var/log/messages
while runing the signal-event post-upgrade
see if there is anything erroring during the post-upgrade
that might help us narrow it down.

[ddougan]

Might the problem be that the procedure doesn't run the pre-backup routine on the old server, and that therefore the MySQL database(s) haven't been dumped and then brought over?

[raem]

netspirit

> I still do not have the MySQL db. So I copied them by hand (stopping MySQL during copy) but still getting database connection errors on all web site and Horde.

Did you copy the mysql db too ?
If not you need to add all your users back in there.
Also set correct permissions
cd /var/lib
chown -R mysql:mysql mysql

[raem]

netspirit

> the MySQL database(s) haven't been dumped

See
/etc/e-smith/events/actions/mysql-dump-tables

which will create a dump file called
/home/e-smith/db/mysql/mysql.dump

[raem]

Probably requires a
/sbin/e-smith/signal-event pre-backup

See alternative method for restoring directly from disk

http://forums.contribs.org/index.php?topic=30745.msg144369#msg144369

see my original post of 19 Sept revised 24 Sept

[netspirit]

Hi Ray,
Sorry for late reply. So I did a lot of migration since my last message here and everytime I had to copy manually all databases from the old server to new one but it is working as per your idea. So thanks a lot !

An happy SME 7.0 user from France

[netspirit]

Hi all,
I found after few days of several migrations from 6.0.01 to 7.0 some big problem for us. In fact it is inside MySQL. Installing PHPmyadmin-multiuser (last release which is PHPMYADMIN 2.9.0.2) has shown us the problem. We can't connect with it properly, we are log-off after each refresh or action inside PHPMyAdmin and we need to set again the langage on the logon page before login again. It's returning always to UTF8 langage which is avoiding us to log-in with an error message saying :"#1045 - Access denied for user 'admin'@'localhost' (using password: YES)". But if we set the langage as ISO-8859-1, we can log in again.
Inside MySQL we found that all migrated databases are in collation : ISO-8859-1 and new ones are in UTF-8 like MYSQL.
So is this a normal consequence of our migration? If yes, is there a solution to solve this or not?
I do not know how to do today and searching, searching...

Regards

[raem]

netspirit

> Inside MySQL we found that all migrated databases are in collation :
> ISO-8859-1 and new ones are in UTF-8 like MYSQL.
> So is this a normal consequence of our migration?

That sounds like something you should be reporting to the bugtracker.
That approach may also be the quickest way to resolve your issues.

[actron]

Log in as root to the local (new) server, enter
cd /root/.ssh
ssh-keygen -t rsa
(choose a randomish name eg "keyname" and press enter twice when prompted for a passphrase)

chmod 600 keyname*
scp keyname.pub [ip of remote server]:/root/.ssh/
(enter root password)

- Now connect to the Old server (replace 192.168.1.1 with actual ip)
ssh 192.168.1.1
(log on as root)
cd /root/.ssh

if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi
(above "if" command should all be on one line)
cat keyname.pub >> authorized_keys
rm -f keyname.pub
exit

tried, but it does not work.

[byte]

Log in as root to the local (new) server, enter
cd /root/.ssh
ssh-keygen -t rsa
(choose a randomish name eg "keyname" and press enter twice when prompted for a passphrase)

chmod 600 keyname*
scp keyname.pub [ip of remote server]:/root/.ssh/
(enter root password)

- Now connect to the Old server (replace 192.168.1.1 with actual ip)
ssh 192.168.1.1
(log on as root)
cd /root/.ssh

if [ ! -f authorized_keys ]; then touch authorized_keys ; chmod 600 authorized_keys ; fi
(above "if" command should all be on one line)
cat keyname.pub >> authorized_keys
rm -f keyname.pub
exit

tried, but it does not work.

Can you tell us what you mean by it does not work? i.e which bit didnt work?

[actron]

Can you tell us what you mean by it does not work? i.e which bit didnt work?

Excuse, i was not clearly enough
want to say, all of this work. but password must always enter.

[jfarschman]

Brian,

  Thanks for this.  I read it over and saw the controls and logic immediately.  For those working through this script I have a couple of comments.

  1.  First you need to set up an SSH trust between the remote $RMH machine and the machine you are running the script on.  This is pretty easy.  Just do a  search for "ssh-keygen" and you'll find plenty of help.

  2.  Watch your disk space.  This script doesn't worry about disk space and copies the files to /root/tmp and then to appropriate destination.  Generally, this is a good idea, but if you are like me and have a pile of i-Tunes video dowloads you may run out of disk space.

      For disk space conscious people change line 55;

Code: [Select]

rsync -vPa -e ssh $RMH:/home/e-smith/ $LTD/home/e-smith/
       
      To write directly to your user and ibays directories.  Also comment out line 73 which copies the tmp ibays to the live location.

Code: [Select]

rsync -vPa -e ssh $RMH:/home/e-smith/ /home/e-smith/
     
I like the idea of having all the settings and templates in a /root/tmp directory, but my ibays are enormous.

  Thanks Brian.

[jfarschman]

About Key generation

I generally do not specify a key name.  My process goes like this:

ssh-keygen -trsa

# just hit return a few time and ignore the prompts

This will generate id_rsa.pub and the private key as well.  Copy the contents of this key into the authorized_keys file, that may or may not exist... and copy it to the authorized_keys of the other machine.