Koozali.org: home of the SME Server

Upgrading vs new install + restore

Offline chris burnat

  • ****
  • 1,135
  • +2/-0
    • http://www.burnat.com
Upgrading vs new install + restore
« on: February 17, 2006, 02:04:17 PM »
I am reprinting below a summary of exchanges on various lists on this subject.  Many of us are/will be facing this challenge, and a pathway through it honed by the community would be great.  If we come out with a good set of recommendations, then we should host it on the contribs wiki (thanks to Brian Read for the idea).

[Answers below are provided by Gordon Rowell].
Q: Updates from a lower versions are going to bring lots of challenges, like it or not.  
A: They work cleanly for unmodified systems and systems with released updates. The
dev team has now done a huge number of upgrades from 6.x of various flavours
and quite a lot of upgrades of modified systems.

Q: How about recommending new installation of 7, and restore of user data via destop method?  
A: That won't help with custom templates and some contribs still install custom templates.
Also, many people have too much data for desktop backups.

Q: This should work irrespective of whether contribs have been installed on the previous version?
A: No, but this would:

- Backup
- Install 7.0
- Restore
- Disable all custom templates:

cd /etc/e-smith
mv templates-custom templates-custom.old
mkdir templates-custom

mv templates-user-custom templates-user-custom.old
mkdir templates-user-custom

signal-event post-upgrade
signal-event reboot

Or this:

- Remove all packages which are not on the 6.0/6.0.1/6.5 CD
- Disable all custom templates:

cd /etc/e-smith
mv templates-custom templates-custom.old
mkdir templates-custom

mv templates-user-custom templates-user-custom.old
mkdir templates-user-custom

- Upgrade from CD
Carefully reinstall SME7 versions of the contribs, if they exist.

Comments by Steve Bush:
Great info.....It'll come in handy when I roll out the SME7 upgrades.
I ran into this problem with a contrib that I missed uninstalling before my
first sme7 upgrade. I installed from scratch, restored from desktop and couldn't figure out that
there were hidden (.qmail) files in the templates-user-custom directory.
If there was a check box in the restore page "don't restore custom apps", it
would probably fix a lot of problems. Thanks again for the information.

Comment by Gordon Rowell:
“Great info.....It'll come in handy when I roll out the SME7 upgrades.”
Raised in bug 812. The dev team are raising lots of things in the "SME Server Documentation" category in Bugzilla.  We need some serious effort on the SME7 FAQ. I would really like to start again - much of the FAQ needs to be rewritten with SME7.
Volunteers - please contact docteam@lists.contribs.org

Comment by Charlie Brady:

“ Also many people have too much data for desktop backups”
 I'd go further and suggest that most people will have too much data for desktop backups.

Comments from Brian Read:
My "home" server (6.0.1) has about 80Gb of data in ibays.  My plan is to
build a new server on SME7, then "break" the raid mirror on the old
server, and move one of the hard discs to the new server.  I will then
boot from the SME6 drive, and unlink the "files" directory from the ibay
(and link it somewhere else - perhaps under /opt or somewhere.  Then
I'll do a desktop backup.  Re-boot from the SME7 drive(s), and restore
the desktop backup.  Then I'll copy the Ibay data from the old Hd (I'll
need to mount it).  Et Voila!

Any better ideas or variations?

Q: Brian, I suspect that your desktop backup will not include any ibays. So you will have to re-create all ibays before restoring the data from your mounted drive, correct?

A: no, I am expecting the ibays structure to be part of the desktop backup, but with no data (I mean't the "files" dir under the ibay)

Comment from Gordon Rowell:

Brian is correct.

Comment from Filippo Carletti:
“Any better ideas or variations?”
Use the new restore from disk that Charlie recently added to e-smith-backup.
See here:
http://bugs.contribs.org/show_bug.cgi?id=615

(Edited by Gordon Rowell to remove the "from desktop" part of the title)
(unsticky, is in the FAQ)
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

Offline brianr

  • *
  • 988
  • +2/-0
Upgrading vs new install + restore
« Reply #1 on: February 27, 2006, 02:05:43 PM »
I have now tried Charlie's disc restore, but it doesn't seem to have worked - how can i check that anything has been copied across?

I had to change tape-restore to "restore-tape" and it certainly hung around for a while afterwards, with a fair amount of disc activity.  But after the  re-boot, the password of the "new" server is still different to the one in the "old" system, and the ibays have not been created.  

The old system was 6.0.1 up to date with all updates.  I can access the old HD through the mount point and see the stucture and files etc.
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Upgrading vs new install + restore
« Reply #2 on: February 27, 2006, 04:19:08 PM »
Quote from: "brianr"
I have now tried Charlie's disc restore, but it doesn't seem to have worked - how can i check that anything has been copied across?

I had to change tape-restore to "restore-tape" and it certainly hung around for a while afterwards, with a fair amount of disc activity.  But after the  re-boot, the password of the "new" server is still different to the one in the "old" system, and the ibays have not been created.  


You'll need to have e-smith-backup-1.13.4-09 installed before you attempt restore from disk.
The version I originally released in 1.13.4-07 had some debugging code still in, and restored the files to /tmp/test.restore.

Alternatively to upgrading, you could edit /etc/e-smith/actions/restore-from-disk and change /tmp/test.restore to / before attempting the restore. Unless you are patient, and a quick reader, you might also want to change "-duvmp" to "-dump".

Offline brianr

  • *
  • 988
  • +2/-0
Upgrading vs new install + restore
« Reply #3 on: February 27, 2006, 06:18:00 PM »
Quote from: "CharlieBrady"
The version I originally released in 1.13.4-07 had some debugging code still in, and restored the files to /tmp/test.restore.


Ok, so I had a look at /tmp/test.restore and can see some restored data, but it seems to be missing my "other" ibays and other users (other than the users already created in the SME7 version).  Not sure where it has come from at all, maybe from the current Hd, not the old one?

I have editted the file as you suggest and am trying again...

Ok, still no real change, the restore-tape run happened far too quickly, and with no output at all.  No change in the password, etc.

Do you want me to open a bug report for this?
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline brianr

  • *
  • 988
  • +2/-0
Upgrading vs new install + restore
« Reply #4 on: March 07, 2006, 12:45:46 PM »
I have copied this to the original bug tracker entry, see:

http://bugs.contribs.org/show_bug.cgi?id=615
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline brianr

  • *
  • 988
  • +2/-0
Upgrading vs new install + restore
« Reply #5 on: March 20, 2006, 12:17:37 PM »
..and the bug seems to have been resolved.

However an alternative approach is also here:

http://forums.contribs.org/index.php?topic=31159.0
Brian j Read
(retired, for a second time, still got 2 installations though)
The instrument I am playing is my favourite Melodeon.
.........

Offline chris burnat

  • ****
  • 1,135
  • +2/-0
    • http://www.burnat.com
Upgrading vs new install + restore
« Reply #6 on: March 31, 2006, 01:02:49 PM »
I have finally updated a couple of production systems using Gordon Rowell's recipe:
- Backup
- Install 7.0
- Restore
- Disable all custom templates
[Refer first post for detailed recipe]

The old systems consisted of version 6 to which had been added at least 15 contribs.  The total size of the data was relatively small, around 10GB for each server. First of all, I made four separate backup jobs using Dungog-backup, the tgz files were stored on a Centos4 workstation ( on the road, this could be done over a laptop).  The first backup dealt with config files and users + a couple of small ibays.  The other three dealt with larger ibays not exceeding 2GB (compressed), thanks to good management and a bit of luck.  The next stage consisted in the removal the old drives from the server, fitting two new 80GB IDE and installing sme7RC1 from CD. I decided to buy a couple of new drives to start with, the advantage being, I still could revert to the old system if things went wrong. Syncing of the RAID array took a couple of hours.  Tested the system - all OK.  The next step was to restore the backups, starting with config files and users.  I then dealt with the custom templates (+ reconfiguration)  before reinstalling a number of contribs, mostly from Dungog, using yum.  And Big Hammer again (reconfiguration), just to be on the safe side... I estimate that the whole job took around five hours work, two of them waiting for the syncing to complete - just to be on the safe side.

Both of these systems have now been online for a week, dealing with 300-500 emails a day + serving a number of websites.  One of them is my own box, the other is also acting as a file server for 3-4 people in a busy office.  Not a problems. Great result, and another 15 or so boxes to go!

In conclusion.  New install, backup and restore for small systems works fine for me. A complete migration to sme7 can be performed in around 5 hours, and possibly less, if one is prepared to take the risk of rebooting during the syncing process of the RAID array. Bonus: you end up with a "clean" install, new drives, all your data, and a couple of spare drives which can be recycled for the next job if on a budget.  Having a system 7 on line for a few days is like driving a Bentley. There is no way back.

The next step is to deal with large restore, say 200GB of data... It would be great hearing about first hand experience/confirmation on this topic.  Anyone?
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

Offline raem

  • *
  • 3,972
  • +4/-0
Upgrading vs new install + restore
« Reply #7 on: April 01, 2006, 05:50:19 AM »
burnat

> First of all, I made four separate backup jobs.....  
> The next stage consisted in the removal the old drives from the server..

If you are removing the old drives anyway, then you might want to save yourself some time and use the Restore from disk feature in sme7. You can restore from the old hard disk (mounted) to your newly installed sme7 OS, without needing to go through the backup process (as I understand it).
I'm not saying not to make backups, but you may not need to depending on what you do with the old disks and whether you need to keep a copy of the old server (as a backup file). If everything restores to the new server OK and runs fine for a month, then you may be happy to trash/reuse the old drive, as all data is on the new server anyway.
See
http://bugs.contribs.org/show_bug.cgi?id=615
...

Offline chris burnat

  • ****
  • 1,135
  • +2/-0
    • http://www.burnat.com
Upgrading vs new install + restore
« Reply #8 on: April 01, 2006, 09:04:58 AM »
Ray, thanks, you hit it on the nail.  If it works, this is the way to go.  This is the next port of call - meaning a fair amount of testing before doing it on other people's  live systems... It would be great hearing from someone who has actually done it, and confirmed that all is OK over a period of time.
- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org)  - check: http://wiki.contribs.org/Bugzilla_Help .  Thanks.

Offline mdo

  • *
  • 355
  • +0/-0
Upgrading vs new install + restore
« Reply #9 on: April 16, 2006, 02:11:50 AM »
burnat

Doing it this way
- Backup
- Install 7.0
- Restore
- Disable all custom templates

we came across the following problem:

Network printers not working:

Although all Samba printer definitions are restored, lpd directories are missing! The following command will show which printers are wrong = which folders are missing:

checkpc

Example output:

Warning -  bad directory - //var/spool/lpd/brother1240
Warning -   Printer_DYN 'brother1240' spool dir '/var/spool/lpd/brother1240' needs fixing
Warning -  bad directory - //var/spool/lpd/kxp32001
Warning -   Printer_DYN 'kxp32001' spool dir '/var/spool/lpd/kxp32001' needs fixing

The manual fix at this stage is to create manually the folders and to change ownership accordingly:

mkdir /var/spool/lpd/brother1240
mkdir /var/spool/lpd/kxp32001
chown lp:lp /var/spool/lpd/*
chmod go-rx /var/spool/lpd/*

I would then restart lpd and Samba daemons (or better: the whole server).

Michael
...

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Upgrading vs new install + restore
« Reply #10 on: April 19, 2006, 05:22:36 PM »
Quote from: "mdo"

Network printers not working:

Although all Samba printer definitions are restored, lpd directories are missing!


Please report problems only via the bug tracker, not via this gossip column :-).

Offline mdo

  • *
  • 355
  • +0/-0
Upgrading vs new install + restore
« Reply #11 on: April 19, 2006, 09:04:26 PM »
Apologies for not reporting it in the bug tracker. I did not realise it to be a bug because the idea of this thread seemed to me to echange some experiences about an "upgrade procedure" which is not standard yet (?).

Anyway, the bug tracker entry is here (http://bugs.contribs.org/show_bug.cgi?id=1280) and the currently suggested, manual workaround (checkpc -f) sounds much easier then my manual creation and change of ownership.
Michael
...

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Upgrading vs new install + restore
« Reply #12 on: April 19, 2006, 09:33:10 PM »
Quote from: "mdo"
Apologies for not reporting it in the bug tracker. I did not realise it to be a bug because the idea of this thread seemed to me to echange some experiences about an "upgrade procedure" which is not standard yet (?).


I see nothing non-standard in your procedure:

Code: [Select]

- Backup
- Install 7.0
- Restore
- Disable all custom templates


SME server should "just work". If it doesn't, please report via the bug tracker. Thanks.

Offline CharlieBrady

  • *
  • 6,918
  • +3/-0
Upgrading vs new install + restore
« Reply #13 on: April 19, 2006, 09:35:08 PM »
Quote from: "mdo"

... the currently suggested, manual workaround (checkpc -f) sounds much easier then my manual creation and change of ownership.


Fixing the bug in e-smith-LPRng is a better idea.

[Amazingly enough, I think this bug has been around since e-smith 3.x]

tandum

Upgrading vs new install + restore
« Reply #14 on: May 02, 2006, 04:41:30 PM »
I just reinstalled a dual opteron box on 3 x 40gig scsi disks with a 200 gig sata backup using rc2. It took all 3 scsi disks as raid 5 and the sata drives are at my disposale for backup.

SME ROX