Topic: Upgrading vs new install + restore

[chris burnat]

I am reprinting below a summary of exchanges on various lists on this subject.  Many of us are/will be facing this challenge, and a pathway through it honed by the community would be great.  If we come out with a good set of recommendations, then we should host it on the contribs wiki (thanks to Brian Read for the idea).

[Answers below are provided by Gordon Rowell].
Q: Updates from a lower versions are going to bring lots of challenges, like it or not.  
A: They work cleanly for unmodified systems and systems with released updates. The
dev team has now done a huge number of upgrades from 6.x of various flavours
and quite a lot of upgrades of modified systems.

Q: How about recommending new installation of 7, and restore of user data via destop method?  
A: That won't help with custom templates and some contribs still install custom templates.
Also, many people have too much data for desktop backups.

Q: This should work irrespective of whether contribs have been installed on the previous version?
A: No, but this would:

- Backup
- Install 7.0
- Restore
- Disable all custom templates:

cd /etc/e-smith
mv templates-custom templates-custom.old
mkdir templates-custom

mv templates-user-custom templates-user-custom.old
mkdir templates-user-custom

signal-event post-upgrade
signal-event reboot

Or this:

- Remove all packages which are not on the 6.0/6.0.1/6.5 CD
- Disable all custom templates:

cd /etc/e-smith
mv templates-custom templates-custom.old
mkdir templates-custom

mv templates-user-custom templates-user-custom.old
mkdir templates-user-custom

- Upgrade from CD
Carefully reinstall SME7 versions of the contribs, if they exist.

Comments by Steve Bush:
Great info.....It'll come in handy when I roll out the SME7 upgrades.
I ran into this problem with a contrib that I missed uninstalling before my
first sme7 upgrade. I installed from scratch, restored from desktop and couldn't figure out that
there were hidden (.qmail) files in the templates-user-custom directory.
If there was a check box in the restore page "don't restore custom apps", it
would probably fix a lot of problems. Thanks again for the information.

Comment by Gordon Rowell:
“Great info.....It'll come in handy when I roll out the SME7 upgrades.”
Raised in bug 812. The dev team are raising lots of things in the "SME Server Documentation" category in Bugzilla.  We need some serious effort on the SME7 FAQ. I would really like to start again - much of the FAQ needs to be rewritten with SME7.
Volunteers - please contact docteam@lists.contribs.org

Comment by Charlie Brady:
“ Also many people have too much data for desktop backups”
 I'd go further and suggest that most people will have too much data for desktop backups.

Comments from Brian Read:
My "home" server (6.0.1) has about 80Gb of data in ibays.  My plan is to
build a new server on SME7, then "break" the raid mirror on the old
server, and move one of the hard discs to the new server.  I will then
boot from the SME6 drive, and unlink the "files" directory from the ibay
(and link it somewhere else - perhaps under /opt or somewhere.  Then
I'll do a desktop backup.  Re-boot from the SME7 drive(s), and restore
the desktop backup.  Then I'll copy the Ibay data from the old Hd (I'll
need to mount it).  Et Voila!

Any better ideas or variations?

Q: Brian, I suspect that your desktop backup will not include any ibays. So you will have to re-create all ibays before restoring the data from your mounted drive, correct?

A: no, I am expecting the ibays structure to be part of the desktop backup, but with no data (I mean't the "files" dir under the ibay)

Comment from Gordon Rowell:
Brian is correct.

Comment from Filippo Carletti:
“Any better ideas or variations?”
Use the new restore from disk that Charlie recently added to e-smith-backup.
See here:
http://bugs.contribs.org/show_bug.cgi?id=615

(Edited by Gordon Rowell to remove the "from desktop" part of the title)
(unsticky, is in the FAQ)

[brianr]

I have now tried Charlie's disc restore, but it doesn't seem to have worked - how can i check that anything has been copied across?

I had to change tape-restore to "restore-tape" and it certainly hung around for a while afterwards, with a fair amount of disc activity.  But after the  re-boot, the password of the "new" server is still different to the one in the "old" system, and the ibays have not been created.  

The old system was 6.0.1 up to date with all updates.  I can access the old HD through the mount point and see the stucture and files etc.

[CharlieBrady]

I have now tried Charlie's disc restore, but it doesn't seem to have worked - how can i check that anything has been copied across?

I had to change tape-restore to "restore-tape" and it certainly hung around for a while afterwards, with a fair amount of disc activity.  But after the  re-boot, the password of the "new" server is still different to the one in the "old" system, and the ibays have not been created.  

You'll need to have e-smith-backup-1.13.4-09 installed before you attempt restore from disk.
The version I originally released in 1.13.4-07 had some debugging code still in, and restored the files to /tmp/test.restore.

Alternatively to upgrading, you could edit /etc/e-smith/actions/restore-from-disk and change /tmp/test.restore to / before attempting the restore. Unless you are patient, and a quick reader, you might also want to change "-duvmp" to "-dump".

[brianr]

The version I originally released in 1.13.4-07 had some debugging code still in, and restored the files to /tmp/test.restore.

Ok, so I had a look at /tmp/test.restore and can see some restored data, but it seems to be missing my "other" ibays and other users (other than the users already created in the SME7 version).  Not sure where it has come from at all, maybe from the current Hd, not the old one?

I have editted the file as you suggest and am trying again...

Ok, still no real change, the restore-tape run happened far too quickly, and with no output at all.  No change in the password, etc.

Do you want me to open a bug report for this?

[brianr]

I have copied this to the original bug tracker entry, see:

http://bugs.contribs.org/show_bug.cgi?id=615

[brianr]

..and the bug seems to have been resolved.

However an alternative approach is also here:

http://forums.contribs.org/index.php?topic=31159.0

[chris burnat]

I have finally updated a couple of production systems using Gordon Rowell's recipe:
- Backup
- Install 7.0
- Restore
- Disable all custom templates
[Refer first post for detailed recipe]

The old systems consisted of version 6 to which had been added at least 15 contribs.  The total size of the data was relatively small, around 10GB for each server. First of all, I made four separate backup jobs using Dungog-backup, the tgz files were stored on a Centos4 workstation ( on the road, this could be done over a laptop).  The first backup dealt with config files and users + a couple of small ibays.  The other three dealt with larger ibays not exceeding 2GB (compressed), thanks to good management and a bit of luck.  The next stage consisted in the removal the old drives from the server, fitting two new 80GB IDE and installing sme7RC1 from CD. I decided to buy a couple of new drives to start with, the advantage being, I still could revert to the old system if things went wrong. Syncing of the RAID array took a couple of hours.  Tested the system - all OK.  The next step was to restore the backups, starting with config files and users.  I then dealt with the custom templates (+ reconfiguration)  before reinstalling a number of contribs, mostly from Dungog, using yum.  And Big Hammer again (reconfiguration), just to be on the safe side... I estimate that the whole job took around five hours work, two of them waiting for the syncing to complete - just to be on the safe side.

Both of these systems have now been online for a week, dealing with 300-500 emails a day + serving a number of websites.  One of them is my own box, the other is also acting as a file server for 3-4 people in a busy office.  Not a problems. Great result, and another 15 or so boxes to go!

In conclusion.  New install, backup and restore for small systems works fine for me. A complete migration to sme7 can be performed in around 5 hours, and possibly less, if one is prepared to take the risk of rebooting during the syncing process of the RAID array. Bonus: you end up with a "clean" install, new drives, all your data, and a couple of spare drives which can be recycled for the next job if on a budget.  Having a system 7 on line for a few days is like driving a Bentley. There is no way back.

The next step is to deal with large restore, say 200GB of data... It would be great hearing about first hand experience/confirmation on this topic.  Anyone?

[raem]

burnat

> First of all, I made four separate backup jobs.....  
> The next stage consisted in the removal the old drives from the server..

If you are removing the old drives anyway, then you might want to save yourself some time and use the Restore from disk feature in sme7. You can restore from the old hard disk (mounted) to your newly installed sme7 OS, without needing to go through the backup process (as I understand it).
I'm not saying not to make backups, but you may not need to depending on what you do with the old disks and whether you need to keep a copy of the old server (as a backup file). If everything restores to the new server OK and runs fine for a month, then you may be happy to trash/reuse the old drive, as all data is on the new server anyway.
See
http://bugs.contribs.org/show_bug.cgi?id=615

[chris burnat]

Ray, thanks, you hit it on the nail.  If it works, this is the way to go.  This is the next port of call - meaning a fair amount of testing before doing it on other people's  live systems... It would be great hearing from someone who has actually done it, and confirmed that all is OK over a period of time.

[mdo]

burnat

Doing it this way
- Backup
- Install 7.0
- Restore
- Disable all custom templates

we came across the following problem:

Network printers not working:

Although all Samba printer definitions are restored, lpd directories are missing! The following command will show which printers are wrong = which folders are missing:

checkpc

Example output:

Warning -  bad directory - //var/spool/lpd/brother1240
Warning -   Printer_DYN 'brother1240' spool dir '/var/spool/lpd/brother1240' needs fixing
Warning -  bad directory - //var/spool/lpd/kxp32001
Warning -   Printer_DYN 'kxp32001' spool dir '/var/spool/lpd/kxp32001' needs fixing

The manual fix at this stage is to create manually the folders and to change ownership accordingly:

mkdir /var/spool/lpd/brother1240
mkdir /var/spool/lpd/kxp32001
chown lp:lp /var/spool/lpd/*
chmod go-rx /var/spool/lpd/*

I would then restart lpd and Samba daemons (or better: the whole server).

Michael

[CharlieBrady]

Network printers not working:

Although all Samba printer definitions are restored, lpd directories are missing!

Please report problems only via the bug tracker, not via this gossip column .

[mdo]

Apologies for not reporting it in the bug tracker. I did not realise it to be a bug because the idea of this thread seemed to me to echange some experiences about an "upgrade procedure" which is not standard yet (?).

Anyway, the bug tracker entry is here (http://bugs.contribs.org/show_bug.cgi?id=1280) and the currently suggested, manual workaround (checkpc -f) sounds much easier then my manual creation and change of ownership.
Michael

[CharlieBrady]

Apologies for not reporting it in the bug tracker. I did not realise it to be a bug because the idea of this thread seemed to me to echange some experiences about an "upgrade procedure" which is not standard yet (?).

I see nothing non-standard in your procedure:

Code: [Select]


- Backup
- Install 7.0
- Restore
- Disable all custom templates


SME server should "just work". If it doesn't, please report via the bug tracker. Thanks.

[CharlieBrady]

... the currently suggested, manual workaround (checkpc -f) sounds much easier then my manual creation and change of ownership.

Fixing the bug in e-smith-LPRng is a better idea.

[Amazingly enough, I think this bug has been around since e-smith 3.x]

[tandum]

I just reinstalled a dual opteron box on 3 x 40gig scsi disks with a 200 gig sata backup using rc2. It took all 3 scsi disks as raid 5 and the sata drives are at my disposale for backup.

SME ROX

[crazybob]

I am going to convert a server to SME7.0RC2 this weekend, and want to make sure I have the proper steps.

I have tried to follow the bug report, and I am probably being a little thick, so I want to verify the proper steps.

Starting with a clean install of SME7.0RC2
Add one of the old hard drive as primary slave.
old system system is standard install of SME6.01 with added rpms removed.


1.  cd /
2.  mkdir /mnt/tmp
3.  mount /dev/hdb2 /mnt/tmp
4.  /sbin/e-smith/db configuration setprop backup Program CopyFromDisk status enabled
5.  /sbin/e-smith/signal-event pre-restore
6.  /sbin/e-smith/signal-event restore-tape /mnt/tmp
7.  /sbin/e-smith/signal-event post-upgrade
8.  /sbin/e-smith/signal-event reboot

[raem]

crazybob

Your steps are in accordance with
http://bugs.contribs.org/show_bug.cgi?id=615

It's always good to give something you have never done before a try out in a test situation first, in order to prove your procedures.
If it doesn't work then it should not be a problem as you still have the original disk.

Let us know how it goes as I 'm sure there are plenty of us out there who will make use of this feature as sme7 gets implemented on production sites currently running sme 6.x.

[crazybob]

Thanks Ray,

Will do.  I am hoping Charlie, Lloyd, or Grodon will take a look at it also. It looks like they have done a lot testing with it.

[crazybob]

It works great. Unload the rpms that could cause a problem. Put in a couple of new hard drives, Install SME7.0rc2, and went through the steps above. Only problem was identifing the proper drive, but that was not a big deal. 60Gb of data in about an hour or so.

Thanks to Charlie, Lloyd, and Gordon for making this so easy


Bob

[robw]

I have a 5.5 server which has served long and faithfully. I now want to upgrade and am looking at installing 7.0RC2 on new hardware and then restoring a backup of the 5.5 server onto it. Is this OK/possible?

[crazybob]

check out
http://forums.contribs.org/index.php?topic=30797.0

Bob

[robw]

Thanks for that Bob. This link talks about an upgrade process. I am looking at building a new 7.0RC2 server and restoring the 5.5 backup onto it. Has anyone had experience with this?

[chris burnat]

I believe that you need to upgrade from 5.5 to 5.6 first of all, then do a backup.  Use this backup to restore your data on the fresh install of vs7.  I have done a couple of these, but went all the way from 5.5 to 5.6, then 5.6 to 6.0, dfone a backup, then restored along the lines discussed in this thread. Had no problems, just time consuming.

[robw]

Hi Burnat,

many thanks for the advice which I will, of course, take. My only problem now is where can I get a copy of 5.6 to do the initial upgrade?

Again, many thanks,

Rob

[brianr]

>My only problem now is where can I get a copy of 5.6 to do the initial upgrade?

email me  - I have a 5.6 iso, plus the update rpms

[chris burnat]

Please report problems only via the bug tracker, not via this gossip column .

Gooday Charlie, not sure what you mean here, would you kindly elaborate?
Regards, chris.

[kruhm]

Did a fresh install of 7.0rc2 (1 drive on hda)
Removed a RAIDed drive from V6 and installed on the second channel

During boot it recognizes the second drive as hdc.

Then errors out:

ERROR: /bin/lvm exited abnormally! (pid 194)
Creating root device
Mounting root filesystem
mount: error 6 mounting ext3
mount: error 2 mounting none
Switching to new root
switchroot: mount failed: 22
umount /initrd/dev failed: 2
Kernel panic - not syncing: Attempted to kill init!

[chris burnat]

Did a fresh install of 7.0rc2 (1 drive on hda)
ERROR: /bin/lvm exited abnormally! (pid 194)
!

Kruhm, it may be a good idea to report this problem to the bugtracker, you can access it from the menu on the left of the screen {Bug Tracker].  The Devs may not have read this post, it may assist them in improving things.  No harm done even if it is not a bug.

[kruhm]

Just as a follow up:

Relevant bug is here:
http://bugs.contribs.org/show_bug.cgi?id=961

I had to:
-remove a drive from the old system
-install in new system on hda
-boot off knoppix
-change all partitions type to 'linux' for hda

Then I could boot off the new install and transfer all the info by following the instructions above.

However it doesn't seem to be any different than just simply upgrading using the ugrade feature when booting off the CD.

Also, for people trying to understand LVM, I put a lvm exercise here:
http://no.longer.valid/phpwiki/index.php/lvm
(if you see errors, just edit the doc)

[kruhm]

burnat (or anyone else), can you take a look to see if this will remove all contribs: http://forums.contribs.org/index.php?topic=32381.msg136939#msg136939

[chris burnat]

If you are removing the old drives anyway, then you might want to save yourself some time and use the Restore from disk feature in sme7. You can restore from the old hard disk (mounted) to your newly installed sme7 OS, without needing to go through the backup process (as I understand it).  http://bugs.contribs.org/show_bug.cgi?id=615

Tested this method today successfully, it was painless notwithstanding a number of contribs installed on the old sme6 server. Moved around 50GB of data in 45 minutes, and found users, ibays etc intact on the new box.  Reinstalled contribs and all seems fine.  I followed a howto found on contribs a while back, unfortunately I forgot where.  My thanks to whoever put this together, here it is:

On the new machine do the following:
1. Fresh install 7.0 {was 7.03)
2. Shut down.
3. Install old 6.0 drive and start machine.
4. cd /
5. mkdir /mnt/tmp
6. mount /dev/hdb2 /mnt/tmp (adjust "hdb2" according to where the 6.0 drive is installed).
7. /sbin/e-smith/db configuration setprop backup Program CopyFromDisk status enabled
8. /sbin/e-smith/signal-event pre-restore
9. /sbin/e-smith/signal-event restore-tape /mnt/tmp
(note: this step may take a long time depending on how much data is being restored)
10. cd /etc/e-smith
11. mv templates-custom templates-custom.old
12. mkdir templates-custom
13. mv templates-user-custom templates-user-custom.old
14. mkdir templates-user-custom
15. /sbin/e-smith/signal-event post-upgrade
16. /sbin/e-smith/signal-event reboot
17. Shutdown and remove old VS6 drive

And it works!  ('')

Additional information (edited on 4/8/06):  When adding the old version6 drive to a full version7 system (Raid1), the old IDE drive may take precedence in the boot order, especially if the new system is built with SATA drives.  Check:
http://forums.contribs.org/index.php?topic=33093.0

Also (edited 4/9/06):
"Prior to removing the old drive, remember to run a pre-backup to dump mysql db"  Refer Ray Mitchell's post further down or check:
http://bugs.contribs.org/show_bug.cgi?id=615

[raem]

burnat

>...And it works!

Good one, but I'm sure Charlie & Gordon intended it to work !

[perelandra]

Hi folks,

yesterday I did a successful upgrade of my 6.01 to 7.0!! And it works "like a charm" (as there is a saying in english - iirc... As you may see in the signature, it's a quite simple and average machine behind an ADSL-Router. I use it for our home network and for the hosting of several smaller and bigger websites (and since yesterday as a SlimServer ...

How did I upgrade?

Kruhm's thread here --> http://forums.contribs.org/index.php?topic=32381.0 "inspired" me and because I'm no geek I didn't fiddle out the comparison of original and additionally installed rpms. It was more a "trial'n'error" thing. So here I went:

1. Diskimage to new and bigger harddrive
2. Resize partition on new hardisk to use unallocated diskspace
3. Keep the old disk as a backup.
4. Remove BY HAND  and  with --nodeps all additional rpms which did'nt appear in the install log of the first 6.01 installation (which I found in the root folder)... that took me some time and about two beers  
5. Rename the custom-template-folders to *.bak
6. Reboot from original 6.01-CD and do an upgrade with it to reinstall all the missing --nodeps-RPMS.
7. Test it with a reboot to see if it works: works with some small exceptions ("qmail-authentication-mod-thing" e.g.)
8. Put in 7.0 CD - perform a CD check - and do the upgrade
9. Reboot and enjoy! 3rd beer!

Maybe some of you laugh for this way is not very elegant. But as we say in Germany (and maybe also elsewhere?): "Alle Wege führen nach Rom!"

I'm so glad I did it! This is such a good work! WELL DONE!!!!

What I really like are some additional contribs and how they work:
- backup2
- remoteuseraccess
- awstats
- slimserver
- eaccelerator
- config apache error pages
- new phpmyadmin
- ...

and of course some of the basic functionalities:
- yum
- software installation
- antivirus and spam
- smtp-authentication @ provider
- and all the others I didn't discover yet!!!

Though it seems, that booting takes a bit longer and the OS needs better hardware... But for my kind of server it still seems fast enough!

I really love it! You are saving me a lot of nerves and time and also some money. Time to donate I think!

Hope I didn't forget anything! Anyway: I can sleep well now! Safe and sound...!

Cheers,
Johannes.

[ltc6netspec]

Here is the qmail log:

2006-08-04 10:24:27.350359500 info msg 4657290: bytes 2078 from <xxxxxx@xxxxxx.xxx> qp 29749 uid 453
2006-08-04 10:24:27.366464500 starting delivery 16637: msg 4657290 to remote xxxxxx@verizon.net
2006-08-04 10:24:27.366468500 status: local 0/10 remote 1/10
2006-08-04 10:24:27.742314500 delivery 16637: failure: Connected_to_206.46.232.11_but_sender_was_rejected./Remote_host_said:_550_You_are_not_allowed_to_send_mail:sv3pub.verizon.net/
2006-08-04 10:24:27.813595500 status: local 0/10 remote 0/10
2006-08-04 10:24:27.911743500 bounce msg 4657290 qp 29751
2006-08-04 10:24:27.911746500 end msg 4657290
2

[at9]

Upgrade from v. 6.0

- Backup  - done
- Install 7.0 - done
- Restore...  - and here i have trouble - choose tgz destination and click submit - next page is epmty...standard error HTTP 404.
What is wrong?

[raem]

at9

> - Restore...  - and here i have trouble - choose tgz destination and click > submit - next page is epmty...standard error HTTP 404.

Try a different browser eg Mozilla or Netscape.
Did you verify the backup file to prove it was OK immediately after doing the backup ?
Is the backup file larger than 2Gb, if so you may have trouble restoring from it.
How was the original backup done, using backup to desktop ?

[at9]

IE - error; OPERA -  not reply (stop work);
BUT:
I didn't verify the backup file..
and
My backup have 2,8Gb.

Now i will try moziila and verify file...

[raem]

at9

> My backup have 2,8Gb.

As far as I'm aware, there is a limit of 2Gb if you want to successfully restore files.
That's why the backup to workstation contrib is so poplular, you can limit the backup file size to 2Gb parts.

I suggest you redo the backup using backup2ws/backup2 contrib, assuming you still have the original server.
Alternatively remove some data and redo the backup to desktop to get the resultant file size below 2Gb.

[at9]

Well, i deleted some mail (copy to other place) and sudennly my arch had 650MB  - four step done and WORK IT. Now i will test www with DB (all change I did on the second machine) I hope will be ok.
So big thanks for all here especially for RAY

[ltc6netspec]

I have rebuilt a SME 7 3 times using the 1st method in this post.  Using the current backup from a SME 6 running for 3 years.  1400 accounts, less than 2 Gigs backup data.

The thing is I have never received the message on the bottom of the restore page saying that the "Restore is complete".   The last time I used console and TOP to watch for when the machine was done.  I've even let the machine sit for 18 hours while waiting for the "Restore is complete".

The question is will this make a difference in the configuration of all files and configuration data.  If it does, then what should I do?

At a lost now.  Because this machine has been zombie mass attacked 2 times.

Any suggestions appericated.

Ps.  I also have a SME 7 w/ 950 accounts that worked with the 1st method and " Restore is completed" showed up.

[raem]

ltc6netspec

> I've even let the machine sit for 18 hours while waiting for the "Restore > is complete". The question is will this make a difference in the
> configuration of all files and configuration data.

If restore is not complete then all files have not been restored !

Did you Verify the backup file to check it is OK immediately after the backup ?

[ltc6netspec]

Thanks for replying.

Unfortunely no  

I had restored twice from two different backups.   I now no longer have the old SME 6.01, just before this second zombie attack everything seemed to be going so well.

  By the way, the SME 7 server with spamassin, qpsmtpd, and clamav held up trying, but the internet pipe was bogged down (10,000 messages a minute at peak).

Should I try to create a current backup of the 7.  Then do a verify and fresh install w/restore.  Or is there a way to generate a "registry cleanup" like windows products.

[asjunk]

I used the method in Burnat's post, above, to facilitate my upgrade to 7.0.  Everything went well except that the mysql databases apparently didn't make the transfer (which is somewhat problematic, since there's a lot of data in those databases).   I read through the bugzilla entry 615, but couldn't make heads or tails out of it with relation to my problem.

Anyone have any suggestions on the best way to get the mysql databases from one hard drive to the other?  I must admit I'm a linux moron, and haven't even been able to figure out exactly where the hell the databases get stored.  I've tried using the phpMyAdmin contrib to export the databases, but that doesn't seem to work (rather than allowing me to save it as a file, it displays a text file; the text file, when pasted into a text document, doesn't seem to import properly although mysql doesn't give me any errors).

Thanks!

[raem]

asjunk

>...the mysql databases apparently didn't make the transfer...
>.. I read through the bugzilla entry 615

Well the last 2 entries refer to your issue.
To quote from
http://bugs.contribs.org/show_bug.cgi?id=615

"Prior to removing the old drive, remember to run a pre-backup to dump mysql db"

As a workaround, you can copy the dbs from
/var/lib/mysql

You should stop mysqld first though, as open dbs wil not be copied.

/etc/init.d/mysqld stop

You may need to recreate correct permissions after the copy.
cd /var/lib
chown -R mysql:mysql mysql

[at9]

Hi. First update to 7.0 was ok but my other arch (copy handly) was wrong and i must do it again.
The upgrate i did on the other server and IP was other too. When i change IP on two servers mysqladmin say that can't connect with DB.
I write in: mysqladmin -u root password "tralalala" and work ok. (each server).
I did new upgrade and now DON'T work...
"There seems to be a problem with the MySQL server, sorry for the inconvenience. We should be back shortly." - message PHP NUKE:)
What can i do??

[at9]

I take first backup and now all ok. Propably my next backup was wrong. Now all is ok. I don[t understand that but...i hope my work is finish:)

[netspirit]

Hi at9,

Did you find a solution for MySQL problems during upgrade from 6.0.1 to 7.0?  I'm facing same problem.
Thanks.

[chris burnat]

Edited, please see next entry.

[chris burnat]

Did you find a solution for MySQL problems during upgrade from 6.0.1 to 7.0? I'm facing same problem.

Prior to removing the old drive, remember to run a pre-backup to dump mysql db" Refer Ray Mitchell's post  22 Aug 2006 19:35    Post subject: Re: Restoring MySQL. further up this thread.  Also check:
http://bugs.contribs.org/show_bug.cgi?id=615

I think what is required is:

Before and after the restore you will need to run some SME commands
signal-event pre-restore
-> restore files
signal-event post-upgrade
signal-event reboot

Ray, can you confirm?

Hope it helps.

[raem]

burnat & others

Edited to include pre-backup event on old sme6 before shutdown - 24/9/06

> Before and after the restore you will need to run some SME commands
> signal-event pre-restore
> Ray, can you confirm?

Here is a mini howto that someone wrote which I luckily copied & saved. This post & others went missing in the recent "disappearing posts" event.

Question:
have Installed a clean v7.0 on a new server, and after removing
custom templates and add ins to the best of my ability, used the
copyfromdisk method to copy the OLD sme 6.01 data to the new server.

Answer:
If I understand you correctly, you may not have followed the recipe
correctly when upgrading your server. You should cleanup your custom
templates (templates-custom and templates-user-custom to be precise) AFTER
restoring your data from the old vs6 drive. Check steps 10 to 14 below, and
follow them stricly. Removing custom-templates before restore of old data
from your VS6 drive achieves nothing - just check relevant directories and
see what is there, probably some templates from the old VS6 drive, and
possibly your problem. Hope it helps.


1. On the old sme6 server prior to shutting down do
/sbin/e-smith/signal-event pre-backup
then shutdown & remove the drive
2. On the new machine do a fresh install of sme7.0 OS
3. Shut down & install old 6.0 drive and start machine.
4. cd /
5. mkdir /mnt/tmp
6. mount /dev/hdb2 /mnt/tmp (adjust "hdb2" according to where the 6.0 drive
is installed).
7. /sbin/e-smith/db configuration setprop backup Program CopyFromDisk status
enabled
8. /sbin/e-smith/signal-event pre-restore
9. /sbin/e-smith/signal-event restore-tape /mnt/tmp
(note: this step may take a long time depending on how much data is being
restored)
10. cd /etc/e-smith
11. mv templates-custom templates-custom.old
12. mkdir templates-custom
13. mv templates-user-custom templates-user-custom.old
14. mkdir templates-user-custom
15. /sbin/e-smith/signal-event post-upgrade
16. /sbin/e-smith/signal-event reboot
17. Shutdown and remove old sme6 drive

[raem]

To all

One other issue that is probably worth including in this thread, is that of the different partitions in sme7 compared to sme6.
An upgrade install from CD on a existing sme6 will not change the partitions to the new sme7 format (sme6 has 3 partitions, sme7 has 2 partitions). There are other forum posts & bug tracker entries that refer to this eg
http://bugs.contribs.org/show_bug.cgi?id=572

The effect of this is that an upgraded sme6 retains the old partitions and the "very nice" feature in the admin console panel to add a drive to a broken software RAID1 array WILL NOT work.

I believe the developers plan to fix this in a later release of sme7 (7.1 ?) so that the partitions are automatically converted during an upgrade, but no guarantees here.

The workaround (particularly if you want to avail of this new feature to easily rebuild a software RAID1 array in sme7), is to do a fresh install of sme7 either with one or two disks, as software RAID1 will be configured by default (degraded mode for one disk installs).
Then you can restore from your sme6 backup using one of the methods described, or just even manually reconfigure the new server from scratch.

[raem]

burnat & others

> Prior to removing the old drive, remember to run a pre-backup to dump mysql db

Yes, I think the step people are overlooking to do is:

On the old sme6 server prior to shutting down do:
/sbin/e-smith/signal-event pre-backup

then shutdown & remove the drive and transfer it to the newly built & installed sme7 server
Then follow the instructions for mounting the drive & restoring etc etc.

I have edited the steps posted earlier

[Smitro]

Hi all,

I'm going to upgrade my server this Friday. It's currently holding about 50gb of data. SME 6.5 serving Websites / Email / and a couple of SMB shares.

It seems that no one has suggested this method it seemed like the easiest one to me... but there could be some major faults in it.

- Backup using Backup2ws (911 Disaster Recovery)
- Wipe Drive
- Install SME 7.0
- Install Backup2ws
- Restore backup from 6.5 (listed in first step)

Then I'm not sure if I still have to rename the custom templates folder. (I'm not sure if it's in the 911 Disaster Recovery backup)

Does this sound like it will work? Anyone tried it? Is the folder stucture much different in SME 7?

[raem]

Smitro

> It seems that no one has suggested this method...backup2ws

It is a long established way of backup & restore ever since Darrell released the contrib (2 or 3 years ago now).


> It's currently holding about 50gb of data.

Keep in mind it will take you many hours to back that up and many more hours to restore that amount of data, enjoy the wait.


> - Backup using Backup2ws (911 Disaster Recovery)
> - Wipe Drive
> - Install SME 7.0
> - Install Backup2ws
> - Restore backup from 6.5 (listed in first step)

> Then I'm not sure if I still have to rename the custom templates folder. > (I'm not sure if it's in the 911 Disaster Recovery backup)

The custom templates are in the 911 backup as is everything else (data & config) that is changeable on the sme6 server.
You are going to have the same issues, you will need to remove all the custom templates and configuration changes that any sme6 contribs have made, as these will clash with sme7.

You also have the unreversable situation of having completely wiped your sme6 before establishing that your sme7 will be working correctly after the restore. You don't have a fallback if things go wrong, which they likely will with the procedure you have outlined.

The safer way would be to uninstall all contribs & add ons & remove custom templates and get your sme6.x back to standard.

Then just do a upgrade to sme7 directly, or if you want a "clean" system do a backup of sme6.x, and restore on new sme7.

If you can find a spare hard drive then it may be easier to do the restore from disk method outlined earlier in this thread (4 posts back) as it effectively removes the backup step & saves a few hours.

Whichever method you use you cannot avoid removing the contribs & data & config & templates that will clash with sme7.

[Smitro]

RayMitchell, as always you you've been very helpful, and I see where you are coming from.

At the moment there is a 120gb drive in the machine. I have a spare 200gb drive. But I don't want to make the 200gb drive the permanent drive. (I know part of me is sayinging I'm going to regret this). I'd like to stay with the 120gb because it's proven to be more than enough over the last year and a bit that I've been running the server. The reason I purchaced a was so I could hold a couple of backups on it.

My question is... can I use the 200gb drive to aid this process.

I think there is a program that comes with knoppix that will alow me to image the drive. So I think I'll image across everything onto the 200gb drive.. make sure its 100% then do as you said, wipe the 120gb drive, then install SME 7.0 and follow on with the process.

Is there an easier way of doing this? and is there any better programs for duplicating drives that you would recommend?

[Smitro]

Hey, I thought I better let you know my findings.

I had a SME 6.5 server that I just upgraded. I used a program called "Roadkil's Raw Copy" (http://www.roadkil.net/diskutils.html) to duplicate the 2 disks (this process took 4 hours). So I had 2 exact copies of my origonal server. I booted the second drive made sure it worked, got it ready for copying from (as mentioned in step one above). I wiped the hard drive that I wanted to use for my server. and followed your instructions exact to the end. And it worked a treat. Now I'm just going through and trying to find all the new version of the contribs I used in the past.

Thanks once again.

[byte]

> It's currently holding about 50gb of data.

Keep in mind it will take you many hours to back that up and many more hours to restore that amount of data, enjoy the wait.

Think that's bad I've got to do an upgrade shortly with about 200GB of data  

[dmay]

I just upgraded a server from SME 5.5 to SME 7. Here are the basic steps I followed:

1) make a current backup

2) list all installed rpms:

# rpm -qa | sort > /root/rpmlist

3) view the above list and remove all contrib rpms:

# rpm -e blah

4) rename templates-custom and templates-user-custom

 # mv /etc/e-smith/templates-custom /etc/e-smith/templates-custom.55
 # mv /etc/e-smith/templates-user-custom /etc/e-smith/templates-user-custom.55

5) upgrade to SME 6.0
6) test and make a current backup

7) upgrade to SME 7.0
8) test and make a current backup

9) add 7.0 contribs

Darrell

[elmarconi]

1. On the old sme6 server prior to shutting down do "/sbin/e-smith/signal-event pre-backup" then shutdown & remove the drive
2. On the new machine do a fresh install of sme7.0 OS
3. Shut down & install old 6.0 drive and start machine.
4. cd /
5. mkdir /mnt/tmp
6. mount /dev/hdb2 /mnt/tmp (adjust "hdb2" according to where the 6.0 drive
is installed).
7. /sbin/e-smith/db configuration setprop backup Program CopyFromDisk status
enabled
8. /sbin/e-smith/signal-event pre-restore
9. /sbin/e-smith/signal-event restore-tape /mnt/tmp
(note: this step may take a long time depending on how much data is being
restored)
10. cd /etc/e-smith
11. mv templates-custom templates-custom.old
12. mkdir templates-custom
13. mv templates-user-custom templates-user-custom.old
14. mkdir templates-user-custom
15. /sbin/e-smith/signal-event post-upgrade
16. /sbin/e-smith/signal-event reboot
17. Shutdown and remove old sme6 drive

Well, I found sort of similar approach: Other than mounting the harddrive in step6 I just mounted the latest rsnapshot.hourly whole old disk off the running server thru NFS. Someting to remember is to export as ro,no_root_squash otherwise things will blow up in your face.
I had forgotten one thing, as to remove the one and only testuser on the new SME7, being me myself and I, which I used for preliminairy testing, resulting in a lot of files with missing uid and groupid. Again relatively easy to fix by applying:
find /home/e-smith/files/ -nouser -print0 | xargs -0 chown memyselfandi
find /home/e-smith/files/ -nogroup -print0 | xargs -0 chgrp memyselfandi

Other than that I did not notice any abnormalities yet. Keep you posted.

[geoff]

burnat & others

1. On the old sme6 server prior to shutting down do
/sbin/e-smith/signal-event pre-backup
then shutdown & remove the drive
2. On the new machine do a fresh install of sme7.0 OS
3. Shut down & install old 6.0 drive and start machine.
4. cd /
5. mkdir /mnt/tmp
6. mount /dev/hdb2 /mnt/tmp (adjust "hdb2" according to where the 6.0 drive
is installed).
7. /sbin/e-smith/db configuration setprop backup Program CopyFromDisk status
enabled
8. /sbin/e-smith/signal-event pre-restore
9. /sbin/e-smith/signal-event restore-tape /mnt/tmp
(note: this step may take a long time depending on how much data is being
restored)
10. cd /etc/e-smith
11. mv templates-custom templates-custom.old
12. mkdir templates-custom
13. mv templates-user-custom templates-user-custom.old
14. mkdir templates-user-custom
15. /sbin/e-smith/signal-event post-upgrade
16. /sbin/e-smith/signal-event reboot
17. Shutdown and remove old sme6 drive

Huge thanks to everyone who worked on this - an excellent solution which worked almost perfectly several times over the last week or so.

"Almost" perfectly because "hdb2" on SME 6.0 is surely swapfile space? (See #6 above)

Once changed to hdb3 this "HowTo" worked flawlessly.

Thanks to all who worked on SME 7 and HowTos like this one for great work.
 

[schuitz]

I believe that you need to upgrade from 5.5 to 5.6 first of all, then do a backup.  Use this backup to restore your data on the fresh install of vs7.  I have done a couple of these, but went all the way from 5.5 to 5.6, then 5.6 to 6.0, dfone a backup, then restored along the lines discussed in this thread. Had no problems, just time consuming.

I just wanted to post for posterity that I was able to follow the procedure described here from 5.6 straight to a fresh install of 7 without much difficulty (once I figured out that / was on partition number 6 of the v5.6 drive).  

However, for whatever reason, the primary ibay did not make it over, and hence the main website.  Investigating.

[schuitz]

I should also mention that something happened to the samba machine accounts for the domain server, ie smbpasswd database is corrupt.  However this may very well be because I added machine accounts manually under 5.6. My mind is a little fuzzy on that.

Anyway, some care is required in fixing this because XP user profiles can be handily blown away if you add the machine to the domain after the upgrade.  Beware!

[chris burnat]

Just in case, check: http://bugs.contribs.org/show_bug.cgi?id=1698
and: http://bugs.contribs.org/show_bug.cgi?id=1327

[schuitz]

Thanks, it sounds more like old tracker bug 1254663, https://sourceforge.net/tracker/index.php?func=detail&aid=1254663&group_id=96750&atid=615772, which appeared to have been fixed by Charlie in 7.0beta1.  Namely, machine accounts listed in smbpasswd are not in /etc/passwd after the restore from disk procedure outlined in this thread.

All of the machine accounts are listed in smbpasswd still.  But the machine has to be re-added to the domain for an entry to appear in /etc/passwd.

I'm a little thick on the purpose of secrets.tdb.  At any rate, if I get a chance I'll restore the old secrets.tdb and see if that resolves it.  However we don't have that many machines and I may just use the brute force approach.

[crazybob]

Can I do something like this to move SME 7.0 Final to a new pair of hard drives?

1. On the old sme7 server prior to shutting down do
/sbin/e-smith/signal-event pre-backup
then shutdown & remove the drive
2. On the new machine do a fresh install of sme7.0 OS
3. Shut down & install one old 7.0 drive and start machine.
4. cd /
5. mkdir /mnt/tmp
6. mount /dev/hdb2 /mnt/tmp (adjust "hdb2" according to where the 7.0 drive
is installed).
7. /sbin/e-smith/db configuration setprop backup Program CopyFromDisk status
enabled
8. /sbin/e-smith/signal-event pre-restore
9. /sbin/e-smith/signal-event restore-tape /mnt/tmp
(note: this step may take a long time depending on how much data is being
restored)
10. cd /etc/e-smith
11. mv templates-custom templates-custom.old
12. mkdir templates-custom
13. mv templates-user-custom templates-user-custom.old
14. mkdir templates-user-custom
15. /sbin/e-smith/signal-event post-upgrade
16. /sbin/e-smith/signal-event reboot
17. Shutdown and remove old sme7 drive

I want to use the existing hardware, just replace the existing hard drives. If I can do the disk to disk copy, it is much faster than the backup and restore.


Thanks

Bob

[kruhm]

Hi Bob,

That looks good. Just be careful of the installed contribs (i had problems with modindex & httpd.conf at one point).

Once your certain they're reinstalled, you can move the custom templates back to their correct location and:

signal-event post-upgrade
signal-event reboot

Let me know if you have any problems

Take care,

[crazybob]

I will give it a shot this weekend, and let you know how it works out.

Thanks

Bob

[mackayr]

I'm trying to migrate my entire server to new hardware (single drive to single drive).  The approach in crazybob's post (two posts ago) seems to be what I'm looking for.  However, since I'm moving from and to the same version (ie. SME 7), is it necessary to remove custom templates?  Also, is it necessary to reinstall contribs or will they be migrated with the restore-tape procedure.  I've had limited success with backup2, most likely due to missed steps on my part, or installing different versions of contribs and this procedure seems to save a lot of time (ie. no "backup" is actually required - merely temporarily attach the old hd to the new server).  Please steer me in the right direction.

Thanks,

Rob

[raem]

mackayr

> ...since I'm moving from and to the same version (ie. SME 7), is it necessary to remove custom templates?  

No

> ...is it necessary to reinstall contribs

Yes

>...will they be migrated with the restore-tape procedure.  

Installed contribs are not normally part of a backup and will therefore not be restored and so will need to be reinstalled.
The config  files associated with installed contribs usually are part of a backup & will be restored.

The concept of a full backup & restore, is that the backup is restored to a clean installation of the sme operating system, which by default does not have any contribs installed. Contribs must be therefore be reinstalled.


> I've had limited success with backup2.

You don't say what the limited success is related to.
If you mean restoring contribs, then the same comments as above apply. Contribs are not usually backed up etc...


The only "method" that will backup & restore your fully configured server (including installed contribs) is a "bare metal" disk clone, using an appropriate disk cloning or disk copying program. Linux has the dd command which will do a complete disk copy (slowly).
There are a number of references in the forums eg Mondo, or alternatively you swap out a disk of a RAID1 software array and use that as a backup.

[stephen noble]

There is a space in the new wiki FAQ for this topic

http://smeserver.sourceforge.net/sme7/manual
http://smeserver.sourceforge.net/sme70/sme70/Manual/FAQ?v=1b0h#h1094-5

It would be nice if these 5 pages could be distilled to the best method(s)

[marsa_matruh]

I did a backup/restore from 6.0 to 7.1. I mostly followed instructions at the beginning of the topic.

- Remove added rpm
- Backup
- Install 7.0 under vmware on my workstation
- Restore
- Disable all custom templates:

cd /etc/e-smith
mv templates-custom templates-custom.old
mkdir templates-custom

mv templates-user-custom templates-user-custom.old
mkdir templates-user-custom

signal-event post-upgrade
signal-event reboot

At this point, I had a small problem. There was no ethernet connection. ifconfig did not shown eth0. I went through the admin console. I did not changed any parameter. At the end, it asked me restart the server, like if something had been changed. I restarted and eth0 was back.

After 24h under vmware, I did :

- Backup from vmware.
- Install 7.0 on my server
- Restore

signal-event post-upgrade
signal-event reboot

Again, through the admin console to get back eth0.

Server is up and running. I am still doing small tunning, reinstalling rpm and thinks like that.

[stephen noble]

Thanks
Changes here http://smeserver.sourceforge.net/sme70/sme70/Manual/FAQ?v=14e3#h1094-3

[byte]

Dont know why this was removed as a sticky as it contained lots of useful information.

Super Mod can you replace back to sticky ?

[stephen noble]

it's replaced by this
http://wiki.contribs.org/SME_Server:Documentation:HowTo:UpgradeDisk

i raised a bug and no one objected
http://bugs.contribs.org/show_bug.cgi?id=2365

[tariqf]

crazybob, your method is more or less sound however the mounting of the old sme disk will not work as the disk is RAID and LVM. If you try and mount it it will fail asking to specify the filesystem type.

If you switch the filesystem on the old drive to ext3 first then it will work ok but am not sure if you can ust use fdisk and force the type to ext3. Seem to remember doing something similar before but can't remember. Will let you know if I get a chance to test this idea.

[crazybob]

It seemed to work quite well going to  7.0 final. I have not had need to try it going to 7.2

Bob

[tariqf]

Hmmm tried it on 7.2 yesterday and image one of the no joy. Oh well I'm going to try the new clonezilla livecd and see if I can just copy once of the drives to a larger one and have it expand the partition table. Will post results.