Topic: In-place upgrade from 6.0-1 to 7.2

[markehle]

Folks -

Our email/web/DNS server here at the library is running 6.0-1 and I would like to upgrade it to the latest version of SME. Since we rely on it pretty heavily for our operation (and it has never failed us - a great tribute to SME!), I don't want to screw it up. Currently, I am backing up using BackupPC from another machine. I am grabbing /home/e-smith/files/users, /home/e-smith/files/ibays, and /etc. What else, besides MYSQL should I be concerned about?

What would be my best bet as far as upgrading? Will 7.2 know what to do if I popped it in the cd drive and told it to upgrade?

Thanks!

Library Mark

[cactus]

What would be my best bet as far as upgrading? Will 7.2 know what to do if I popped it in the cd drive and told it to upgrade?

That will almost certainly fail. I remember there were certain update instructions for upgrading from 6.x to 7.x releases. One of them was to remove all contribs installed on the machine as well as the custom templates in templates-custom and templates-user-custom (http://forums.contribs.org/index.php?topic=38543.msg174737;topicseen#msg174737).

It might be wise to not upgrade your system straight away as there have been some major problems with some people upgrading.

I suggest you make a backup of your server in the current state (just a safety feature I always do before making major changes). After that remove all installed contribs and custom templates and make an other backup, this  backup will be used to restore later on a new fresh install.
Shutdown the server and take out the old hard disk, mount a new clean hard disk and do a clean install of SME Server 7.2 (on this new disk), after that restore the 2nd backup (without the contribs/custom-templates) on this new system.

Make sure you use a backup routine that is implemented in the server as this will take care of dumping and restoring configuration parameters like users as well as MySQL data. Unfortunately I am not sure if the backup options for SME Server 6.0 and SME Server 7.2 (latest release) are the same, but to have all critical stuff a so called minimal backup would include at least the following elements:

Code: [Select]

etc/e-smith/templates-custom
etc/e-smith/templates-user-custom
etc/group
etc/gshadow
etc/passwd
etc/samba/secrets.tdb
etc/samba/smbpasswd
etc/shadow
etc/smbpasswd
etc/ssh
etc/sudoers
home/e-smith
To make sure you also have a valid dump of the MySQL data you can execute the following command before backing up your files:

Code: [Select]

signal-event pre-backupThis command should make a dump of your MySQL data and store it in /home/e-smith/db/mysql/ with a file for each database. If you make a backup with the build-in SME Server backup features (if possible due to file size limits) this should already been done automatically (as well as a restore of these files on your new install). If you do things manually you will probably have to install the dump files yourself as well (and probably need to reset yor root password as well).

Please do not install on your current harddisk as it will be hard to recover from a failed installation and you run a great risk of ending up with a non running server as well. By setting apart the extra hard disk (with the running 6.x system) you have an easy way of restoring a running system. If the update/installation fails you can easily take out the drive with the failed install and replace it with the old drive. Boot up your server and you should have your old copy running again. This way you have a fall back build in, in case thing go haywire.

[markehle]

WOW - thanks for all the info! I never thought of pulling the hard drives out. The server is a Proliant ML370 so pulling them is a snap, and an excuse to get bigger ones.

Thanks again!

Library Mark

[markehle]

While we are talking about it,. on my home server that I reload with great frequency because I use it to play with, i have never been able to make the webmail rules come back once I reload a machine. Where are they stored?

Thanks -

Library Mark

[raem]

markehle

...on my home server that I reload with great frequency.... i have never been able to make the webmail rules come back once I reload a machine. Where are they stored?

Webmail options are stored in mysql databases, so it sounds like you are not dumping the mysql dbs and restoring them.

A simple backup/restore solution you could use is in the wiki, which allows config to be backed up and restored without data in ibays & user data in user folders etc. You can tailor it somewhat as required.
In your case (assuming you install other contribs that use mysql dbs), then as you probably don't want certain mysql data, then it may be best to delete (or move) mysql dbs before doing a backup, that way unwanted mysql stuff is not carried across to your rebuilt server. Look in /var/lib/mysql/.... and delete (or move) the unwanted contrib dbs. You should stop mysql first, and restart it after.

See
http://wiki.contribs.org/Backup_server_config

Another way of approaching this, is to build a clean machine with basic setup eg webmail rules and limited users etc, and back that up using the method suggested. Then go ahead and play and make lots of changes etc. When you are ready to trash all that, just reinstall the sme OS and restore the backup from the clean original server.

Yet another way is to create your original server with minimal config & users eg webmail etc, make a clone of the hard drive using a disk cloning tool, and when you are ready to trash your server just restore the clone image to your drive. You need to use the rght clone tool/software that supports the volume type you use etc.

[CharlieBrady]

That will almost certainly fail.

Really? That's news to me. That would mean that we failed in an important design objective.

I remember there were certain update instructions for upgrading from 6.x to 7.x releases. One of them was to remove all contribs installed on the machine as well as the custom templates in templates-custom and templates-user-custom (http://forums.contribs.org/index.php?topic=38543.msg174737;topicseen#msg174737).

Yes, that's standard recommendation, as is making sure that you have a full current backup.

[byte]

What would be my best bet as far as upgrading? Will 7.2 know what to do if I popped it in the cd drive and told it to upgrade?

As long as you make sure your 6.x version is clean i.e remove/rename custom templates as per other threads in here and remove contribs then you can upgrade from a 6.x to an SME Server 7.2 as I tested this a month or so ago.

But remember "if it doesn't work" then please search our bug tracker as your issue may already be in the list, if not then please open a new ticket. Thanks.

[raem]

markehle

There was a very long thread on 6 > 7 upgrades (6 pages) so read that for possible gotchas depending on your configuration.
See
http://forums.contribs.org/index.php?topic=30745.0

Make sure you read the last two or three pages, but keep in mind that the "restore from disk" method has been summarised in the following wiki article.

The restore from disk method allows you to keep your old system intact (just in case), and saves you many hours creating backups ie it restores directly from your mounted sme6 disk. It's listed in the FAQ.
see
http://wiki.contribs.org/UpgradeDisk


As you say you rely heavily on that server, then you may want to do a test upgrade first, discover and resolve any problems or user oversights, then do a live upgrade. If you use the restore from disk method, then your whole sme6 system is still intact and can be put back into service if you have any major hiccups.

The folders you backup are not really sufficient.
Do a complete backup using say the well proven dar2 contrib (add /opt as an inclusion in the backup job), as dar2 also includes a dump of all mysql dbs. If you accidently deleted the contents of your sme6 disk, then you have a full backup to rely on.

Just adding this note, as you are using sme6 then you would need to use a backup solution that runs OK on that OS, try the well proven backup2 contrib (ie backup2ws or backup2usb), and select a disaster recovery backup which includes the mysql dbs, and add /opt to it.
http://distro.ibiblio.org/pub/linux/distributions/smeserver/contribs/dmay/smeserver/6.x/contrib/backup2/

[markehle]

Man - you guys are the best!

As things would have it, I talked my boss into new hardware (she's going to let me spend up to $6K!) so I think that my game plan will be to load the new server with a naked copy of 7.2, and try to bring things over to it from the old server.

The old server is so hacked that I am not sure that it would upgrade without difficulty, so the safest option for me is not to touch it and replace it when the new server is ready.

Thanks again for all your help!

Library Mark

[raem]

markehle

You don't say how you plan to transfer config data etc, but if you have Windows workstations which have joined the domain (assuming your sme is a PDC), then make sure you bring across the machine accounts & profiles or you will need to run round to every workstation and disjoin & rejoin them from the domain and then setup new profiles for every user.

[markehle]

This machine is not my PDC. In fact, right now I don't have a domain. We are a pretty small operation; I found having a domain more trouble than it is worth right now.

[markehle]

OK - I think I have most everything ironed out, except the email thing.

I can get the folders over, and when I rename them from having a ';' in front to a '.', they show up just fine.

But - How do I transfer over the horde mysql database from 6.0-1 to 7.2? Looks to me that there were some major changes between the two. 6.0-1 uses imp for rules, and 7.2 used ingo?

Thanks -

Library Mark

[CharlieBrady]

I can get the folders over, and when I rename them from having a ';' in front to a '.', they show up just fine.

You don't need to do that. "signal-event post-upgrade; signal-event reboot" will do that for you.

But - How do I transfer over the horde mysql database from 6.0-1 to 7.2? Looks to me that there were some major changes between the two. 6.0-1 uses imp for rules, and 7.2 used ingo?

If you did "signal-event pre-backup" before copying /home/e-smith/files, the "signal-event post-upgrade; signal-event reboot" will restore and then convert your horde db.

[markehle]

Thanks, Charlie -

The problem with this server is the root mysql password is busted. I try 'signal-event pre-backup' and all /home/e-smith/db/mysql/mysql.dump contains is 'FLUSH PRIVILEGES;'. This box is pretty hosed.

I will try to get root back, but because this box has run so well for so long, I didn't want to risk screwing it up.

So - to recap, I run 'signal-event pre-backup' , then copy over everything in /home/e-smith/files, then run
'signal-event post-upgrade; signal-event reboot' and it will convert the mail directories and restore the mysql databases and ibays.

Right?

[markehle]

hate to pester you guys again, but how do I backup and restore users? Does copying /home and running the above commands restore them, too? Or is there some files from /etc that I will need?

Thanks again!

Library Mark

[raem]

markehle

Do a search on backup, and you will find a list of all the files needed for a successful restore.

[mmccarn]

Here's a note that may help you reset your MySQL password: http://wiki.contribs.org/MySQL#Resetting_the_MySQL_root_password

Otherwise, you can backup and restore user logins using Lazy Admin Tools

[markehle]

Sorry - no method that I have tried has restored root access on mysql. There is no /var/service/mysqld directory or runsvctrl  command on this box, and running msyqld with --skip-grant-tables does not work. It acts like it should - I can gain root access when in this mode, but when you restart mysql, root still doesn't work. I think that mysql on this box was 'upgraded' with the help of some bad advise. I am able to log in as horde and dump the horde database.

Looking for a list of files to backup using 'backup' yields nothing useful. Ray - can you be more specific?

Does lazy admin restore users including passwords?

[raem]

markehle

Looking for a list of files to backup using 'backup' yields nothing useful. Ray - can you be more specific?

Look harder, it has been answered recently and about a dozen times in the last year.

[warren]

markehle

have you done the following to reset the mysql password:
/etc/rc.d/init.d/mysqld stop
/etc/e-smith/events/actions/conf-mysql-password
/etc/rc.d/init.d/mysqld start
 or

/etc/rc.d/init.d/mysqld stop
/sbin/e-smith/expand-template  /root/.my.cnf

[markehle]

Warren -

Thanks for the reply. When I run  '/etc/e-smith/events/actions/conf-mysql-password' I get:

Code: [Select]

071219 13:38:45  Warning: Asked for 196608 thread stack, but got 126976
So, like I said, mysql is hosed on this machine!

I am really close to having this thing nailed, though. My ibays are all set on the new machine. I can tranfer over people's email and files. The problem is I don't know how to grab all the account info. I tried just copying /home/e-smith/files/users/* and group, gshadow, passwd, shadow, and smbpasswd. This did not work. The accounts do not show up in the server-manager. When I set up the accounts beforehand, they work, but I loose the password info, and given the folks that I have to deal with, the less change for them the better!

What am I missing?

Thanks again!

[markehle]

"Look harder, it has been answered recently and about a dozen times in the last year."

Thanks - that is so useful.

[raem]

markehle

The problem is I don't know how to grab all the account info. I tried just copying /home/e-smith/files/users/* and group, gshadow, passwd, shadow, and smbpasswd.

Because you didn't also copy the accounts & configuration & other required databases from home/e-smith/db/... and you didn't do a dump of the mysql tables (which have basic config for webmail by default, plus anything else you may have added), and you forgot a few other files that are also needed.


After copying all the required files you need to also do
signal-event post-upgrade
reboot


Before you do the backup & restore, the
signal-event pre-backup
command needs to be run on the old server to dump the mysql tables, and you then need to copy that dump file to the new server ie from
/home/e-smith/db/mysql

but that should be included if you copy all of /home/e-smith
as per the list below.


This link will give you clues as to the general process to follow
http://wiki.contribs.org/UpgradeDisk

which refers to this long thread that has lot's of inteersting tid bits
http://forums.contribs.org/index.php?topic=30745.0

and as you cannot seem to find the correct list of files that must be transferred here is the list

/etc/e-smith/templates-custom
/etc/e-smith/templates-user-custom
/etc/group
/etc/gshadow
/etc/passwd
/etc/samba/secrets.tdb
/etc/samba/smbpasswd
/etc/shadow
/etc/smbpasswd
/etc/ssh
/etc/sudoers
/home/e-smith
/root

and add in
/opt
if you have stuff installed there.




Now there is a fairly simple procedure outlined in a Howto, which uses the standard desktop backup & restore  functions, but can exclude ibay and user data etc. You can delete any unwanted users or ibays after the restore is complete.
It's easy to do, and is essentially the same as you would do by doing it all manually (and possibly getting it wrong).
See
http://wiki.contribs.org/Backup_server_config

[raem]

markehle

"Look harder, it has been answered recently and about a dozen times in the last year."
Thanks - that is so useful.

And if you took that advice you would find what you needed, but you are being lazy.

This is not kindergarten school stuff here, if you can install sme server then you are capable of reading all the FAQ's and searching the forums on keywords.

But as you seemed to still be struggling I have provided all the answers you need in my previous post.

[markehle]

/etc/e-smith/templates-custom
/etc/e-smith/templates-user-custom
/etc/group
/etc/gshadow
/etc/passwd
/etc/samba/secrets.tdb
/etc/samba/smbpasswd
/etc/shadow
/etc/smbpasswd
/etc/ssh
/etc/sudoers
/home/e-smith
/root

Copied them all.

signal-event pre-backup DOES NOT WORK on this machine. I cannot restore root access to mysql. lat-tools did not install correctly on my new server.

Look, guys - I guess that I am on my own with this one. I am sorry to be a pest, but honestly, the documentation is not the best. Ray - you can go stuff it. I have worked hard on this project. I contribute (or did, I will say) to this project. I don't think that your comments will make me want to keep that up.

Good day, Gentlemen.

Library Mark

[CharlieBrady]

I am really close to having this thing nailed, though. My ibays are all set on the new machine. I can tranfer over people's email and files. The problem is I don't know how to grab all the account info. I tried just copying /home/e-smith/files/users/* and group, gshadow, passwd, shadow, and smbpasswd. ...

You are going about this in a quite laborious way. I note that the Subject of this thread still refers to "in-place upgrade". Did you decide not to do that?

[raem]

markehle

...but honestly, the documentation is not the best. Ray - you can go stuff it.

That's not nice, considering that earlier you said "Man - you guys are the best!"

As Charlie has suggested you have taken a difficult approach to this task of upgrading.

You could have just pulled one drive out of the old server and did a disk upgrade using this method
http://wiki.contribs.org/UpgradeDisk

and then did a disk grow using this method
http://wiki.contribs.org/Raid#Upgrading_the_Hard_Drive_Size

If you have a problem with root & mysql on your old server, then you should sort that out first as it's obviously interfering with your attempts to upgrade.

[warren]

markehle

Thanks for the reply. When I run  '/etc/e-smith/events/actions/conf-mysql-password' I get:


Code:
071219 13:38:45  Warning: Asked for 196608 thread stack, but got 126976
So, like I said, mysql is hosed on this machine!

This does not mean its hosed, only that the mysql was looking for 196Kb thread stack but only got 126kb.
 did you run, /etc/rc.d/init.d/mysqld start after the error warning ? followed by :
[root]# mysql
to actually check wether root could now access mysql ?

check here for details on this issue :
http://bugs.contribs.org/show_bug.cgi?format=multiple&id=262
This bug pointed to the cause of this problem been a non standard upgrade to mysql.

[markehle]

To all who are trying to help -

I apologize for my earlier comments. I am very frustrated with this process and took it out on you guys. That was stupid and I am sorry.  I do love SME and I know you guys work hard on it.

No - I have a new server to move into. I can't remember what I did to mysql on the old server. It was many years ago, and I have tried many things to fix it to no avail.

"This does not mean its hosed, only that the mysql was looking for 196Kb thread stack but only got 126kb.
 did you run, /etc/rc.d/init.d/mysqld start after the error warning ? followed by :
[root]# mysql
to actually check wether root could now access mysql ?"

Yup - sure did. Root still busted.

I am able to log in as horde and get that database. Ditto with the various other db's that the old server had. I just can't get there with root.

At this point, all I need is to be able to keep the old user's passwords. I can create the user accounts by hand (there is only 20 or so) and pull their email over.

Again - Sorry for my poor behavior, especially toward Ray. You didn't deserve it.

[raem]

markehle

At this point, all I need is to be able to keep the old user's passwords.

The passwords are in various files and those files contain more than just passwords, and it is not a simple task to only extract user passwords. All the files mentioned are needed. You have been told what these files are a few times in this thread now, and how to correctly use them, so I don't see what your current issue is with "getting" users passwords.

[markehle]

OK - Thanks.