Topic: Upgrade? or restore from tape - any experiences?

[Lloyd Keen]

Hi,
I'm preparing to move a site from SME 5.0 on an old box to SME 6.0 on a new box and was wondering what would be the preferred option - either Upgrade the existing, then backup to tape, install new and restore from tape or just a clean 6.0 install on the new followed by a restore using the backup from SME 5.0? Does anyone have any experiences moving from one version to another?

[Kelvin]

Hi Lloyd,

Avoid restoring an older version backup to the new version.

Recent job : SME 5.1.2 -> SME 5.6U4 (will not go to 6 as it is not production yet)

The old server will not allow me to install SME 5.6 so, I cannot upgrade first, then backup. So, I backed up the current SME 5.1.2, installed the new SME 5.6U4 server, then restored the SME 5.1.2 desktop backup to the new server.

The new server now gives errors every day about not being able to find quotas for all the users, including new users that I create now on the new server. I cannot even set quotas through server manager anymore. A few other smaller issues also came about when I first rebooted the server but I managed to work out each one (but cannot remember exactly which problems they were - sorry - had too many things to sort out other than those errors). The new server also does not power off anymore when I shutdown.

All these problems, and all I really wanted from the old server was the user accounts and passwords (which I have repeatedly said that there should have been a function to backup and restore only just these and nothing else !).

If possible, do :-

1. Backup as is now
2. Perform in place upgrade to new version
3. Check it starts correctly
4. Backup again
5. Restore latest backup to new server


Kelvin

[Ray Mitchell]

Lloyd

Seeing that you are moving through a few versions that did have database changes and firewall rules changes, you might want to take a conservative approch so that the updates run smoothly.

Rather than upgrade directly frrom old to new version, upgrade each version one at a time as has been suggested in these forums by Mitel staff so that any conversions occur correctly.
ie 5.0 to 5.1.2
then 5.1.2 to 5.5
then 5.5 to 5.6
as fas as going to v6beta3 just yet, it is still not a finished product so it is NOT a good idea for a production server.

You might like to backup the original v5.0 system first and retain the box and hard disk as is, in case of possible problems you can go back to it temporarily.

Then do a clean instal of v5.0 on your new box and instal any add on rpms and then restore from the original backup. Check everything is running OK

Uninstal any add in rpms that are not compatible with newer versions particularly rpms related to firewall access.
Temporarily remove (move) any custom templates

Then apply the upgrade OS versions one by one to the new box.

When the final version is installed ie v5.6U4, then reinstal your add in rpms making sure they are compatible with v 5.6 and apply your custom templates one by one.
You will see if any of the templates break the system and that way you can undo the damage and fix the problem, hopefully with a minimum of inconvenience.

Then you can backup the final version.

Personally I would do a clean instal of the latest 5.6U4 OS and only instal new rpms that were necessary and restore the minimum of data possible.

Hope this helps
Regs
Ray

[Ray Mitchell]

Lloyd & Kelvin
 
What about this HOWTO at
http://www.tech-geeks.org/index.php?topic=how-to

E-Smith - Migrate Users, Home, Emails, Apps to a new server

which in part says:
1. Use http://www.tech-geeks.org/contrib/loveless/batch_users/ to replicate users on new server.............

I have not used or tried the method.
Will that do it ?
Regs
Ray

[Kelvin]

Hi Ray,

Thanks for your pointer.

However, as was brought up by others before (such as the lazy admin toolkit contrib), this does not actually export / import an existing servers users. It is merely a batch addition / deletion script which relies on a text list of users which you must supply. OK for new setups, but not existing ones.

What I am actually after is something that can be installed / run from an existing SME server with users already which then examines the current server's users / groups / pseudonyms / existing password etc, and exports it to a file (obviously, passwords can be kept in their encrypted form, since you cannot get the original password from the encrypted version, nor do I wish to do it). We can then carry the file to another (or any number of) server(s) and run the reverse process which puts those users / groups / pseudonyms and passwords back onto the other server(s). Voila ! Instantly gain access to multiple SME servers without needing to recreate each user again (especially if you did not start out with a text list to begin with). In fact, if the program could be scripted to do it, we could even set cron jobs to perform the sync operation on a regular basis so that user accounts may be always replicated across x number of servers.

Kelvin

[Andrew Gray]

How hard would it be to write your own script which copies /etc/passwd, /etc/group, the home directories, and the e-mail pseudonyms?  I mean, this isn't a hard process!

That is, of course, unless there is user information stored in the e-smith configuration database.  Does anyone know if any user information is stored in the database?  Because if it is, then it's not near as simple to migrate!

- Andy Gray

[Kelvin]

Hi Andy,

>How hard would it be to write your own script which
>copies /etc/passwd, /etc/group, the home directories, and the e-mail
>pseudonyms? I mean, this isn't a hard process!

Ahhh... if only it were .....

Have you tried merely copying the passwd, shadow etc. files across.. coz, I have, and let me tell you, it does not work.

>unless there is user information stored in the e-smith configuration database

Yes there is.

Since Mitel put it together, it would be a simple process for someone from Mitel (hey, by your reckoning, a dead simple thing to do right ??) to put something like this together in less than an hour !

Kelvin

[Dan Brown]

Actually, there's a howto out there on using rsync to migrate from an old server to a new one.  Don't have the URL handy, but a search here on rsync migrate should find it.

[Charlie Brady]

Kelvin wrote:

> All these problems, and all I really wanted from the old
> server was the user accounts and passwords (which I have
> repeatedly said that there should have been a function to
> backup and restore only just these and nothing else !).

If you think that is an essential feature, then you should provide the code, or hire someone to write it for you. The software if GPL precisely so that you can make such changes which are essential to you.

Charlie

[Kelvin]

Charlie,

>then you should provide the code

Sure, but first provide :-

Which exact files stores all user names, accounts groups, etc.
Which database(s) stores anything pertinent to user accounts, groups, etc.

If the database(s) stores more than just users, groups etc., what are the relevant keys etc that need exporting and reimporting and what are not.

And any other related info pertaining to SME that is not part of a run of the mill RedHat installation.

Then, let's talk about writing a program to do this.

Kelvin

[Ray Mitchell]

Kelvin
Along with password info etc does this also help

> Which exact files stores all user names, accounts groups, etc.
> Which database(s) stores anything pertinent to user accounts, groups, etc.

/home/e-smith/accounts
/home/e-smith/configuration (and others in the /home folder)
/etc/group

> If the database(s) stores more than just users, groups etc., what are the
> relevant keys etc that need exporting and reimporting and what are not.

I think that "identification keys" are indicated in each file

/var/log/messages
and possibly also
/var/log/httpd/admin_error_log
will also tell you what is going on when changes are made

Sorry I'm not knowledgable enough to go further that that.
Regs
Ray

[Kelvin]

Hi Charlie,

>There's no economic or moral reason that Mitel (or anyone else) should
>spoon feed you.

I hardly think writing a few lines of reply in the form :-
Databases inolved = this, that and that
Keys involved = this key, that key and that other key
Simplest way to extract keys is ....
Simplest way to import keys is ....

is called spoon feeding. It saves a hell of a lot of time from having to wade through source code(s) if someone with the info, be it Mitel (the logical one) or someone else.

As for economic reason, maybe not.
Moral reason -- I've never been afraid to share what I find out or my code. I mean, if I manage to get scripts or code out that will do this job (and any other I may come across now or in the future), this benefits not just me, but anyone else in the Mitel users community who might find it useful, be it users of the unsupported versions or even the paying customers, if they want to.

>The SME server code is free software - *you* are free to do what *you*
>want with it (as long as you share).

As above, sharing is never the problem. How about, some of you get off your high horses once in a while and do the same. In the time it takes some of you to retort back with useless slams against posters who are trying to contribute but find it frustrating to have to reinvent the wheel decoding your codes, you could have already provided the necessary info and we could have all moved on. Our lives don't just revolve around SME, so if we could save time and achieve something to the betterment of the core product, we should.

>All the information you need is in the code, and the existing
>documentation.
>
>If you feel the documentation should be improved, then you are free to
>improve it. If you demand that I improve it, you do nothing but irritate
>me, and waste your time and mine.

Sure, improvements are definitely required, just as in any software, OS or otherwise. However, the same thing again applies. Who better to provide the information to improve the docs but you yourselves (not you in particular, but Mitel people in general). I demand nothing. If you can't take criticisms of your products, systems, or processes, I suggest that the Computing field is not for you.

>Please stop looking this gift horse in the mouth.

Hardly. You and other Mitel members have often said, pay for support. However, due to your own reasons for setting the pricing for SME the way it is, it is just not a viable option. Granted, I started with ESSG just after or around the time the restructuting at Mitel occurred, you stopped having a lower cost *paid* product which might have been a viable paid for product to sell. Given a choice between paying the current asking price for a supported version of Mitel SME and a Windows server, all my clients will go for the Windows product whatever its shortcomings.

Kelvin

[Charlie Brady]

Kelvin, it's extremely improper of you to post private communication into a public phorum. Goodbye.

Charlie

[Kelvin]

Charlie,

>it's extremely improper of you to post private communication into a public
>phorum. Goodbye.


Given that you were writing in reply to my (public) post on this thread and addressing it to a mailbox used exclusively for public forums, I replied as such.

However, on checking, you did send it off list. In future, I will look out for *your* replies or posts to see if they are indeed public or not.

Note that I have nothing to hide. Do you ?

Kelvin

[Michiel]

Hi Kelvin,

I had a stab at it some time ago and I think I got it working. However, I never thoroughly tested it, so maybe I overlooked something.

AFAIK the following config files are involved:
/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow
/etc/smbpasswd
/home/e-smith/accounts

You can't just copy these files from macine1 to machine2, as all user accounts and groups need to have the same uid/gid on both machines. So I did the following:

1/ Installed the lazy_admin_tools on both machines (see contribs.org)

2/ Ran "lat-dump -d -o=/root" on machine1.
This creates a series of config files in /root, that will allow you to recreate all user accounts and groups with the right uid/gid on machine2. They are called hostname1.Users, hostname1.Groups, etc, with "hostname1" the hostname of machine1.

3/ copied these config files to machine2.

4/ recreated the accounts on machine2:
lat-users -a -i=/root/hostname1.Users
lat-groups -a -i=/root/hostname1.Groups
Now the user accounts on machine1 and machine2 are identical, except for the passwords.

5/ Finally I copied /etc/smbpasswd and /etc/shadow from machine1 to machine2

That's all. As said, I believe this to work, but it would be really nice if you, or anyone else could test this procedure and comment on it.

Note that this will not work (yet) with SME 6beta, as there are some issues with the lazy_admin_tools on SME 6beta.

Michiel

[Kelvin]

Michiel,

Thank you ! I was not aware the LAT had a dump option (I was only aware that you can create users from a list provided to it). Does it also copy pseudonyms as well ?

I will test it out and provide you with feedback when next I get a moment (catching up on tax work at the moment ).

Kelvin

[Michiel]

> it). Does it also copy pseudonyms as well ?
The whole set: quota, pseudonyms, ibays, hostnames and domainnames.

[Kelvin]

Hi Michiel,

>The whole set: quota, pseudonyms, ibays, hostnames and domainnames.

OK.

Is it possible to exclude things like ibays, hostnames and / or domainnames ? I'm thinking further ahead and planning on extending the use of the export / import feature to allow say username updates on multiple servers, ie. schedule a regular export of the usernames / groups / pseudonyms / passwords to a file which is copied to one or more servers which then imports them regularly so that each of the servers have the same set of users / groups / passwords. Of course, we can script in some smarts to check if the exported file is any different to a previous exported file before sending the file out to the other servers so that they don't have to reimport the file if no changes exists.

Kelvin

[Michiel]

lat-dump will create different input-files for each of the tools, i.e. host.Users for the user accounts, host.Pseudonyms for the pseudonyms, etc. You only use those you need and can safely igonore the rest.

I guess your exporting scheme should work, but I can see one caveat. Lat-dump will only create a list of existing users, so if an account is deleted on the master server the slaves will not necessarily know they have to delete that user. You probably could solve that by running lat-dump on both machines and do a diff on the results.

Michiel

[Kelvin]

Hi Michiel,

>if an account is deleted on the master server the slaves will not necessarily know
>they have to delete that user

Good point. Goes to understanding the dump and import process.

It's still early days yet on the plan to do multi-server account sync. Will give it more thought when the time comes and will post results and scripts back to contribs if successful.

The more immediate plan will be to test single server account backup and restores.

Kelvin

[Ray Mitchell]

My experiences with backup and restore show that if you restore to an existing server with accounts etc, then the restored stuff is additive to what's already there.
That's why Mitel say to restore to a freshly installed OS, (which does not have accounts and ibays etc) and that way you end up with only the accounts etc that were in the restore file.

Seems to me you would have to delete all the existing accounts before you import new ones.

Regs
Ray

[Kelvin]

G'Day Ray,

>Seems to me you would have to delete all the existing accounts before you
>import new ones.

I did think of this as well in passing. However, I would imagine, there would need to be a process whereby the exported file(s) will need to be "intelligently" cut down or somehow processed into "to-be-imported" and "to-be-deleted" lists before sending across to the down-level servers, again, something to be further examined in more detail down the track.

Kelvin

[Ray Mitchell]

Kelvin
This rpm at
http://nx.dyndns.info/

nx-sme-adduser-1.0.2-0.noarch.rpm

looks interesting. I have not tried it.
Looks like part of the procedure has already been done, just a matter of easily getting the existing users (for transfer) into the csv file, but perhaps that's where the lat dump function comes in.

Regs
Ray


The doco says:
Automatically adds new users on a SME server from a CSV input file

Invocation: csv-adduser file.csv [delete]

With the delete option the script delets all user accounts found in the CSV file.
There is no further confirmation needed. Use carefully!

Format of the CSV-File:
----------------------
Each field must be quoted by double quotes. Fields must be seprarated by commas.
Spaces around the commas ar not allowed.

1. Line (Header):
"user","groups","password","City","FirstName","Phone","Street","Company","Dept","LastName"

The columns "FirstName", "LastName" are mandatory, all other are optional.
Is the "user" column not provided, the script generates a username, using the following rule:
First character of the firstname appended with up to 7 characters of the lastname
Example: Joe Sixpack --> jsixpack, Mary Foo --> mfoo
Is the "passowrd" column not given, the script generates it and write a list of all
passwords to /home/e-smith/files/users/admin/home/defaultpassword.csv and to
/home/e-smith/files/users/admin/home/defaultpassword.rtf for printing using a
word-processor program.

2. Datalines:
Each dataline contains the user data for the account to add. Empty fields "" are allowed.
The column count of the datalines must match the count of the header line.
In all columns german umlauts were converted (ä->ae, ß->ss etc.)
Groups must be created manually in the server-manager, before running the script!

[Kelvin]

Thanks for the info Ray. Much appreciated.

Other projects have got most of my time currently, so I have not been able to look into this again at the moment. I will once things quieten down again.

Cheers,

Kelvin