Topic: Back up !

[Arne]

I know the question about how to back up things has been in this discussion before. Actually I knew allmost nothing about how to make Linux backups before I red theese discussions and made some experiments according to them.

I have tried the ordinary default method and I have also tried the bacup2ws contrib. These two seems to be working rather the same way exept for some improvements in the bacup2ws contrib. Both are for backing up via network to a windows machine (or possibly you can use another Samba server, this I have not fount out yet.)

I have also tried the dd kommand that I found in the discussions here. Worked great and made a digital identic copy of the disc. I also found the rsync command that can do these things very effectivlely. I have tested this as well.

I think I found a very good overview of some basic prinsipals for how to back up things here on this link:
http://www-106.ibm.com/developerworks/linux/library/l-roadmap8/

But now my question:

The backup2ws contrib also has a download for "Winrar". Using this tool I can open the content og the automated e-smith backups to see their content. Wieving theese directories and files I can se that the backup files generaly is only copy from directory to directory and from file to file. Restoring this is generally only to copy back to the same locations. Theese same operations should be easy to maintain also if you want to make your own bacup routines, from hd to hd as an example.

There is one exeption from this general prinsipple of copying from directory to directory and from file to file.

There are two set of files:
/home/e-smith/db/mysql/mysql.dump
/home/e-smith/db/ldap/domainname.com.ldif
(If you have more domains the last one will be one per domain.)

Theese two files also exist on the machine that the backup is taken fromm so it is also a question of copying files. But I believe that theese two files are made during the automated e-smith backup prosess.

I believe the backup prosess works this way.
1. Make the mysql and the ldap dump, write them to the local disk.
2. Copy all files over the network.

And restoring the other way:
1. Copy all files over network.
2. Restore the ldap and the mysql dump files.

If I should be able to develop my ovn backup routines (exept for the dd command) that mooves datas from hd to hd as an example, I will need to know:

1. How to make the mysql and the ldap dump files
2. How to restore them.

Does anyboudy know how this can be done ?

(I wonder if there is not some hidden perl scripts somewhere that are able to do this job, but I don't know if this is the case or if they excist at all.)

If the prinsipples of these dump files vere known, then it would be a not so difficult thing, to make any kind of backup system for the e-smith.

[Arne]

I found this info regarding backing up the mysql. It is obvious that you can not only copy the database files on a running database. It will be neccesarry to make a dump, backup the dum file, and then eventully restore the dump file.

http://www.mysql.com/doc/en/Backup.html

(If any of you have practical experience with theese database dump procedures or any further information, I would be courious to know !)


The other problem the ldap dum, ot what ever it is, I have not found anything about.

[Michael Herger]

I don't know too much about ldap. But there's some information about ldif ("man ldif") and how you could import it to your server ("man ldapadd"). And of cours on http://www.openldap.org

http://www.mysql.com/doc/en/mysql.html explains how you can feed a dump to the command line mysql interface. Or use phpMyAdmin to import the dump. It's basically one big collection of all SQL statements necessary to recreate your databases.

Regarding your backup idea: you'll find the "hidden" scripts you're looking for in /etc/e-smith/events/pre-backup/ and /etc/e-smith/events/post-backup/. They are executed before and after the backup.

Michael

[Arne]

I have been carring out some experiments using the /etc/events/pre-backup/ and post-backup scripts.

It apears that theese are just excactly the functions I am searching. I have tested on a e-smith 6.0 final. It seems like that when you are running the /pre-backup scripts it make an automatic generation of the mysql dump and the ldap file.

So then to barform a backup to any kind of media should be just to first run the pre-backup scripts and then copy over all the apropriate files and directories to the storage media. One way to see which files this is is to open one "standard e-smith backup" with the Winrar program. You just have to build up that same file structure on that storage media or storage method you might prefer. Whith the pre-backup and post-backup script sets in use, it seem like the rest of it, is just a question of "copying and storing files and directories".

By the way .. this was also a quite big jump for me to start in direction to understand how the automated configuration and administration routines work on the e-smith, so thanks a lot !!


Arne.

[Michael Herger]

I'm currently backing up to a second drive using flexbackup. There's a contrib that shows how to change from tape to disk.

My backup drive isn't mounted except for backup. So I created a simple S10mountBackupDrive in the pre-backup folder which mounts the drive before, and another one in post-backup which unmounts it after the backup. This does work perfectly well. And I don't risk to accidently kill my backups.

Michael

[Arne]

Thanks a lot again !

I have discovered one thing .. to make a dump of all datas and to make a perfect backup that is one thing. To restore it and to make it work again, thats quite another story ..

There is no post-restore as I can se. One way or the other the datas from the databasefiles and things like that has to be initiated and restored to their proper places after they have been copied in.

So far each attemt to make a restore by script based on rsync has ended with disaster and a new installation ..

One thing I have a litle doubt about .. should the files be saved and restore as just datas or should they also have properties like ownership and security rights ?? When it comes to a file like the password file can it just be copied in again or is there any more smart method that is required ? I really don't know.

I will try to work further on and make some more disasters before I eventually make the concusion that a manual (or automated by script) is not the right way to go.

I vill also try to find the flexbackup module and try to use that.

Arne.

[Derek]

Here is a Perl script that you may find useful. It dumps each database on the server to a dump file in /root and sends it to a remote location via scp. You can automate it via a cron job. Edit the  "scp $Location root\@backup:/backup/ckuamain/mysql/ "; line to change the scp destination. Look into ssh-keygen to remove the scp password prompt. You can restore the databases with phpMyAdmin or the commandline.

This script was written by a Linux SysAdmin student named Jason Bath while doing his practicum at my workplace.

#!/usr/bin/perl

# mysqlshow lists all of the databases in mysql
chomp(@db=mysqlshow);

for $db (@db) {
        $db=~s/\W|\s|Databases//g;
        if ( ! $db == "^\s*\$" ) {
                $date=date +%F;
#               print "$date \n"; # Debug
                $Location="/root/${db}-${date}";
                print "$Location\n";
                print "Backup database: $db\n";
                system ("mysqldump -B $db > $Location ");
                if ( ! -f "$Location" ) {
                        chomp($Location);
                        print "MySQL Databases and Data dumped to $Location\n";
                        system "scp $Location root\@backup:/backup/ckuamain/mysql/ ";
#                       print "scp $Location root@backup:/backup/ckuamain/mysql/ ";
                }else{
                        print "$Location does not exist, can't send to backup server.\n";
                }
        }
}

[Derek]

hmmm.....posting the script blew away the formatting, (tabs etc...).

[Arne]

Thanks a lot. Guess the syntax is still the same, with or without formating. I will test it.

[Arne]

In the end it looks like it start working .. I hacked into the script that makes the standard backup to workstation and could see it was using the "post-upgrade" script for restoring datas.

rsync -u (update) did not work as I had expected. It did not make all the fileupdates that I bilived it would. (It did not update files when the files that should be updated had a newer dato than the orginal fire, as would be the case for a fresh installation that should be updated with the old datas.

So I changed to "rsync -a " (-a archive, with most atributs like ownership, will also write over all old datas as there is no -u ) After doing the rsync I run the post-upgrade script: this way: "/sbin/e-smith/signal-event post-upgrade"

By the way, I first tried NTFS as the file system at the backup disc. Did of cource not work. I then tried Fat 32 (Thinking that it might be practical to have "could be red by a Windows machine" as an option.). Worked during backup but gave a list of errors as rsync could not transfer ovnership of files etc. Then changed the filesystem to the backup disc to ext3. The "rsync -a" backup could then be performed without any error msgs.

Everything seems to work now, with one exeption. The user accounts does not get transfered to the Horde email system. I think this is because the machine that the bacup is made from does have a horde installation that is rather different from the orginal e-smith Horde installation. I belive this happen not because it is something wrong with the backup prosedure itself, but because the horde datas does not longer have the structure they are expected to have.

I believe the error with the horde accounts would be there also if using the standard backup prosedure. (But I don't know for sure.) All in all I believe that the "manual bacup" now works like it should.

[Arne]

I tested the script and after a few adjustments of the adresses it worked.

I then made a few changes to make it fit bast of possible with my requirements (even though I know nothing about Perl programming, this was my first


#!/usr/bin/perl

# mysqlshow lists all of the databases in mysql
chomp(@db=mysqlshow);

for $db (@db) {
 $db=~s/\W|\s|Databases//g;
  if ( ! $db == "^\s*\$" ) {
  $date=date +%F;
  # print "$date \n"; # Debug
  $Location="/backup/${db}-${date}";
  print "$Location\n";
  print "Backup database: $db\n";
  system ("mysqldump -B $db > $Location ");
  }
}


As it is now it only dumps all the databases to the mounted volume or directory as you want it to do.

In this line:

$Location="/backup/${db}-${date}";

you can only change the string "/backup/" to any mounted or adressable directory i belive. (I just made a directory /backup on my machine for testing purposes.)

I believe the restore can be easily done using phpmyadmin.

I think this was a wery good script to have. I belive that just copying database files or using dd to copy a whole hd normally or at least very easy can lead to corrupted database files, as the database can be a quite living thing. If you are using the dd to copy a the whole hd or other methods for copying files (like the rsync) I belive that it will be a good idea to dump the databases anyhow, flush it after restore, and then reinstall the database tables form the dump.  

I think the script will work also for a little bit larger and dynamic databases. The reason why I belive this is because I think it use the standard mysql mysqldump function (!?). I knew it were such a function but I did not know how to make it to perform a dump of more than one database. I belive that this perl script is just using this standard mysql function in a loop in such a way that you get all the tables.

I wouldnt have found out this by myself - thanks again !

Arne.

[Arne]

I was wrong about the phpmyadmin. It did int his case not do the job for reinstalling the database files, as far as I can see.

I then had to read in the mysql docomentation and found sameting about mysql in batch mode:

http://www.mysql.com/doc/en/Batch_mode.html

I then run this command:   "mysql < /backup/webstart-2003-12-30"

where  mysql < /backup/webstart-2003-12-30 is the filename and directory to the db dump.

Seems to be working ok.

I also found something about restoring here that I must admit I really do not understand.

http://www.mysql.com/doc/en/Backup.html

If anybody can see incorrectnes in the applied method for restore, please leave a msg.

(The command: "mysql < /directory/path/dump-filename")

[Arne]

Is it this one ??

http://www.e-smith.org/docs/howto/contrib/flexbackup-to-disk-howto.htm

(I can see that they are mounting a smb share and make the backup via net. Should be easy to change this to a linux mount on the same machine if the rest of it is ok.)