Topic: Backup software or better backup procedures sought

[daniel]

I have been trying several different backup solutions for my SME server. None of the built in ones seem to be able to offer everything I need.  I'm willing to go with a commercial product if someone has had good success with SME.

My server, 1TB drive mirrored, currently 130GB storage which will increase to probably about 400gb once I finish moving the other servers to the SME server.  I need to be able to do a full nightly backup to be able to carry it offsite the next day for full recovery.  I have multiple GB mysql databases I need to backup.  The prebackup is dumping them to a folder so they can be backed up.  I'm ok with that if I can just get them onto tape with everything else.

Flexbackup will not allow me to switch tapes so I can't rely on tape.  I prefer tape backup, easy to take offsite and if it fills up one tape, I can add a second when I arrive in the mornings.  I've had great success with Backupexec on Novell through the years and would love to find a tape software that would work on SME that won't break SME or SME won't break it with updates.

I'm experimenting with USB backup using backup to local USB, however the recovery seems horrible and right now I can't seem to figure out how to browse and restore data.  Plus moving 400GB to USB, might take a while. 

Has anyone tried using removable hard drives with SME?  I bought a SATA mobile Rack thinking I could add the hard drive to my SATA card and issue a DD command to it each night.  This would give me a duplicate of the hard drive each day and would meet my full rebuild from an offsite copy.

Ideal backup requirements
1.   Full server drive to tape or removable media if necessary
2.   Browse tape or removable media for file catalogs
3.   Redirect and restore certain files/folders for end user mishaps.
4.   Built into SME functionality either at console level or server-manager level
5.   Emails with success or fail details to multiple people

So, does anyone in the community have any successes I might be able to try.  If this belongs in SME general discussion please move it there.

[pfloor]

I have been using rdiff-backup for several years now with great success.  Fairly simple and it features...

-Backup to USB device.
-Backup to another SME server (local or remote via ssh and certs).
-Can do both local and remote at on the same server (one after the other).
-Incremental backups configurable how long to keep old versions.
-Stores the files with the same tree structure as the server.
-Stores files in regular format with no compression.  (This is handy as hell when someone needs a file from 5 days ago but it also wastes space).

The first backup will take quite a long time but the incremental backups are quite quick as it only transfers the changed/new data.

Personally, I have a USB drive on-site and I leave it there for that occasional "I need that xxx file from last week".  I also do a remote backup to an off-site server for redundancy/theft/fire/etc.  I also do the backup from the remote server back and the other way.  IOW...

USB1 <-- SME1 <----Internet----> SME2 --> USB2

-SME1 backups every night to USB1 and SME2
-SME2 backs up every night to USB2 and SME1

All settings are command line and there is no manager panel.

[janet]

daniel

Also see Affa
http://wiki.contribs.org/Affa

It is similar to r-diff.
You can backup to a locally connected USB disk etc, or backup to an offsite dedicated server.

If using an offsite server, the first full backup should be done onsite as it will take some time, and then the backup server can be moved to the remote location. Daily incremental backups will then run across the Internet connection.
Saves the hassle of removing and returning disks etc, and it all happens automatically, and your backup is already offsite immediately it happens.
Many other nice features with Affa including remote restore, and rising the backup server to be a full live server in the event of a major failure with your main server.

[daniel]

An Update:
I've done some work with the built in backup and restore to workstation.  I can backup 130GB to an external USB drive in a little over 2.5 hours.  THats OK for now.  I am able to read the dar files with the addon in MC, but todate I've not had good success at restoring certain files into a redirection folder.

I used the smeserver console backup to USB.  It built one file, I was able to copy that file to my windows based laptop and use winrar to view it and extract the files I wanted to recover.  This might be a good basic backup for me.  HOwever I didn't see it do the mysql dumps before it backed up.  Plus, the only restore would have to be manual after moving the usb drive to a win machine and copying the smeserver.tgz file to windows. 

I haven't tried Affa yet, I looked at it about a year ago and there were a few things I didn't think fit well in my backup/recover scenarios. 

Question for some of the developers/programmers.  I have external SATA cases that are both USB and esata connectors.  Having multiple sata connectors in the server, could I use an esata port on a server to connect a drive to and use it as a backup drive in the same fashion as the USB backup.  I know it would have to be mounted and unmounted before disconnect.  Getting the faster bandwidth through esata could possibly allow for flexbackup to write to a Hard drive, or dar, or rsync, and have the drive carried out at night.  Has anyone worked with esata drives as additional drives on SME?  All suggestions have been appreciated 

[janet]

daniel

You are doing non standard restores. The backup routines in sme server are designed to work with (ie restore to) a fresh install of sme server.

In a worst case scenario, it would be relatively easy and quick to install sme OS to a spare workstation/hard drive, restore your sme backup and reinstall contribs, as per recommended recovery procedures.
Then all your data is easily available across the network.

I'm aware that one of the standard backup methods, as well as the rdiff-backup and Affa contribs, offer individual file restore options.
The Dar2 contrib mc extension wiki article indicates that files can be restored to
/opt/dar2-mc directory.
See
http://wiki.contribs.org/DAR2#Midnight_Commander_.28MC.29_dar-extension

I am able to read the dar files with the addon in MC, but todate I've not had good success at restoring certain files into a redirection folder.

The Dar2 contrib is meant to be used with sme server, not with Windows.
Please be more specific re what "not had good success" means.

I didn't see it do the mysql dumps before it backed up.

All standard backups do a dump of all mysql databases. Look in the backup file at
/home/e-smith/db/mysql/
Alternatively you can see the dump file created during the backup process on a live server. The dump file is deleted at the end of the backup process, after it has been backed up.

...the only restore would have to be manual after moving the usb drive to a win machine and copying the smeserver.tgz file to windows

Restore is meant to be done to a fresh install of sme server OS. You are given the option to restore from USB or CD/DVD the first time you run the OS.
For further possibilities (ie reset this option etc) see
http://wiki.contribs.org/Backup_server_config#Command_line_manual_Restore_from_USB_drive_or_memory_stick
and
http://wiki.contribs.org/Backup_server_config#Restore_on_initial_reboot_after_fresh_OS_install_-_How_to_Reset_option

Here is more info about backup and restore concepts
http://wiki.contribs.org/Backup_server_config#Backup_and_Restore_concepts.2C_issues_and_other_information

I have external SATA cases that are both USB and esata connectors.  ....could I use an esata port on a server to connect a drive to and use it as a backup drive in the same fashion as the USB backup.  I know it would have to be mounted and unmounted before disconnect.

ESATA drives would do the job quickly, many of the backup methods available support pre backup and post backup scripts to cater for alternative mounts etc.
As far as I understand it, the issue with removable/external SATA (ESATA) is that the sme OS (CentOS) does not support hot swapping of drives, so you would need to power down your server when connecting and disconnecting. Not sure if the underlying CentOS in sme 8.x will support hot swap.
Someone with more specific knowledge may be able to comment further on this approach.

[daniel]

Thanks Mary, that explains my question about e-sata on SME.  It was worth a shot though, if it would support hotswap. For some faster backup transfer options I thought I would try firewire to external disk.

I understand most of the backup options are designed for rebuild or fresh installs.  However, in my business environments, many times data files need restored.  Windows files get deleted, or changed the wrong way.  Thus having daily backups are important for restore purposes.  I simply expect one backup option to give me both, individual redirected restore of files, plus full rebuild disaster recovery.  It's not uncommon to have to restore something from a previous month for testing or regeneration purposes.

As to "not having good success", I wasn't able to recover a test file.  I'm still teaching myself all the ins and outs.  In my line, I do alot of development and often have to simulate problems with software, having backup and restores from months ago are common in my development.  I will read up some more on the Dar2 extension for mc.  I must have missed that one link in my wiki research.

I'll get this figured out eventually the way I want.  My next thought was trying a TSA for linux from my backupexec software.

[janet]

daniel
 

I simply expect one backup option to give me both, individual redirected restore of files, plus full rebuild disaster recovery.

Did you not carefully read the answers I gave ?
Both types of restore are available.
Why do you ignore the answer given and want to run off and try some other option not yet even mentioned here ?

In server manager under the standard Backup or restore panel, there is an option "Selective file restore from a workstation".
Obviously you will have needed to configure and performed a workstation backup first. This uses the dar program, and is referred to as "backup with dar using e-smith-backup" here:
http://wiki.contribs.org/Backup_with_dar

Affa will do full backups to local disks (USB etc) or remote servers and is very configurable. You can keep many historical records to suit your needs. I have a backup server with over a years worth of backups, ie monthly, weekly, daily etc, as many of each as I desire and my backup server disk size can handle.
You can also do full restores or selective file restores from any of the retained backup sets, see
http://wiki.contribs.org/Affa#Restore_single_files_or_directories

I have not used it but am aware that rdiff-backup has similar capabilities, you will have to search a bit to find it as it does not appear to be documented on the wiki. Many people maintaining sme servers use it.

Both Affa & rdiff & for that matter the standard Selective restore from workstation, can all be managed remotely. Just get a local user to connect the appropriate drive and then you get onto Putty/server manager wherever you are and restore what is required. Better still with an offsite Affa backup server, just login with Putty select to restore from the required historical backup set, no other user intervention needed.

I don't know what your problems were with Affa, but I have tried all the popular backup solutions, and Affa is about as good as it gets, obviously there is no GUI interface, but the restore commands are so simple to issue.

As to "not having good success", I wasn't able to recover a test file.

That still does not tell us very much about what your issue is.

[pfloor]

Daniel,

If you haven't already done so, you may also want to look at enabling the Recycle Bin(s) on users and ibays.  This will allow users to quickly restore accidentally deleted files on the server without any admin intervention...

http://wiki.contribs.org/RecycleBin

The recycle bins also have an option to keep versions.  You can also (via custom scripts and templates) choose to have the bins automatically purged and even hide them from the users if you want.

I use this and it has saved me countless hours digging through backups and reduced those "oops, I accidentally deleted my file, can you please help me?" calls.