Topic: won't backup to disk - flexbackup

[Bruce Friedman]

Greg,

I have modified Des Dougan's desktop_backup.sh script to use tar multi-volume backup successfully.  There are assumptions about the directory to backup to - it is on your workstation and is SMB mounted to /smebackups.  The backup scripts contained here are expected to be located in /root/backup.  The size of the fragments used is 1GB.  When I tried to get close to 2GB, I ran into file size limitations in tar, however I think that could be tweaked to get under the limit if you wanted to.

You can then create a cron job for this to automate it - however your workstation must be available so that the directory can be mounted.

regards,
Bruce Friedman

Here is backup.sh
-------------------cut-------------
#!/bin/sh

if [ $# -lt 4 ]; then
        echo "Usage: $(basename $0) "
        exit 1;
fi

SMBHOST=$1
SMBSHARE=$2
SMBUSER=$3
SMBPASS=$4

#
# FRAGSIZE is the size in 1KB of the uncompressed file fragment
# value of 1048576 is 1GB
#
FRAGSIZE=1048576


mount -t smbfs -o username=${SMBUSER},password=${SMBPASS} //${SMBHOST}/${SMBSHAR
E} /smebackups

/bin/echo 1 > /root/backup/.lastfile
/bin/rm -f /smebackups/backup.tar

/sbin/e-smith/signal-event mysql-delete-dumps
/sbin/e-smith/signal-event mysql-dump-tables

/bin/nice /bin/tar --directory / --create home/e-smith etc/e-smith/templates-custom etc/e-smith/templates-user-custom etc/passwd etc/shadow etc/group etc/gshadow etc/smbpasswd etc/ssh root/.ssh --tape-length ${FRAGSIZE} --multi-volume -new-volume-script /root/backup/newvol.sh --file=/smebackups/backup.tar

# the last fragment needs to be named with the last sequence number
a=/bin/cat /root/backup/.lastfile
/bin/mv -f /smebackups/backup.tar /smebackups/backup.$a.tar

/sbin/e-smith/signal-event mysql-delete-dumps

umount /smebackups
-------------------cut-------------

Here is newvol.sh
-------------------cut-------------
#!/bin/sh

a=/bin/cat /root/backup/.lastfile

#------------------------------------------------------------
# You would think that tar'ring to the local drive and
# gzip'ping there would be the fastest method, but you would
# be wrong.
#
# Experimental results are that tar directly to samba share is
# only slightly slower (14 minutes for 1GB versus 13 minutes locally)
# than local hard disk storage.  Network was 100Mbps switched ethernet
# local server is Celeron 660, remote is P4-1400
#
# Tests also indicated that gzip was producing less than 10%
# compression on the sample files.
#------------------------------------------------------------
#
#/bin/nice /bin/gzip -f /tmp/backup.tar
#/bin/mv -f /tmp/backup.tar.gz /smebackups/backup.$a.tar.gz

/bin/mv -f /smebackups/backup.tar /smebackups/backup.$a.tar

a=/usr/bin/expr $a + 1

/bin/echo $a > /root/backup/.lastfile
-------------------cut-------------

[Greg O]

Thank you for the desktop-backup alternative, it's almost exactly  what I was talking about/asking for (:

But, given your comments (below) that flexbackup needs to do the splitting, not something afterwards (ie tar) - now I'm really out of my depth. I've crashed flexbackup a few times (is that supposed to be possible?) playing with settings/options, trying to figure out multi-voluming. In fact, is multi-voluming possible? Surely you can also play with tape-size in flexbackup as well as tar??

Hmm, even Darrel's contrib
http://www.myezserver.com/downloads/mitel/howto/flexbackup-howto.html
didn't really help me.

> Given that, if you can't write a bigger file directly, then
> flexbackup would have to be modified to split the file on the
> fly, so there never was a file created bigger than 2GB.

Cheers for all of the feedback on this,
Greg.

[Bruce Friedman]

Greg,

I am not sure why you would want to use flexbackup for backup to a desktop if the tar command includes all the directories you wish to backup.  What does the tar solution not include that you wish to utilize from flexbackup for backups to files (desktop)?

regards,
Bruce Friedman

Greg O wrote:
>
> Thank you for the desktop-backup alternative, it's almost
> exactly  what I was talking about/asking for (:
>
> But, given your comments (below) that flexbackup needs to do
> the splitting, not something afterwards (ie tar) - now I'm
> really out of my depth. I've crashed flexbackup a few times
> (is that supposed to be possible?) playing with
> settings/options, trying to figure out multi-voluming. In
> fact, is multi-voluming possible? Surely you can also play
> with tape-size in flexbackup as well as tar??
>
> Hmm, even Darrel's contrib
> http://www.myezserver.com/downloads/mitel/howto/flexbackup-howto.html
> didn't really help me.
>
> > Given that, if you can't write a bigger file directly, then
> > flexbackup would have to be modified to split the file on
> the
> > fly, so there never was a file created bigger than 2GB.
>
> Cheers for all of the feedback on this,
> Greg.

[Greg O]

Well, flexbackup is very e-smith-compliant, ie runs with the ready-running webpanels etc etc. I guess my main concern is come restore time, what do I do? Do I install e-smith first, or just format a machine with ext2, or.... ? It just opens up all new ground again instead of going with the tried and true methods that e-smith supplies.

If configured correctly, a flexbackup restore for me would simply entail:
1. Install SME 5.x (or latest) on a blank machine
2. Set up the backup device (eg /mnt/backup in fstab or whatever)
2. Copy across the backup/restore custom template and expand it
3. Run the restore.

Hate to bite the hand that feeds me -- the howto you provided (with a couple of minor changes to suit my particular setup) is working perfectly! -- but I don't really have much experience with backups, so I'm being cautious about getting off the beaten track.

Cheers,
Greg.

[Rob Wellesley]

Bruce Friedman wrote:
>

> The size of the fragments used is 1GB.  

This is great work..  Thank you

Can you indicate how one does a reinstall from the fragments?


TIA  Rob

[Bruce Friedman]

Greg,

I'm glad the procedure worked for you, too.  Don't worry about biting the hand that feeds you - I don't take this personally

Your point about staying with existing SME procedures is a good one, and one which I usually want to follow, too, for the same reasons you give.

However, in this case, backup to desktop is not a working feature on the stock SME server when your backup file size exceeds either 2GB or 4GB - in my case it appears to die at 4GB.

I imagine it might be possible to build a flexbackup like tar file that flexbackup would restore, but I was overwhelmed by the size of the flexbackup script - thousands of lines of perl, and I wouldn't want to venture into that solely for allowing a restore.

The process I would use to restore (haven't tried it BTW) is to follow your list
below, including mounting of the backup directory like in the script I gave, and then run the following command:

tar xf backup.*.tar

I suppose that wouldn't be too difficult to put in a script that would be accessible from the admin web panels.  Could start with Darrell May's restore-now RPM as a base...

BTW - if you only want to see what files are in the backup set, you can issue the following command to list them all:

tar tvf backup.*.tar

regards,
Bruce Friedman

Greg O wrote:
>  
> If configured correctly, a flexbackup restore for me would
> simply entail:
> 1. Install SME 5.x (or latest) on a blank machine
> 2. Set up the backup device (eg /mnt/backup in fstab or
> whatever)
> 2. Copy across the backup/restore custom template and expand it
> 3. Run the restore.
>

[Rob Wellesley]

Bruce Friedman wrote:
>
> The process I would use to restore (haven't tried it BTW) is
> to follow your list
> below, including mounting of the backup directory like in the
> script I gave, and then run the following command:
>
> tar xf backup.*.tar
>

So I point the restore at the directory with the backup files and execute the above?  No need to define which segment restores in what order?

Rob

[Bruce Friedman]

I haven't tried a restore yet, but based on the fact that tar will not allow compression when producing a multi-volume set, I believe they each stand on their own.

Keep in mind that the restore paths are relative, so you need to be in the directory you wish to restore to.  For example, if you were at "/", you need that to be your current directory when you enter the "tar xf backup.*.tar" command.

Rob Wellesley wrote:
>
> Bruce Friedman wrote:
> >
> > The process I would use to restore (haven't tried it BTW) is
> > to follow your list
> > below, including mounting of the backup directory like in the
> > script I gave, and then run the following command:
> >
> > tar xf backup.*.tar
> >
>
> So I point the restore at the directory with the backup files
> and execute the above?  No need to define which segment
> restores in what order?
>
> Rob

[Greg O]

I've decided to do some testing by attempting a restore from said backups, and I can't make it work.

I've tried:
1.
tar --directory / --extract --file /mnt/backup/backup.*.tar
(or cd /  ;   tar xf /mnt/backup/backup.*.tar)
Seems to process the first, then returns 3 lines:
/bin/tar: /mnt/backup/backup.2.tar: Not found in archive
/bin/tar: /mnt/backup/backup.3.tar: Not found in archive
/bin/tar: /mnt/backup/backup.4.tar: Not found in archive
It seems to me that tar is looking IN backup.1.tar for the files backup.<2-4>.tar!

2.
tar --directory / --extract --file /mnt/backup/backup.1.tar
tar --directory / --extract --file /mnt/backup/backup.2.tar
tar --directory / --extract --file /mnt/backup/backup.3.tar
tar --directory / --extract --file /mnt/backup/backup.4.tar
Seems to work, then gives errors, including messages about "from another volume" etc. My guess is that because I'm running the command separately 4 times, the files that span archives aren't restoring properly.

Now I'm sure this is a simple problem, but I can't think of how to fix it.

TIA,
Greg.

[Bruce Friedman]

Greg,

> I've tried:
> 1.
> tar --directory / --extract --file /mnt/backup/backup.*.tar
> (or cd /  ;   tar xf /mnt/backup/backup.*.tar)
> Seems to process the first, then returns 3 lines:
> /bin/tar: /mnt/backup/backup.2.tar: Not found in archive
> /bin/tar: /mnt/backup/backup.3.tar: Not found in archive
> /bin/tar: /mnt/backup/backup.4.tar: Not found in archive
> It seems to me that tar is looking IN backup.1.tar for the
> files backup.<2-4>.tar!
>

Try this command on backup with the multivolume option (--multi-volume) that was used to create the backup.

regards,
Bruce

[Greg O]

> Try this command on backup with the multivolume option
> (--multi-volume) that was used to create the backup.

Ok, I've tried the following:
tar --directory / --extract --multi-volume --new-volume-script /home/e-smith/files/ibays/backups/files/new_restore_volume.sh --file /mnt/backup/backup.*.tar

tar --directory / --extract --multi-volume --new-volume-script /home/e-smith/files/ibays/backups/files/new_restore_volume.sh --file /mnt/backup/backup.1.tar

new_restore_volume.sh simply increments a number and outputs to screen: "Processing volume "

Using either method, the procedure keeps going and going and going. I have reached very high 'volume' numbers (eg 77 - remember, the backup files that i'm trying to restore are only 4 in number!) before cancelling (ctrl-c) and trying another way. FYI, I've sometimes checked the contents of the server after I've cancelled a restore. It _seems_ that I've got all (or at least an overwhelming majority) of my files restored and complete.

TIA,

Greg.

PS: Arg, getting so frustrating. It's such a simple thing, there's gotta be SOMETHING dumb that I'm doing to make it not work.

[Greg O]

Fixed it!

Consorted with my local linux users' group, and found out that tar multi-volume always wants to look at the same filename. ie i've now set the restore newvolume script to delete the most recent (ie finished with) tar file, and rename the next one to it.

eg files:
backup.tar, backup.1.tar, backup.2.tar

how the script works:
process backup.tar
delete backup.tar
rename backup.1.tar to backup.tar
process backup.tar
delete backup.tar
rename backup.2.tar to backup.tar
process backup.tar
delete backup.tar

The dude that found my problem actually suggested that I use symbolic links, but I'm using an smbfs share (he said I'd have to work around that), so I decided it was easier to just delete files and rename. I suppose I don't need to delete, but it's easy and smooth.

Works like a charm!

Thanks for all the help, I'm now content that I can restore in the event of a catastrophe!

Cheers,
Greg.

[rob wellesley]

Greg O wrote:
>
> Fixed it!
>
> Consorted with my local linux users' group, and found out
> that tar multi-volume always wants to look at the same
> filename. ie i've now set the restore newvolume script to
> delete the most recent (ie finished with) tar file, and
> rename the next one to it.
>

Hi

Any chance of a copy of your reworked script?

and any helpful hints would be much appreciated


Rob

[Lester Bennett]

I started off using the standard backup to workstation option in SME 5.5 to backup to a Win2K workstation. If a user deleted or over wrote a file I could use Winzip on the workstation to find and extract it. Then I hit the 3.99GB brick wall. I've just made the changes in the flexbackup-howto but it's gone from bad to worse, I now only get a 2GB file. If I implement the fragmented file option, how do I extrat a single file from the backup at the workstation?

Lester Bennett

[Greg O]

*sigh* Is it really 8 months since I last posted on this? I suppose the computer never lies ):

Anyway, I never posted that complete script, because I was waiting until I had the time to get a backup restored properly, as a test. As you can see, I never got that test done - in fact, I tried a few times, and it failed for different reasons (mainly to do with tar...)

Basically we've given up on tar backups. They're backed up ok (except maybe for any file that's stored partially over two volumes), but the voluming doesn't ever seem to work properly when restoring. We actually had a critical outage of some data, and had to use these bodgy backups, which was quite a problem, because as I told the new guy - they don't restore! Anyway, he knows more than I, and whipped out a desktop distro with a graphical backup utility - ark, I think it was. Ark saw the contents of the individual tar files ok, and restored them... all except for those files that span across to tar files.

Long story short, we're now using rsync to backup, and are considering looking at Mike Rubel's snapshot method:
http://www.mikerubel.org/computers/rsync_snapshots/
In fact, Rob had a suggestion along with that - although I can't remember what it was, since the job of our backups is no longer mine, and our current setup actually works - praise the Lord!


So Lester, grab ark if you want an individual file. I'd suggest copying the whole archive (pick the specific volume if you know which one it is) to the local machine that's got ark or whatever on it. btw WinZip plain refused to see a 2GB file (:

Yep, I ticked the "commercial tape backup" option on the survey at http://www.e-smith.org (:

Bruce & Rob and others - thank you for all of your help over the last 12 months with this problem. I've really appreciated having people willing to help in a land where I am rather a stranger. I only hope you haven't had such issues (it sounds like you haven't).

Good luck!
Greg.