Topic: won't backup to disk - flexbackup

[Rob Wellesley]

Hi

followed des duggans how to and ended up with this /sbin/e-smith/backup

#!/bin/sh

#------------------------------------------------------------
# DO NOT MODIFY THIS FILE! It is updated automatically etc.
#------------------------------------------------------------

# Tape backups are enabled

export PATH=$PATH:/sbin

/sbin/e-smith/signal-event mysql-delete-dumps > /dev/null 2>&1
/sbin/e-smith/signal-event mysql-dump-tables > /dev/null 2>&1
mount /mnt/backup
/usr/bin/flexbackup -fs all

/sbin/e-smith/signal-event mysql-delete-dumps > /dev/null 2>&1

umount /mnt/backup

#------------------------------------------------------------
# TEMPLATE END
#------------------------------------------------------------

But it don't backup to the share (i can mount and browse from command line ok) and cron mailed me this...

flexbackup version 0.9.8
/etc/flexbackup.conf syntax OK

|------------------------------------------------
| Trying "mt defblksize" instead of "mt setblk"
Error setting block size
Neither of these commands worked:
  mt -f /dev/nst0 setblk 0
  mt -f /dev/nst0 defblksize 0

Any ideas??..              Rob

[Greg O]

I had similar problems, but an upgrade to SME 5.1.2 and following the howto again exactly, start to finish, fixed the problem. For the meantime when I couldn't backup, I simply shared a drive (as per part of Des' howto) on a client workstation, mounted it, and copied the entire contents of /home/e-smith/files to it, ignoring the 'could not preserve permissions' errors.

Now I have another problem. Every time a backup reaches 1.99GB it stops with the error below. It seems to be regardless of the projected size of the backup - I've had between 5Gb and 10Gb of data - same problem.
  DUMP: Broken pipe
  DUMP: The ENTIRE dump is aborted.

For two full logs, see:
http://gw.carey.wa.edu.au/20020322.txt
http://gw.carey.wa.edu.au/20020323.txt
TIA,
Greg.

PS I'd also like to know what the problem was with "Error setting block size" but for the moment it's fixed for me (and hopefully for Rob), so if no-one can help with that little gem, then so be it.

[Filippo Carletti]

You're hitting the 2G file size max of SME.
Nothing to do.

Setting block size refers to the tape, don't worry.

[René]

I had the exactly the same problem as Rob

But I think I am not hitting the 2G file size of SME.

[root@sme2 /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             2.0G  335M  1.5G  18% /
/dev/sda1              15M  2.9M   11M  21% /boot
[root@sme2 /]#

Any ideas??

René

[Greg O]

What about partial backups then Filippo? Is there someone out there who's writing to-tape or to-desktop web-panel options that include, eg, "users a*-h*, h*-p*, q*-z*" in 3 separate files? With the obvious benefit of different partial backups over different days. Hmm, should I simply go ahead and look at flexbackup syntax and do it myself? (if it's easy enough...)

Because surely there are 3 major (size-wise) parts to an e-smith system: users, ibays, and everything else?

Rene, as per what Filippo said, that error message is a tape thing, so if you're not backing up to tape, ignore it. I would suggest what I tried though, go through the whole howto again, start to finish, removing and re-creating any files/dirs. If you're using e-smith 5.x, there's a small change - in /etc/e-smith/templates-custom/etc/cron.d/
instead of the file 'backup' under the folder 'backup', simply create the file 'backup' in the ./ directory.

Ie 4.x:
dir: /etc/e-smith/templates-custom/etc/cron.d/backup/
file: /etc/e-smith/templates-custom/etc/cron.d/backup/backup
5.x:
dir: /etc/e-smith/templates-custom/etc/cron.d/
file: /etc/e-smith/templates-custom/etc/cron.d/backup

Cheers,
Greg.

[Des Dougan]

Filippo,

I think actually it's the desktop OS causing the problem here. The SME limit seems to come into play when restoring. Greg, is the desktop being backed up to a Win 9x/ME system? I think this is a FAT32 limitation.

Des Dougan

[Filippo Carletti]

Ok, I'll try to be clear.
1. flexbackup to file on sme: there's a 2 Gb file size limit on SME
2. desktop backup to some win pc: win filesystem limits and more (see below)
3. flexbackup to tape: ok

The best way to overcome the 2 Gb limit would be upgrading to kernel 2.4.
When testing desktop backup to a pc, I found a 4 Gb limit, which seems to be related to network download. I still have to test a non win client and a server driven solution (automated backup to desktop).

I would like to have somebody confirming the 4 Gb limit.

[Matt]

I've been trying to backup to desktop and it also stops at 3.99 GB (I'm using a FAT32 partition on Windows 2000). Would this same thing happen if I configured backup to disk and used that instead of backup to desktop? If not, how can I backup 4+ GB of files?

[Tom Keiser]

Change your FAT32 partition to NTFS. FAT32 has a 4GB single-file size limit.

[Greg O]

Ok, the next obvious question for me is: Is there a way of making flexbackup use 'volumes' like Winzip does on a diskette? I've perused the flexbackup helpfile, even crashed flexbackup a couple of times. I've also been completely confused by all the options in /etc/flexbackup.conf

Surely there is someone out there who knows this stuff (or knows enough to work it out) - can I tell flexbackup to save x number (however many it takes) of 2Gb volumes of backups for each job?

Of course, you'd (potentially) later need to be able to restore those volumes... again - how?

Thanks for the dialog - it's all been very helpful. Just trying to solve my backup problems once and for all (:

TIA,
Greg.

[Bruce Friedman]

I have a Windows XP box with NTFS file system.  It is stalling at 4GB as well.  I have read the MS KB article:

Windows NT File Size and Partition Size Limits
http://support.microsoft.com/search/preview.aspx?scid=kb;en-us;Q93496

Which states that the limit is 2^64 on NTFS, however, I'm not seeing this.  Has anyone else seen this behaviour on NTFS?  Any ways around a backup to desktop of greater than 4GB?

[Tom Keiser]

IIRC, there's an issue with NTFS. If your installation partition is being used, it will have a 4GB limit because of the way the install and subsequent file system conversion works. I seem to remember that NTFS partitions newly created from a running vers of windows will not have the limitation. I don't recall seeing anything about a workaround for this, and I can't remember where I saw it, but it might put you on the track of something.Bruce Friedman wrote:
>
> I have a Windows XP box with NTFS file system.  It is
> stalling at 4GB as well.  I have read the MS KB article:
>
> Windows NT File Size and Partition Size Limits
> http://support.microsoft.com/search/preview.aspx?scid=kb;en-us;Q93496
>
> Which states that the limit is 2^64 on NTFS, however, I'm not
> seeing this.  Has anyone else seen this behaviour on NTFS?
> Any ways around a backup to desktop of greater than 4GB?

[Bruce Friedman]

The NTFS installation partition issue may have been part of the problem, but it did not solve the problem!

I repartitioned my disk with a smaller installation partition and created a new partition out of the free space from within WXP.  I then created a logical drive and formatted it with NTFS.  Thus it should support files of size 2^64 or less.

When running the backup from the server manager, the problem showed up again at 3.99GB. I thought I knew what was wrong because I figured that the temp files were being stored on my original installation partition which still may have this limitation of 4GB.

Then I thought I would get smart and use Des Dougan's backup script because it does not create a temp file - it mounts a shared directory under SMB and writes to it directly (it is faster, too).

Still halts at 4GB.  I received the following note from Des that indicates the problem is in the RedHat kernel.

>My understanding is that it's a kernel limitation. There has been traffic on this >subject in the past on the e-smith lists which might explain it better than I can, >but my understanding is that this is a limitation of the stock Red Hat kernel
>(they have an Enterprise kernel without this limitation, IIRC). It's not an NT >issue, because the problem is with a filesystem you've mounted on your server >via Samba.

If that is true, I'm not really enthusiastic about swapping out the kernel and no longer being on the main SME branch.  Anybody have ideas on how to break the streaming backup into 4GB or smaller files on the fly?  2GB would probably be the best size, since FAT32 supports that size.

Tom Keiser wrote:
>
> IIRC, there's an issue with NTFS. If your installation
> partition is being used, it will have a 4GB limit because of
> the way the install and subsequent file system conversion
> works. I seem to remember that NTFS partitions newly created
> from a running vers of windows will not have the limitation.
> I don't recall seeing anything about a workaround for this,
> and I can't remember where I saw it, but it might put you on
> the track of something.

[Greg O]

How about doing a flexbackup to local disk (of course, this presumes you've got more than 50% free disk space, but it's a start), then using tar or some other sort of voluming tool to pop into byte-sized chunks to send off to your backup device? (ie 2Gig/4Gig or whatever you choose).

Problem is, I don't know how. I've read the tar docs, and got nowhere. Specifically,
http://www.gnu.org/manual/tar-1.12/html_node/tar_42.html
says very little about the -M option. I'm guessing someone out there knows how to use tar WAY better than I do. I'm not even sure the -M option is the one needed.

still hoping to fix backup probs, (or should I say, complete non-functionality)

Greg.

[Bruce Friedman]

Greg,

This has the same problem as mentioned earlier, if the problem is in the ext2fs on Linux 2.2.19.

I found the following article referencing a 2GB file size limit:

http://groups.google.com/groups?q=ext2fs+%22file+size%22+maximum&hl=en&scoring=d&selm=20010411124006.C2072%40navel.introspect&rnum=5

Q:  Is there any way around the 2GB file-size limit in Linux?  Are there
    any stable patches to fix it?

A:   Short answer:  In a practical sense, no.  The 2GB limit is deeply
    embedded in the versions of Linux for 32-bit CPUs: in GNU libc, in the
    Linux kernel's filesystem drivers and VFS layer, in the fundamental
    design of some filesystems, and in the function calls used in the
    compiled applications and utilities furnished in (and for) typical Linux
    systems.  There is thus currently no VA-supported configuration.  Long
    answer:  All Linux ports for 32-bit CPUs (i.e., all architectures other
    than Alpha and IA-64) use 32-bit integers for file access and locking,
    yielding a maximum size of 2^31 - 1 =3D 2GB.  Creating a workaround for
    this situation without breaking existing code is an obstacle for all
    32-bit Unixes[1], whose creators met to design a standard 64-bit
    file-access programming interface, at a meeting sponsored by X/Open,
    called the Large File Summit (LFS)[2].  The resulting large-file
    standard and related system calls are likewise called LFS.

While this refers to VA Linux, I'm assuming it applies to RedHat 2.2.

Given that, if you can't write a bigger file directly, then flexbackup would have to be modified to split the file on the fly, so there never was a file created bigger than 2GB.  

However, all this begs the question, if 2GB is the limit, then why do we see problems at 4GB - that's one more bit than 2GB?

Greg O wrote:
>
> How about doing a flexbackup to local disk (of course, this
> presumes you've got more than 50% free disk space, but it's a
> start), then using tar or some other sort of voluming tool to
> pop into byte-sized chunks to send off to your backup device?
> (ie 2Gig/4Gig or whatever you choose).
>
> Problem is, I don't know how. I've read the tar docs, and got
> nowhere. Specifically,
> http://www.gnu.org/manual/tar-1.12/html_node/tar_42.html
> says very little about the -M option. I'm guessing someone
> out there knows how to use tar WAY better than I do. I'm not
> even sure the -M option is the one needed.
>
> still hoping to fix backup probs, (or should I say, complete
> non-functionality)
>
> Greg.