Topic: Large backup to hard disk

[LT]

I have an SME 5.1.2 with two 100GB disks and RAID. Currently I do selected backups of e-bay contents manually, but I would like to use a Windows workstation as a backup utility for the entire SME. Using Windows XP and NTFS, will my SME be able to do a backup to a workstation through the server-manager backup routine although the backup file will be larger than 2 GB?

Regards
Lars

[Kelvin]

Hi Lars,

The backup to desktop option in server manager is not usable for large backups (you can backup but the backup file can never be restored).

There is a contrib which allows you to mount a windows share on the SME server and use that as a target for the flexbackup routine instead of a tape drive. I've use this contrib before, but never on a server that needs large backups. Since there is no size limitation when backing up to a tape drive, it might be that if you stream the backup to a file on a windows share, the same might apply.

Just a thought.

Kelvin

[Cyrus Bharda]

I thought there was a problem with the 2GB file size limit?

I know that ext2 and NTFS can handle large files, but I think there are some threads about the flex backup to disk not working for large files,

<<>>

http://forums.contribs.org/index.php?topic=16245.msg62762#msg62762

http://forums.contribs.org/index.php?topic=16245.msg62762#msg62762

http://forums.contribs.org/index.php?topic=16245.msg62762#msg62762

As I understand it, you can do backups to Win2K/XP but will run into the 4Gb files size limit, which I beleive is still inherent in Win2K/XP OS.

The problem is if you try to restore those files, then you run into a 2Gb size limit at the sme server end, so effectively you are limited to 2Gb files sizes for a backup to desktop & restore operation.

For Kelvin, I do video editing work on my Win2K PC and the software automatically splits my 13Gb of data (for 1 hour of video) into 4Gb chunks, so there is a definite 4Gb limit on Win2K. The software also automatically joins the chunks together, so its as if it is one big 13Gb file.
I cannot copy these 4Gb files to my sme v5.1.2 or 5.5 servers though, as the sme has a 2Gb file size limit.

http://forums.contribs.org/index.php?topic=16245.msg62762#msg62762

Very interesting to see that it is mentioned that if you use the flex backup to disk then there is no limit but you cannot restore from it, but I am not too sure about it as there are conflicting reports.

My questions are:

1. When using the flex backup to disk and sending it to a windows share using NTFS are you limited to a certain sized backup file?

2. If you are able to make large files with flex backup to disk, are you able to restore from them?

Thanks and if you are looking for the flex backup to disk Howto:


http://myezserver.com/downloads/mitel/howto/flexbackup-to-disk-howto.html

Cyrus Bharda

[Kelvin]

Hi Cyrus,

As you can see, I've often piped in about this topic because there is a recurring confusion about the limitations involved in the different platforms. To summarise,

On Win95 using FAT 16, largest single file size = 2GB
On Win95B and above using FAT 32, largest single file size = 4GB
On WinNT 4 and above using NTFS, largest file size is in terabytes (don't care about the actual number as I currently cannot possibly fill that up at the present moment ! ). There is NO 4GB limit in WinNT, W2K or XP as long as you use NTFS. If you are hitting a limit, look elsewhere for the cause, it is NOT an inherent limitation of the NTFS file system or the OS (except for NT4's boot partition, but there are work arounds anyway -- search MS Technet, does not affect W2K or XP).

I'm not a Linux expert and don't pretend to be. However, as I understand it,
on SME up to version 5.5, largest single file = 2GB, hence the limit with the server manager restore from desktop function.

I'm no expert with flexbackup either. However, we know that flexbackup can stream to and restore from a tape drive with no limitation on size. If this behaviour is the same when backing up to a mounted windows share as per the how-to, then there should no reason why you can't backup more than 2 or 4GB to the windows share (NTFS, forget about FAT16 or FAT32). Unfortunately the how-to does not go far enough to cover restores as well and for anyone who does not know how (count me in as well), restores could be quite a challenge.

Kelvin

[ryan]

Lars,

Note:  I have not done this in a production environment with a large amount of data.  I would test, test, and test again to verify it will work for you.

If you are only seeking to backup user home folders and your ibays, I have found an easy (somewhat) way to do this.  Edit your /etc/smb.conf file and add the following between [printers] and [Primary] :

[smbfiles]        (you can name this anything you want)
comment = smbfiles
path = /home/e-smith/files
browseable = no
guest ok = no
writable = no

Now add a user named admin to your windows 2k or XP computer.  Use the same password as your root password on SME.  Login as admin and do this:

Start/run   type:  \your_sme_server\smbfiles  

You must enter this manually at run at least once to make Windows aware of the share.  (browsable=no in smb.conf).  Once you do this, the share should appear in network neighborhood or when you browse your SME server from this computer.  You might map a drive to the smbfiles share.  

You can now use Microsoft backup to backup selected or all of the folders within smbfiles.  I recommend Windows XP over 2k because the backup has the shadow copy feature which allows it to backup a locked file.  Note that the smbfiles setting in smb.conf is not writeable for security, so you can not perform a restore to the smbfiles share.  Just remember when logged in as the admin, you can access user home folders by entering them manually at start/run....do this to restore files to home directories.  Ibays can be viewed by all so they are not an issue for a restore.  
 
Note:  If you have added root as an samba user, you can use that account instead of the admin account.  

Hope this helps,

ryan

[LT]

Thank you for the assistance, Ryan, I will experiment with this. Still, I guess it´s funny if a large (> 4GB) restore can´t be performed from a disk when it can be performed from a tape.

Regards
Lars

[ryan]

If you have more than 4GB compressed of backup data, you might consider a cheap IDE backup tape for your Windows computer.  By cheap I mean $500 or less.  I don't know how large the Windows backup file to disk can be???

ryan

[Cyrus Bharda]

>I'm no expert with flexbackup either. However, we know that flexbackup can >stream to and restore from a tape drive with no limitation on size. If this >behaviour is the same when backing up to a mounted windows share as per the >how-to, then there should no reason why you can't backup more than 2 or 4GB >to the windows share (NTFS, forget about FAT16 or FAT32). Unfortunately the >how-to does not go far enough to cover restores as well and for anyone who does >not know how (count me in as well), restores could be quite a challenge.

The restore is just about as easy as the backup:

http://www.e-smith.org/docs/howto/contrib/flexbackup-restore-from-disk-howto.html

So my understanding now is that as long as you are using the flexbackup to disk and sending it to a NTFS shared drive then there should be no problems with the backup or restore from/to large files (>2GB) ? Right?

Thanks,

Cyrus Bharda

[Kelvin]

>So my understanding now is that as long as you are using the flexbackup to
>disk and sending it to a NTFS shared drive then there should be no problems
>with the backup or restore from/to large files (>2GB) ? Right?


That's the theory.

Now, all that's left, is for someone to test it !

Kelvin

[Cyrus Bharda]

Well so far my backups are up to ~500MB so it'll take me a couple of months to get to >2GB but when it happens and I test the restore I'll report back with the results

Cyrus Bharda

[Murray Tracey]

Hi,

I have tested this, (on SME5.6) my backups are 4.7Gig, I am backing up to an NTFS share.  Flexibackup 'aborts the DUMP' at 2Gig

I can backup using 'Backup to Desktop' to the same disk, but am still looking for a solution for a restore.  We will be buying a tape drive in the future, but it is not in the budget just yet.

Cheers
-Murray

[Cyrus Bharda]

Is there anyway of configuring flexbackup to make ~2GB files?

It just seems such a waste for such a great and easy way of backing up to a share could be limited in such a way!!

Cyrus Bharda

[ryan]

If you are currently unable to backup all your data with flexbackup or backup to desktop, you might consider this solution.  I have this working on 5.1.2.

Install IDE CD burner in SME.  The CD should not share a cable with a hard drive.  Install Backup to CD for SME (search the forum).  This allows you to do a Backup to CD in addition to Backup to Desktop from the server manager.  More importantly, you can exclude directories from the backup.  I excluded all ibays and home folders from my CD backup.  This will give you a backup of SME minus these folders.  Note:  I have not tested a full restore with this...but I have restored files from the backup CD..so you should test this.

To backup your ibays and home directories, take a look an earlier posting I made:

http://e-smith.org/bboard/read.php?f=1&i=24089&t=24078

This will allow you to use win2k or XP pro to backup all your ibays and home directories.  Since Windows backup is used, no file limitations will exist if your using NTFS.  You can backup to a disk or tape with this meathod.  Again, I have not performed a full restore of a server with this type of backup....TEST IT!!!!  I like this method because you can quickly and easily restore a single file or folder from any ibay or home directory.

ryan

[Dave Owen]

According to the Flexbackup information at http://www.edwinh.org/flexbackup/README, patches are required for flexbackup to support files larger than 2GB.

According to the Flexbackup information at http://www.edwinh.org/flexbackup/, "This program can use the "buffer" utility to speed up tape and network I/O: (recommended, not required). The RPMs are compiled with changes to allow large file (>2GB) support."

So there's your answer about the 2GB limit.

[Kelvin]

Hi Dave,

This isn't necessarily clear.

The readmes notes the use of buffer to allow support for *files* larger than 2 GB. not *backups* larger than 2 GB, ie. say you have a single file larger than 2GB in your server that needs to be backed up.

This obviously canot happen in SME <= 5.5 because the files system itself does not support single files larger than 2GB. However, with 5.6, I believe the new file system will support files > 2GB.

Flexbackup even in the older versions of SME will support backups > 2GB to tape and even restore from it (not single files > 2GB). We know the server-manager based backup to desktop cannot support backups larger than 2GB because the restore process creates a temp file on the server (which cannot exceed the 2GB limit) in addition to any inherent limitation within flexbackup itself.

However, what we are interested to know is whether or not flexbackup behaves the same way when backing up to a mounted fileshare as when backing up to tape when using the flexbackup to disk how-to. If it *does*, then theoretically, there should be no reason that you cannot backup and restore > 2GB.

Obviously, if you are using SME 5.6 and will have individual files > 2GB, you will need the updated modules and patches (if they are not already in SME). All these patches don't mean a thing to older SME versions because the filesystem does not support it anyway.

Kelvin

[Cyrus Bharda]

I am going to ask these questions to the flexbackup help list to see wether we can get a straight answer out of them

Will report back with results!

Cyrus Bharda

[Cyrus Bharda]

And wow what a response time!

{START QUOTE}

[ Cyrus Bharda wrote: ]
> So here are our questions:
>
> 1. How big can files be created by flexbackup IF it is to a Windows NTFS share?

The limitation would be with smbfs if anything.  I have no clue what the
filesize limits are for it.  It might be your limiting factor.

> 2. Can a file that is >2GB that was created by flexbackup to a Windows
> NTFS share be restored without running into the 2GB limit on the RedHat
> Box?

If the SME core utils are from RedHat 7.x it should handle >2GB files just
fine - large file support has been around a while.

I'm not sure if the SME setup uses "buffer" or not, but buffer needs a
compilation tweak to read/write >2GB files.  If you are just using dd it
should be fine.

Basically any 2GB limits will come from the utilities used in the pipeline
- dd/buffer/dump/afio/whatever, or in the underlying filesystem (smbfs in
your case), and not from flexbackup itself.


--
 Edwin Huffstutler     Linux - because reboots are for hardware changes
 edwinh@computer.org   GnuPG Key ID: AE782DC9

{END QUOTE}

So from this basically there is no problem with flexbackup writing/reading >2GB files, right?

Cyrus Bharda

[Cyrus Bharda]

Or maybe we need to find out if there is a size limitation for smbfs and what type of pipline is used for the backup and if that has a file size limitation.

Then we will have the whole answer in its entirety.

So anyone know the answers, all I get if I google smbfs is kernal numbers which is way beyond the scope of this intermediate-noob.

Also anyone know what type of pipeline is used when SME does use flexbackup?

Thanks everyone for all the info!!

Cyrus Bharda

[Kelvin]

Hi Cyrus,

I have experimented on my old SME 5.1.2 server.

Backup to disk does not support backups > 2GB.

A quick search on google :-

smbfs does not support files > 2GB. There are patches / new versions which support up to 4GB. Still not where we want it.

And smbfs is probably not the only dependency that has limits. All in all, I think we should accept the fact that like the excellent taper program, we should only keep these for backups of small servers or servers only used for mail and gateway services (where the backups are likely to be small).

Which brings to mind a question about the 2GB limit in general. I wonder how recoverable is a large SQL database that's been split into multiple 2GB chunks ? This is the only way you can support large databases under something like the SME server <= 5.5. What if one or 2 of the database "chunks" were corrupted / damaged / accidentally - purposely deleted / etc. Will the database still be recoverable without needing a full database restore ?

Kelvin

[Nico Blok]

Hi there,

I'm following the discussion for a while. Is this what we are looking for?

http://www.cs.helsinki.fi/linux/linux-kernel/2002-22/0764.html

Greetings
Nico Blok

[Cyrus Bharda]

I don't know? All the info on that page is way over my head, might post this topic for the dev-info list and see what they think............thats if no one here can decipher the info on the page that Nico has generously supplied?

Cyrus Bharda

[Nico Blok]

I'm not sure about this, but i think that SME 5.5 is using the 2.2.? kernel. 5.6 is using the 2.4 kernel. The site i mentioned says there is no problem with kernel 2.5. For 2.4 there is a patch.
I think that there is no other way then using this patch on 2.4 (i'm not doing this, cause i dont have experience with rebuilding a kernel). Or we have to wait till an update for SME with a new kernel.

Nico Blok

[Kelvin]

It's more than just smbfs. Samba also needs updating plus you need to check the various libraries (like glibc, etc) to make sure they have LFS support, etc, etc. I'm just not sure it's worth it.

The afio utility would be a great one to use as an alternative except that it does not seem to know how to create a series of backup segment files instead of multiple instances of the same file name (which means it overwrites the previous one) and it also seems to only work interactively when doing segmented backups.

Kelvin

[Kelvin]

Oops... forgot to add ...

Of course, we all know that Darrell May already has something that works with WinRAR that does this....

Kelvin