Topic: Flexbackup slow

[hanvc]

(SME Versions 5.6 and 6.0)

Flexbackup is rather slow making backups (and even slower doing restore).
Example: server with 1GB mem, XEON 2.8GHz processor and DLT 40/80GB tapedrive needs more than 4 hours to backup 10GB.
During the backup the tapedrive never streams.

Is there a way to speed things up?

See log below:


Viewed at Thu May 19 13:21:30 2005.
|------------------------------------------------
| Doing level 0 backup of all using dump
| Retensioning tape...
| Rewinding & erasing tape...
| Creating index key 200505190200.24
| Tape #0
| Filesystems = / /boot
|------------------------------------------------
At block 1.
|------------------------------------------------
| File number 1, index key 200505190200.24
| Backup of: /
| Date of this level 0 backup: Thu May 19 02:00:32 2005
| Date of last level 0 backup: the epoch
|------------------------------------------------
| (dump -0 -b 10 -a -f - / | gzip -4) | buffer -m 3m -s 10k -u 100 -t -p 75 -B \
|  -o /dev/nst0
|------------------------------------------------
  DUMP: Date of this level 0 dump: Thu May 19 02:00:32 2005
  DUMP: Dumping /dev/md1 (/) to standard output
  DUMP: Added inode 8 to exclude list (journal inode)
  DUMP: Added inode 7 to exclude list (resize inode)
  DUMP: Label: /
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 10686951 tape blocks.
  DUMP: Volume 1 started with block 1 at: Thu May 19 02:00:46 2005
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: 2.00% done at 714 kB/s, finished in 4:04
  DUMP: 4.21% done at 749 kB/s, finished in 3:47
  DUMP: 6.23% done at 739 kB/s, finished in 3:45
  DUMP: 7.88% done at 702 kB/s, finished in 3:53
  DUMP: 10.03% done at 714 kB/s, finished in 3:44
  DUMP: 12.34% done at 732 kB/s, finished in 3:33
  DUMP: 14.34% done at 729 kB/s, finished in 3:29
  DUMP: 16.12% done at 717 kB/s, finished in 3:28
  DUMP: 18.22% done at 721 kB/s, finished in 3:21
  DUMP: 20.31% done at 723 kB/s, finished in 3:16
  DUMP: 22.23% done at 719 kB/s, finished in 3:12
  DUMP: 23.80% done at 706 kB/s, finished in 3:12
  DUMP: 25.25% done at 692 kB/s, finished in 3:12
  DUMP: 26.73% done at 680 kB/s, finished in 3:11
  DUMP: 28.78% done at 683 kB/s, finished in 3:05
  DUMP: 30.82% done at 686 kB/s, finished in 2:59
  DUMP: 32.76% done at 686 kB/s, finished in 2:54
  DUMP: 34.40% done at 680 kB/s, finished in 2:51
  DUMP: 36.07% done at 676 kB/s, finished in 2:48
  DUMP: 38.13% done at 679 kB/s, finished in 2:42
  DUMP: 40.81% done at 692 kB/s, finished in 2:32
  DUMP: 44.57% done at 721 kB/s, finished in 2:16
  DUMP: 46.52% done at 720 kB/s, finished in 2:12
  DUMP: 48.61% done at 721 kB/s, finished in 2:06
  DUMP: 50.25% done at 715 kB/s, finished in 2:03
  DUMP: 52.02% done at 712 kB/s, finished in 1:59
  DUMP: 53.82% done at 710 kB/s, finished in 1:55
  DUMP: 55.59% done at 707 kB/s, finished in 1:51
  DUMP: 57.52% done at 706 kB/s, finished in 1:47
  DUMP: 59.51% done at 706 kB/s, finished in 1:42
  DUMP: 61.56% done at 707 kB/s, finished in 1:36
  DUMP: 63.50% done at 706 kB/s, finished in 1:31
  DUMP: 65.61% done at 708 kB/s, finished in 1:26
  DUMP: 67.41% done at 706 kB/s, finished in 1:22
  DUMP: 69.19% done at 704 kB/s, finished in 1:17
  DUMP: 71.14% done at 703 kB/s, finished in 1:13
  DUMP: 73.22% done at 704 kB/s, finished in 1:07
  DUMP: 75.15% done at 704 kB/s, finished in 1:02
  DUMP: 77.34% done at 706 kB/s, finished in 0:57
  DUMP: 79.80% done at 710 kB/s, finished in 0:50
  DUMP: 81.87% done at 711 kB/s, finished in 0:45
  DUMP: 84.06% done at 712 kB/s, finished in 0:39
  DUMP: 85.90% done at 711 kB/s, finished in 0:35
  DUMP: 88.47% done at 716 kB/s, finished in 0:28
  DUMP: 90.38% done at 715 kB/s, finished in 0:23
  DUMP: 91.85% done at 711 kB/s, finished in 0:20
  DUMP: 93.91% done at 711 kB/s, finished in 0:15
  DUMP: 96.03% done at 712 kB/s, finished in 0:09
  DUMP: 97.82% done at 711 kB/s, finished in 0:05
  DUMP: 99.56% done at 709 kB/s, finished in 0:01
  DUMP: 100.00% done at 706 kB/s, finished in 0:00
  DUMP: Volume 1 completed at: Thu May 19 06:15:57 2005
  DUMP: Volume 1 10814250 tape blocks (10560.79MB)
  DUMP: Volume 1 took 4:15:11
  DUMP: Volume 1 transfer rate: 706 kB/s
  DUMP: 10814250 tape blocks (10560.79MB)
  DUMP: finished in 15311 seconds, throughput 706 kBytes/sec
  DUMP: Date of this level 0 dump: Thu May 19 02:00:32 2005
  DUMP: Date this dump completed:  Thu May 19 06:15:57 2005
  DUMP: Average transfer rate: 706 kB/s
  DUMP: DUMP IS DONE
Kilobytes Out 7553800
|------------------------------------------------
| Backup start: Thu May 19 02:00:32 2005
| Backup end:   Thu May 19 06:16:05 2005
|------------------------------------------------
At block 755381.
|------------------------------------------------
| File number 2, index key 200505190200.24
| Backup of: /boot
| Date of this level 0 backup: Thu May 19 06:16:15 2005
| Date of last level 0 backup: the epoch
|------------------------------------------------
| (dump -0 -b 10 -a -f - /boot | gzip -4) | buffer -m 3m -s 10k -u 100 -t -p \
|  75 -B -o /dev/nst0
|------------------------------------------------
  DUMP: Date of this level 0 dump: Thu May 19 06:16:15 2005
  DUMP: Dumping /dev/md0 (/boot) to standard output
  DUMP: Added inode 8 to exclude list (journal inode)
  DUMP: Added inode 7 to exclude list (resize inode)
  DUMP: Label: /boot
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 9885 tape blocks.
  DUMP: Volume 1 started with block 1 at: Thu May 19 06:16:16 2005
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: Volume 1 completed at: Thu May 19 06:16:21 2005
  DUMP: Volume 1 9870 tape blocks (9.64MB)
  DUMP: Volume 1 took 0:00:05
  DUMP: Volume 1 transfer rate: 1974 kB/s
  DUMP: 9870 tape blocks (9.64MB)
  DUMP: finished in 5 seconds, throughput 1974 kBytes/sec
  DUMP: Date of this level 0 dump: Thu May 19 06:16:15 2005
  DUMP: Date this dump completed:  Thu May 19 06:16:21 2005
  DUMP: Average transfer rate: 1974 kB/s
  DUMP: DUMP IS DONE
Kilobytes Out 5450
|------------------------------------------------
| Backup start: Thu May 19 06:16:15 2005
| Backup end:   Thu May 19 06:16:28 2005
|------------------------------------------------
At block 755926.
|------------------------------------------------
| Rewinding...
|------------------------------------------------

 
 



--------------------------------------------------------------------------------
Mitel Networks SME Server 5.6
Copyright 2002 Mitel Networks Corporation.
All rights reserved.

[KelvinLee]

Is there a way to speed things up?

Yes.

Stop using flexbackup.

Look into using the Arkeia contrib or if you're not against spending a little on backup software, look at Tapeware.

Kelvin

[Tib]

hanvc

I use Arkeia on my SME server with a DLT 40/80 and I backup 19gig at 138mb/min and takes 8301 sec ... that equates to 2.3 hrs approx.

Arkeia is quite easy to setup and the gui can be run on windows or linux ... I found ther linux ver has a few nicer features in it but there almost the same.

I wouldn't bother trying to use the server-admin addon for arkeia ... just download the windows or linux gui.

For your sme server you will have to dl the arkeia-light rpm ... remember to read all the docs as well.

Any queries with Arkeia just ask ... I will help as much as I can ... I also have all the files required if you cannot find the files for sme.

Do a search for Arkeia Light on google to find where to d/l the  latest files.

Have fun

Tib

[dave_d]

On the subject of backups .....

Has anyone ever tried getting BRU to work on an SME server?  A quick search of the forums doesn't yield anything.

Regards,

Dave

[tdekeizer]

I had a very similiar problem with Flexbackup backing up 12GBytes on a Dell Poweredge with an internal Adaptec 3960D chipset and a 72GByte DAT drive.  It was taking over 6 hours to complete the backup equating to  about 500kBytes/sec.

As a test I ran a pure tar backup of the same files onto the drive from the command line and the backup took just under 50 minutes.  The only difference between this and the flexbackup command was flexbackup piped the tar output through buffer and then to the tape drive.  

I then modified /etc/flexbackup.conf to set buffer to false.  Flexbackup now took 54 minutes to complete the backup equating to 3.5MBytes/sec.  

It has been running fine(and fast) ever since.

I hope this helps.

Regards
Tony

[rf131]

Hi Tib,

Do any of the older Arkeia versions allow writing backups to disk?  It looks like the more-recent freebies only backup to tape.

I'm going with the USB drives as my backup media.

Thanks,
Kevin

[jackl]

Tony,
Tried your suggestion of setting buffer to false and backup performance improves dramatically like you said. However I use dump rather than tar, to allow me to run interactive restores. When I attempt a restore from tapes made with this configuration I get:

| Logging output to "flexbackup.restore.200509141455.log"
| Reading from CURRENT TAPE POSITION
|------------------------------------------------------------
At block 1.
|------------------------------------------------------------
| dd ibs=10k obs=10k conv=noerror,sync if="/dev/nst0" | gzip -dq | restore
|   -i   -b 10 -f -
|------------------------------------------------------------

gzip: stdin: invalid compressed data--format violated
4+0 records in
3+0 records out
restore > quit
|------------------------------------------------------------
At block 5.
|------------------------------------------------------------

Looks like disabling buffer has upset gzip
Setting compression to 'hardware' instead of 'gzip'
cures the problem and performance stays improved but I reckon I now have no compression.

Any thoughts anyone?


Regards
Jack

[pearless]

I too was having really bad backup times (2 days to back up 69GB onto a SDLT 110/220GB drive at 550kb/s).

Here is step by step how I did it:

Firstly I logged onto my SME Server via putty as root, then I did the following:

mkdir -p /etc/e-smith/templates-custom/etc/flexbackup.conf
cd /etc/e-smith/templates/etc/flexbackup.conf
cp 10Buffering /etc/e-smith/templates-custom/etc/flexbackup.conf/
cp 10Compression /etc/e-smith/templates-custom/etc/flexbackup.conf/
cd /etc/e-smith/templates-custom/etc/flexbackup.conf/
vi 10Buffering
(press <insert> to go into insert mode, make the changes, press <esc> to get out of insert mode and then type in :wq to write the changes and quit.)
vi 10Compression
(press <insert> to go into insert mode, make the changes, press <esc> to get out of insert mode and then type in :wq to write the changes and quit.)


This now means that when your tape job next runs, it will replace your /etc/flexbackup.conf file with the changes as it is supposed to do as you should NEVER directly edit your /etc/flexbackup.conf file as it is overwritten every time the tape backup job runs!

Took a bit of fiddling to figure it all out, so I hope this helps, but it sped things up dramatically, if now takes 1 hour and 56 minutes!! at 10MB/s.

Below are my changed files:
10Buffering:
============start of file===============
$buffer      = 'false';  {# true to use "buffer" program to help streaming}
$buffer_megs = '0';     {# buffer memory size (in megabytes)}
$pad_blocks  = 'true';  {# true to pad blocks to blocksize}
============end of file===============

Note, changing buffer_megs to 0 is important as it generates and error if you have a buffer size defined, but $buffer set to false!

and 10Compression I changed it to:

============start of file===============
$compress    = 'hardware';  {# one of false/gzip/bzip2/compress/hardware}
$compr_level = '4';     {# compression level (1-9) (for gzip/bzip2/zip)}
============end of file===============


My next mission is to figure out how to back up other LINUX servers (Fedora and Debian), and windows workshations (probably view SMB)
I hope this helps,

Cheers Douglas