Topic: Backup to USB - Slowness

[daniel]

I'm running an SME7.4 server using Backup to workstation that backs up to a USB drive.
For a Server that has 322GB of data on it, the backup takes 7.5 hours.  I started watching it late at night and wondered if this is normal for the USB Dar Backup.

Server is Dell Poweredge 1800, 2-Dualcore processors, 4GB Ram, 1TB Raid Drive
Backup configuration is # of Rotating sets = 1, Daily Backups in each set = 1, Optional Timeout 5hrs, Don't timeout on full backup checked, Backup compression is 0, full backup allowed every day.

What I notice is during the DAR backup htop shows a server load of 10~11, yet during the day it usually is around 0.75~1.0.  During the heavy load I don't see any of the processors running much over 5-10% and I don't see the DAR Line using many resources or memory.  I can't figure out whats causing the load to run so high and take 7 hours to backup to USB. 

Anyone have any suggestions on what I could check or verify that this is running normally?

[CharlieBrady]

What I notice is during the DAR backup htop shows a server load of 10~11, yet during the day it usually is around 0.75~1.0.  During the heavy load I don't see any of the processors running much over 5-10% and I don't see the DAR Line using many resources or memory.  I can't figure out whats causing the load to run so high and take 7 hours to backup to USB. 

The load is very likely due to I/O backlog.

[daniel]

Thanks for the information, is there a way to reduce it, or do some sort of mbuffer or buffer to improve it?  I can live with it since it happens during the night, but I'm always looking at ways to enhance SME.

[CharlieBrady]

Thanks for the information, is there a way to reduce it...

Get faster hardware.

[idp_qbn]

Daniel, you say
"Backup configuration is # of Rotating sets = 1, Daily Backups in each set = 1"
Doesn't this mean you do a full backup every day? Couple this with the "Compression=0" option then you are transferring the full 322Gb data every day through a USB so it is going to take some time! 

You could try "Daily Backups in each set = 7" which I think gives you a weekly full backup and daily incrementals.
Compress the data - your Server is fast, it is the USB interface which is slow.
The mathematics:
USB 2.0 transfers about 40 MBytes/sec
322Gb at40 MBytes/sec = 8050 seconds = just over 134 minutes - and that's just the transfer time through the USB. Add processing time on the server and especially the USB disk and it can blow out a lot. I have seen a review stating USB disk write speeds can be 20 MBytes/sec or even slower. In fact, the 20 MBytes/sec was listed as "amazingly fast" for a USB HDD (http://tinyurl.com/ycw6lrz )

If you USB disk is USB1.1 then then it is horrendously worse : USB 1.1 transfers data at about 1.5 MBytes/sec.

If my assumptions about the meaning of "Backup configuration is # of Rotating sets = 1, Daily Backups in each set = 1" are wrong, I apologise and hope someone corrects them soon.

Good luck
Ian

[daniel]

Ian,

Thanks for the suggestions, now I have a few tweaks to try.  I'm going to push compression to 5 tonight.  Also, I had assumed my USB nest was USB2.0 but maybe it wasn't, I do have some Nests that are USB2.0, I'll swap that out too to see if I get any improvement. 

Wow, help from Australia!

[daniel]

Just an FYI,
 I verified I had a USB2.0 nest connected and I added compression =5.  It took the same amount of time to backup as it did before.  7.5 hours.  I didn't get a chance to check the HTOP load last night during the backup.   I simply need to develop a faster backup routine than USB.  I'll do some experimenting, thanks for everyones help.

BTW - whomever may be the expert on the flexbackup to tape programming, I was reading something online about using mbuffer with a tape drive and that could possibly allow for tape swaps when the first tape gets full.  Not sure if SME has looked at that or not. 

[idp_qbn]

Remember, when you change the backup options the first backup after that will be a full backup. After that, you only need incremental backups, which should be quicker.
For example, I have 450Gb data and the full backup takes a l-o-o-o-n-g time, then daily incrementals take less than 2 minutes.
I really recommend setting "Daily Backups in each set = 7" (or 14, 21, 28 ... but 7 is good)
Cheers
Ian

[daniel]

Thats good to know.  I need a backup solution that I can frequently do restores or redirected restores from. 

How often do you do restores from your backups in your configuration?  Do you do command line or a GUI interface when you do restores?

Thanks.

[paradigm]

You could backup your server to a workstation or some kind of NAS , using network insted of usb should be faster .

Remember, when you change the backup options the first backup after that will be a full backup. After that, you only need incremental backups, which should be quicker.
For example, I have 450Gb data and the full backup takes a l-o-o-o-n-g time, then daily incrementals take less than 2 minutes.
I really recommend setting "Daily Backups in each set = 7" (or 14, 21, 28 ... but 7 is good)
Cheers
Ian

This will also help - just set the full backup to be done on the weekend and daily incrementals to run on the rest of the days.

[CharlieBrady]

I need a backup solution that I can frequently do restores ...

Really? I think you should train your users to not delete data which they still need. Restoring from backup is timeconsuming.

[daniel]

I do alot of beta testing and software development for people, plus most of the files are database transactional files, they never get deleted, just changed, thus I need to restore to a previous state for testing again or fix a problem that wasn't forseen.  And sometimes people do screw up a spreadsheet that needs recovered. 

I'm testing using the workstation backup to backup to a share on a windows box now. SO far server load is staying under 2, thats good.  I'm also doing an agent backup across the LAN, but thats not 100% stable for me right now.  Right now I haven't found a great solution for online backup for SME Linux yet either, but working on testing some of them as well.

[paradigm]

I do alot of beta testing and software development for people, plus most of the files are database transactional files, they never get deleted, just changed, thus I need to restore to a previous state for testing again or fix a problem that wasn't forseen.  And sometimes people do screw up a spreadsheet that needs recovered. 

For software development you could use SVN and an SVN agent like tortoise , see here :

http://wiki.contribs.org/Subversion

For items deleted by mistake or screwd up by users you can use RecycleBin and ShadowCopy , see here :

http://wiki.contribs.org/ShadowCopy

http://wiki.contribs.org/RecycleBin

I'm testing using the workstation backup to backup to a share on a windows box now. SO far server load is staying under 2, thats good.  I'm also doing an agent backup across the LAN, but thats not 100% stable for me right now.  Right now I haven't found a great solution for online backup for SME Linux yet either, but working on testing some of them as well.

Affa could be used here to make quick snapshots every 30 minutes or so...

[janet]

daniel
 

For a Server that has 322GB of data on it, the backup takes 7.5 hours.  I started watching it late at night and wondered if this is normal for the USB Dar Backup.

That sounds about right, different systems and file types may cause some variation.

[daniel]

Thanks for the confirmation Mary.  I did some further testing, I ran SME backup to a windows workstation share, and it took the same amount of time as it did to backup to the USB2.0 external drive, about 7 hours for 320GB. Although I don't think its 320GB being backed up, thats the total storage of my server, but I thought the backup procedure didn't backup everything anyway, just the configs and data and customizations. 

I setup Symantec Backupexec 2010 running on my Windows servers to reach out and backup the SME box via the linux agent.  It backed up 270GB (I didn't select the entire drive, just what I thought the SME server was backing up according to what I read in the Wiki) and it backed it up in 4.5 hours. 

I saw in the SME 7.5 RC notes there was some updates to the backup. 
e-smith-backup
- Fix full backup on Sunday displayed as Everyday. [SME: 5624]
- workstation backup: add cifs credentials expand to bootstrap-console-save [SME: 4850]
- workstation backup: using credentials file for cifs mount [SME: 4850]
- workstation backup: allow many backups in the same day [SME: 5393]
- workstation restore: all needed backups must be available before restore [SME: 5395]
- workstation verify: add option to check integrity of backups needed in a full restore [SME: 5396]
- minor fix in translation [SME: 5406]
- Fix DAR e-mail message with regards to incremental backups [SME: 4579]
- Fix discrepancy in maximum compression level [SME: 4841]

I look forward to testing to see if these fixes have given me any performance improvements.