Topic: Backup to USB - Slowness

[daniel]

I'm running an SME7.4 server using Backup to workstation that backs up to a USB drive.
For a Server that has 322GB of data on it, the backup takes 7.5 hours.  I started watching it late at night and wondered if this is normal for the USB Dar Backup.

Server is Dell Poweredge 1800, 2-Dualcore processors, 4GB Ram, 1TB Raid Drive
Backup configuration is # of Rotating sets = 1, Daily Backups in each set = 1, Optional Timeout 5hrs, Don't timeout on full backup checked, Backup compression is 0, full backup allowed every day.

What I notice is during the DAR backup htop shows a server load of 10~11, yet during the day it usually is around 0.75~1.0.  During the heavy load I don't see any of the processors running much over 5-10% and I don't see the DAR Line using many resources or memory.  I can't figure out whats causing the load to run so high and take 7 hours to backup to USB. 

Anyone have any suggestions on what I could check or verify that this is running normally?

[CharlieBrady]

What I notice is during the DAR backup htop shows a server load of 10~11, yet during the day it usually is around 0.75~1.0.  During the heavy load I don't see any of the processors running much over 5-10% and I don't see the DAR Line using many resources or memory.  I can't figure out whats causing the load to run so high and take 7 hours to backup to USB. 

The load is very likely due to I/O backlog.

[daniel]

Thanks for the information, is there a way to reduce it, or do some sort of mbuffer or buffer to improve it?  I can live with it since it happens during the night, but I'm always looking at ways to enhance SME.

[CharlieBrady]

Thanks for the information, is there a way to reduce it...

Get faster hardware.

[idp_qbn]

Daniel, you say
"Backup configuration is # of Rotating sets = 1, Daily Backups in each set = 1"
Doesn't this mean you do a full backup every day? Couple this with the "Compression=0" option then you are transferring the full 322Gb data every day through a USB so it is going to take some time! 

You could try "Daily Backups in each set = 7" which I think gives you a weekly full backup and daily incrementals.
Compress the data - your Server is fast, it is the USB interface which is slow.
The mathematics:
USB 2.0 transfers about 40 MBytes/sec
322Gb at40 MBytes/sec = 8050 seconds = just over 134 minutes - and that's just the transfer time through the USB. Add processing time on the server and especially the USB disk and it can blow out a lot. I have seen a review stating USB disk write speeds can be 20 MBytes/sec or even slower. In fact, the 20 MBytes/sec was listed as "amazingly fast" for a USB HDD (http://tinyurl.com/ycw6lrz )

If you USB disk is USB1.1 then then it is horrendously worse : USB 1.1 transfers data at about 1.5 MBytes/sec.

If my assumptions about the meaning of "Backup configuration is # of Rotating sets = 1, Daily Backups in each set = 1" are wrong, I apologise and hope someone corrects them soon.

Good luck
Ian

[daniel]

Ian,

Thanks for the suggestions, now I have a few tweaks to try.  I'm going to push compression to 5 tonight.  Also, I had assumed my USB nest was USB2.0 but maybe it wasn't, I do have some Nests that are USB2.0, I'll swap that out too to see if I get any improvement. 

Wow, help from Australia!

[daniel]

Just an FYI,
 I verified I had a USB2.0 nest connected and I added compression =5.  It took the same amount of time to backup as it did before.  7.5 hours.  I didn't get a chance to check the HTOP load last night during the backup.   I simply need to develop a faster backup routine than USB.  I'll do some experimenting, thanks for everyones help.

BTW - whomever may be the expert on the flexbackup to tape programming, I was reading something online about using mbuffer with a tape drive and that could possibly allow for tape swaps when the first tape gets full.  Not sure if SME has looked at that or not. 

[idp_qbn]

Remember, when you change the backup options the first backup after that will be a full backup. After that, you only need incremental backups, which should be quicker.
For example, I have 450Gb data and the full backup takes a l-o-o-o-n-g time, then daily incrementals take less than 2 minutes.
I really recommend setting "Daily Backups in each set = 7" (or 14, 21, 28 ... but 7 is good)
Cheers
Ian

[daniel]

Thats good to know.  I need a backup solution that I can frequently do restores or redirected restores from. 

How often do you do restores from your backups in your configuration?  Do you do command line or a GUI interface when you do restores?

Thanks.

[paradigm]

You could backup your server to a workstation or some kind of NAS , using network insted of usb should be faster .

Remember, when you change the backup options the first backup after that will be a full backup. After that, you only need incremental backups, which should be quicker.
For example, I have 450Gb data and the full backup takes a l-o-o-o-n-g time, then daily incrementals take less than 2 minutes.
I really recommend setting "Daily Backups in each set = 7" (or 14, 21, 28 ... but 7 is good)
Cheers
Ian

This will also help - just set the full backup to be done on the weekend and daily incrementals to run on the rest of the days.

[CharlieBrady]

I need a backup solution that I can frequently do restores ...

Really? I think you should train your users to not delete data which they still need. Restoring from backup is timeconsuming.

[daniel]

I do alot of beta testing and software development for people, plus most of the files are database transactional files, they never get deleted, just changed, thus I need to restore to a previous state for testing again or fix a problem that wasn't forseen.  And sometimes people do screw up a spreadsheet that needs recovered. 

I'm testing using the workstation backup to backup to a share on a windows box now. SO far server load is staying under 2, thats good.  I'm also doing an agent backup across the LAN, but thats not 100% stable for me right now.  Right now I haven't found a great solution for online backup for SME Linux yet either, but working on testing some of them as well.

[paradigm]

I do alot of beta testing and software development for people, plus most of the files are database transactional files, they never get deleted, just changed, thus I need to restore to a previous state for testing again or fix a problem that wasn't forseen.  And sometimes people do screw up a spreadsheet that needs recovered. 

For software development you could use SVN and an SVN agent like tortoise , see here :

http://wiki.contribs.org/Subversion

For items deleted by mistake or screwd up by users you can use RecycleBin and ShadowCopy , see here :

http://wiki.contribs.org/ShadowCopy

http://wiki.contribs.org/RecycleBin

I'm testing using the workstation backup to backup to a share on a windows box now. SO far server load is staying under 2, thats good.  I'm also doing an agent backup across the LAN, but thats not 100% stable for me right now.  Right now I haven't found a great solution for online backup for SME Linux yet either, but working on testing some of them as well.

Affa could be used here to make quick snapshots every 30 minutes or so...

[janet]

daniel
 

For a Server that has 322GB of data on it, the backup takes 7.5 hours.  I started watching it late at night and wondered if this is normal for the USB Dar Backup.

That sounds about right, different systems and file types may cause some variation.

[daniel]

Thanks for the confirmation Mary.  I did some further testing, I ran SME backup to a windows workstation share, and it took the same amount of time as it did to backup to the USB2.0 external drive, about 7 hours for 320GB. Although I don't think its 320GB being backed up, thats the total storage of my server, but I thought the backup procedure didn't backup everything anyway, just the configs and data and customizations. 

I setup Symantec Backupexec 2010 running on my Windows servers to reach out and backup the SME box via the linux agent.  It backed up 270GB (I didn't select the entire drive, just what I thought the SME server was backing up according to what I read in the Wiki) and it backed it up in 4.5 hours. 

I saw in the SME 7.5 RC notes there was some updates to the backup. 
e-smith-backup
- Fix full backup on Sunday displayed as Everyday. [SME: 5624]
- workstation backup: add cifs credentials expand to bootstrap-console-save [SME: 4850]
- workstation backup: using credentials file for cifs mount [SME: 4850]
- workstation backup: allow many backups in the same day [SME: 5393]
- workstation restore: all needed backups must be available before restore [SME: 5395]
- workstation verify: add option to check integrity of backups needed in a full restore [SME: 5396]
- minor fix in translation [SME: 5406]
- Fix DAR e-mail message with regards to incremental backups [SME: 4579]
- Fix discrepancy in maximum compression level [SME: 4841]

I look forward to testing to see if these fixes have given me any performance improvements.

[janet]

daniel

See the "Backup config" wiki Howto article. You are best to use a sme backup & restore solution as the mysql databases are dumped, and proper pre and post signal events are run, for both backups and restores.

I don't quite understand why you need a full backup everytime. You can do one full backup (which takes a long time), and then numerous incremental backups  (which usually only take a few minutes depending on how much data has changed).
Using the GUI server manager Backup or restore panel options, you can selectively restore individual files, and you can select any date to restore your whole system to. The restore grabs information from the full backup and the chosen daily incremental backup to rebuild the server as it was on that day, except for installed applications (where they are not in ibays).

If you want a full and a incremental on the one "disk" to remove off site, then have multiple disks and initially run a full backup on each of them, then rotate the disks adding only (quicker to do) incrementals. Set the backup parameters appropriately of course.

Note tha Backup or restore panel gives you an estimate of the actual data being backed up, if you are unsure of the backup data size. My speed comments were based on that amount of total space used on your hard drive. If less, then it should be significantly quicker.

If you want quick backups on a regular basis, which includes the whole system, including applications, and assuming you are using RAID1, then put one drive into a removable caddie or front panel "HDD slot", and simply swap out one drive of the RAID array. Put the new drive in and run Manage disks (or similar name) from the Console to rebuild the array. Yes you do need to shutdown the server as current kernels do not support hot swap, but it will take less than 5 minutes, far quicker than waiting 7.5 hours.
You have to "zero" or blank the replacement drive if it already has data on it. Search the wiki and forums, it has been mentioned many times.

[cactus]

I saw in the SME 7.5 RC notes there was some updates to the backup. 
e-smith-backup
- Fix full backup on Sunday displayed as Everyday. [SME: 5624]
- workstation backup: add cifs credentials expand to bootstrap-console-save [SME: 4850]
- workstation backup: using credentials file for cifs mount [SME: 4850]
- workstation backup: allow many backups in the same day [SME: 5393]
- workstation restore: all needed backups must be available before restore [SME: 5395]
- workstation verify: add option to check integrity of backups needed in a full restore [SME: 5396]
- minor fix in translation [SME: 5406]
- Fix DAR e-mail message with regards to incremental backups [SME: 4579]
- Fix discrepancy in maximum compression level [SME: 4841]

I look forward to testing to see if these fixes have given me any performance improvements.

Not to disappoint you, but I see no performance related bugfixes... Backing up large amounts of data takes time as you already experienced.

Do you really need full backups every time? Otherwise you caould set up a scheme with full and incremental backups, which would speed up backing up at least for subsequent sets as only the difference needs to be archived.