Topic: Slow backup to USB drive

[girkers]

My backup of about 550GB is taking over 24hours to backup but I don't think it should take that long, should it?

The results of read testing are:

Code: [Select]

[root@caine log]# hdparm -tT /dev/sde1

/dev/sde1:
 Timing cached reads:   4384 MB in  2.00 seconds = 2192.33 MB/sec
 Timing buffered disk reads:   92 MB in  3.01 seconds =  30.52 MB/sec

But obviously this is only the read speed and I couldn't work out a way to test the write speed. Could any advice as to how I may speed up my backup or am I destined to have a ssssslllllloooooowwwww backup.

Thanks

Girkers

[piran]

Oh... such temptation, a post like this from someone with 200+
postings logged quoting a computer services dot com ...ohhh;~)

Backups (type unknown, method unknown, extent unknown)
take many resources particularly if compression (extent unknown)
is utilised. Explain why do you think USB stuff is or should be quicker?

...how I may speed up my backup...
Use a faster drive/method/resource/mechanism?

Why should you need ...to test the write speed...  when
your backups are in real life ie not under test conditions.

Try monitoring your SME (version unknown) with HTOP
and see how resources are being used up by whatever.

[cactus]

The results of read testing are:

Did it ever occur to you that reading might be different to writing?

A little bit more details on your setup like piran already suggested would do no harm in us answering your questions.

[girkers]

piran,

I take your point, hopefully your tongue was firmly stuck in your cheek when you typed it 

Details,

I am using a standard SME 7.5 Server + Zarafa, backup is set to an external USB drive and to do a full backup every evening with no time out for full backups. What is happening the backup is set to start at 10pm each evening and once one starts it is not completing before the next one is due to start at 10pm the next evening.

I recently replaced my backup drives (USB) for larger ones due to running out of space on my previous drives and this is where the problem started. I attempted to do write speed testing of the USB drive using dd however I could never get it to work.

Here is what is contained in /var/log/messages when I plug in the drive:

Code: [Select]

Jun 14 17:22:23 caine kernel: usb 1-7: new high speed USB device using address 7
Jun 14 17:22:23 caine kernel: scsi7 : SCSI emulation for USB Mass Storage devices
Jun 14 17:22:23 caine kernel:   Vendor: WD        Model: 20EADS External   Rev: 1.75
Jun 14 17:22:23 caine kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Jun 14 17:22:23 caine kernel: SCSI device sde: 3907029168 512-byte hdwr sectors (2000399 MB)
Jun 14 17:22:23 caine kernel: sde: assuming drive cache: write through
Jun 14 17:22:23 caine kernel: SCSI device sde: 3907029168 512-byte hdwr sectors (2000399 MB)
Jun 14 17:22:23 caine kernel: sde: assuming drive cache: write through
Jun 14 17:22:23 caine kernel:  sde: sde1
Jun 14 17:22:23 caine kernel: Attached scsi disk sde at scsi7, channel 0, id 0, lun 0
Jun 14 17:22:23 caine kernel: USB Mass Storage device found at 7
Jun 14 17:22:23 caine scsi.agent[5186]: disk at /devices/pci0000:00/0000:00:1d.7/usb1/1-7/1-7:1.0/host7/target7:0:0/7:0:0:0

Here is what I would expect to see time wise, showing my work:
Data: approx 575GB = (575 x 1024 for gigabyte to megabyte conversion) 588800MB
Estimated write speed: 25MB/s
Raw backup time: (Data / Write Speed) 23552 seconds or (/3600 to convert to hours) 6.54 hours
Obviously there would be verification time however I still can't see how it could take over 24hours to do my backup.

So with all that could anyone help in working out why my backup is taking so long.

Thanks

[girkers]

It is funny how it only took 1 and 2 hours for people to critise my post, yet a week goes by and no one can help me. I wouldn't mind so much if after posting the extra information that people requested they at least do the courtsey of saying they can't help.

Can anyone suggest what may be slowing down my backup?

[Teviot]

It is funny how it only took 1 and 2 hours for people to critise my post, yet a week goes by and no one can help me. I wouldn't mind so much if after posting the extra information that people requested they at least do the courtsey of saying they can't help.

I'm sorry but I don't have the know how to help you either. But I will say that this is all to common within the community. I have been a member since v4 and have been victim of this sort of behaviour.

All I can suggest is ... persist and someone will help sooner or later

[girkers]

Thanks teviot I appreciate your thoughts.

[byte]

Here is what I would expect to see time wise, showing my work:
Data: approx 575GB = (575 x 1024 for gigabyte to megabyte conversion) 588800MB
Estimated write speed: 25MB/s
Raw backup time: (Data / Write Speed) 23552 seconds or (/3600 to convert to hours) 6.54 hours
Obviously there would be verification time however I still can't see how it could take over 24hours to do my backup.

So with all that could anyone help in working out why my backup is taking so long.

I have about 500GB of data and using the workstation-dar backup within the server-manager that takes about 20 hours backing up to a smb share.  From what I can work out I think its because you are compressing on the fly, much is to do with cpu resources before it even hits the drive, when you start a backup the data is in a temporary directory:

<server-name>.<domain-name>
tmp_dir

Once its completed the backup all the .dar files are then moved in to the <server-name>.<domain-name> directory.

What is your current compression ?

So depending on the size of the hard drive you have in the USB caddy you might be able to use 0 or 1 compression.

[girkers]

The thing was that the backup has only started going slow once I changed to a bigger backup drive, none of the other settings had changed. I have a 1TB drive previously but as the backup was now over 500GB it was not big enough, thus the need for the change. I had the compression level set at 8 and the backup would take about eight ( hours normally, but the time now is ridiculus.

To test the theory I have changed the compression level to 0 and have set the backup to start in five (5) minutes, I see how it goes.

[girkers]

I had changed the compression to 0 to test the theory of the problem being CPU bound, but alas this is not the case. I still believe it is throughput (write speed) of the USB drive that is the problem how I can not determine the speed of the drive as the dd commands I have tried don't seem to give all the information.

Any other thoughts?

[janet]

girkers

I had a somewhat related issue recently. I was transferring backup files from a USB drive to a mounted SATA drive. The copy was going extremely slowly. Upon investigation the transfer rate was only like 1Mb/s. Both drives were obviously detected OK etc.
I disconnected the USB drive, powered it down, waited a bit and reconnected it.
When I copied the files again between mounted drives I had more like 27Mb/s rate.

I just used mc to move the files (approx 200 1.5Gb sized files), and it showed the transfer rate on the mc screen.

So you could check real world transfer rates that way, and perhaps try reconnecting the USB drive.

I didn't bother to investigate why it ran at such a slow speed as that is the only time it had ever happened.

[girkers]

I just did a test with MC and found that the write speed of the drives starts out great but then slows down, so it would appear that the drive is definately the problem. I have another drive that is identical that I haven't used yet that I am just formatting and setting up. I did a quick read about block size and on the original drive I did not set any, so on this 2TB drive I have set it to 4096.

Could anyone else suggest if the format of the drive would have any bearing on the speed, or any other suggestion what so ever?

[janet]

girkers

Write speed slows down to what ?
Format your drives as ext 3.

[girkers]

Thanks for the help mary.

The drive is formatted as ext3 as I follow http://wiki.contribs.org/USBDisks, here is the output of the new drive I just connected:

Code: [Select]

[root@caine media]# mkfs.ext3 -b 4096 -L Backup /dev/sde1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=Backup
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
244203520 inodes, 488378000 blocks
24418900 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=490733568
14905 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Is there a definitive way to test the write speed as I have said the dd command to test write speed doesn't tell me like it normal show it should on the net. I will see how this drive performs overnight. And if still slow I will test with MC and give speeds and file sizes.

[byte]

iostat will give you a helping hand to see if there are any bottle necks.