Topic: Faster Backup than DAR?

[Mirfster]

I know I have been asking a lot of questions lately and I want to say that I am thankful for all the great advice I have gotten from this community. 

With that said, here is my latest question... 

Is there a faster backup option other than DAR or something that can be modified to increase DAR's speed?
* I have already tested setting compression to 0 and it does not increase speed
** I have read http://wiki.contribs.org/Backup_with_dar#NFS and http://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Chapter10#Backup_or_restore

Here is the scenario:

SME Server
Running as a VM Instance on ESXi 5.5 Update 3
2 CPUs allocated
8 GB Ram allocated
2 - 1 GB NICs (Running in FailOver)
2 - 10 GB NICs (Running in FailOver)
6 TB Hard Drive allocated (Is via a ESXi DataStore which uses a PERC H700 running in RAID 6)
All Current Updates Applied

FreeNas Server
Chassis: Dell PowerEdge C2100 Server
Build: FreeNAS-9.3-STABLE-201511040813
CPUs: 2 - Intel Xeon E5506 SLBF8 2.13GHz/4MB/4.8GTs/Quad Core LGA1366 CPU
Memory: 24 GB (6x4GB - 2Rx4 PC3-10600R - DDR3 - ECC)
HBA: Dell H200 Mezzanine - Flashed to LSI 9211-8i IT Mode
Network (1 GB): 2 - Intel 82576 GB Ethernet
   Running in "Fail-Over" LAGG
   Used just for Production Access
Network (10 GB): Chelsio 110-1088-30 10GB 2-Port PCI-e HBA Adapter Card with SFP's
   Running in "Fail-Over" LAGG
   Used just for Backup Routine Access
   Is Direct Connected to ESXi Server w/Static IP Assigned (No Switch Involved)
OS Hard Disk(s): 2 - 120 GB SSD (were a good deal, so I grabbed them...)
Storage Hard Disk(s): 12 - Hitachi Ultrastar HUA723030ALA640 3TB
   Enterprise Rated 7200RPM 64MB SATAIII (6Gb/s) 3.5"
Volume: Composed of 2 RAIDZ2 w/6 Disks in each one

Now, with everything up and running I did some preliminary tests to get an idea of speeds:

iPerf from SME Server to FreeNas Server (Got 8+ GB/Sec, so I was happy about that):
Command: iperf -p 5001 -c 172.20.1.6 -w 512k

Results:
------------------------------------------------------------
Client connecting to 172.20.1.6, TCP port 5001
TCP window size: 244 KByte (WARNING: requested 512 KByte)
------------------------------------------------------------
[ 3] local 172.20.1.5 port 41214 connected with 172.20.1.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.47 GBytes 8.13 Gbits/sec

DD (Write test) on FreeNas (64 Gb File - Got ~805 MB/Sec, happy about that too)
Command: dd if=/dev/zero of=tmp.bin bs=1m count=64k && sync
*** Have a screenshot, can insert if needed.  Otherwise you will have to take my word for it...

Now onto the backup...

Initial "Full Backup" was performed (all via the 10 GB Pipe) and I was surprised by how long it took:

Report from SME Server (Looks like just below 14 hrs):

==================================
DAILY BACKUP TO WORKSTATION REPORT
==================================
Backup of [name removed] started at Wed Nov 11 03:13:04 2015 Destination //172.20.1.6/SMEBackupsCIFs/[name removed]/set1
No existing reference backup, will make full backup Basename full-20151111031304 Starting the backup with a timeout of 24 hours

--------------------------------------------
354816 inode(s) saved
including 0 hard link(s) treated
0 inode(s) changed at the moment of the backup and could not be saved properly
0 byte(s) have been wasted in the archive to resave changing files
0 inode(s) not saved (no inode/file change)
0 inode(s) failed to be saved (filesystem error)
293 inode(s) ignored (excluded by filters)
0 inode(s) recorded as deleted from reference backup
--------------------------------------------
Total number of inode(s) considered: 355109
--------------------------------------------
EA saved for 0 inode(s)
--------------------------------------------
Destination disk usage 473G, 3% full, 16T available Backup successfully terminated at Wed Nov 11 14:56:31 2015

What has me puzzled is say even if I was only getting 200 MB/Sec, then I would think that the "Full Backup" routine (which is ~500 GB) would be:

200 MB/Sec * 60 Seconds = 12,000 MB/Minute
12,000 MB/Minute * 60 Minutes = 720,000 MB/Hour
720,000 MB / 1024 = 703 GB/Hour

So with 500 GB, I could safely presume < 1 hour. Heck even if it was 2 hours, that would be worlds better than the 13+ that is currently showed...

Looking at "Incremental Backups", they seem to be taking about ~ 13 minutes when there is only 2 - 3 GB of data.
While this is all faster than what used to be there (SME to Drobo via 1 GB), it is not exactly on par for the hardware assigned to it.

I understand that I could try out NFS instead of CIFs (and still may), but even if CIFs is single threaded I fail to see why the backups are so long?  I also have a similar system on loan where their SME Server is backing up to a similar FreeNas Server, but only over a 1 GB NIC.  Their initial "Full Backup" was of 351 GB and took ~ 12 hours, so my concept of using a 10 GB Pipe

Is this basically a limitation of DAR or am I missing something?

As always thanks for any input

[Daniel B.]

I don't think it's a limitation of dar. Every backup tools hit more or less the same issues. The bottleneck is most likely not your network speed, but your hard drives. And you cannot just mesure raw read throughput: backups do a lot of metadata reads (to check which file must be saved), this means a lot of random access. The best you can do to increase speed is to provide faster storage (SAS 15k, prefer RAID10 over plain 6, or maybe RAID60, in any case, the more spindles, the better. Or even go with SSD if you can afford)

[DanB35]

Another avenue is to see if your hypervisor has a backup solution that may be relevant.  I recently migrated my SME installation to Proxmox 4, with storage on ZFS.  Proxmox supports ZFS snapshots and replication (among other methods) for backups of VMs.  The initial backup of my SME VM, about 350 GB, ran at near wire speed over GbE, and took about 90 minutes.  Incremental backups are made pretty much "free" by ZFS.

I don't think ESXi does ZFS, but there may be some other backup feature that would work better.

Edit:  Although DAR has been generally stable for me, I've never found it to be particularly fast, even with 0 compression over NFS (and it's backing up to a compressed dataset anyway, so I didn't see any harm in disabling compression on the backup).  Before I virtualized, I upgraded both my SME server and my FreeNAS box with 10G Chelsio NICs; it made no difference in backup throughput.

[Mirfster]

I don't think it's a limitation of dar. Every backup tools hit more or less the same issues. The bottleneck is most likely not your network speed, but your hard drives. And you cannot just mesure raw read throughput: backups do a lot of metadata reads (to check which file must be saved), this means a lot of random access. The best you can do to increase speed is to provide faster storage (SAS 15k, prefer RAID10 over plain 6, or maybe RAID60, in any case, the more spindles, the better. Or even go with SSD if you can afford)

Possibly, but I was measuring actual write speed with the DD Command and got ~ 800MB/Sec.  So when I did an estimate of 200MB/Sec I was actually cutting those numbers down to 1/4th.

[Daniel B.]

Raw read or write throughput is nearly meaningless. First, write speed will be faked by the write-back caches (at the guest OS level and the hardware controller), then, random access means a lot of seeks, on which global throughput has almost no effect, only faster storage can help (more spindles, 15k SAS drives or SSD, more performance oriented RAID levels)

[Mirfster]

Another avenue is to see if your hypervisor has a backup solution that may be relevant.  I recently migrated my SME installation to Proxmox 4, with storage on ZFS.  Proxmox supports ZFS snapshots and replication (among other methods) for backups of VMs.  The initial backup of my SME VM, about 350 GB, ran at near wire speed over GbE, and took about 90 minutes.  Incremental backups are made pretty much "free" by ZFS.

I don't think ESXi does ZFS, but there may be some other backup feature that would work better.

Edit:  Although DAR has been generally stable for me, I've never found it to be particularly fast, even with 0 compression over NFS (and it's backing up to a compressed dataset anyway, so I didn't see any harm in disabling compression on the backup).  Before I virtualized, I upgraded both my SME server and my FreeNAS box with 10G Chelsio NICs; it made no difference in backup throughput.

Yeah seeing similar results.  I could actually just take snapshots and back those up to FreeNas or I could virtualize SME on FreeNas itself and even take snapshots there.  But would want to still have backups from SME/DAR so that they could perform restore of specific files if needed. 

Still mulling over the "SharedFolders" contrib and enabling the Recycle Bin or if I should house the data instead on FreeNas and use "Periodic Snapshots" which will provide "Shadow Copies" where they can restore files in Windows Explorer...

[Stefano]

well.. if you have a backup tool at virtualization level, to restore easily a single file I suggest you to use backuppc

[Mirfster]

Raw read or write throughput is nearly meaningless. First, write speed will be faked by the write-back caches (at the guest OS level and the hardware controller), then, random access means a lot of seeks, on which global throughput has almost no effect, only faster storage can help (more spindles, 15k SAS drives or SSD, more performance oriented RAID levels)

Understood, but just trying to run the scenario through mentally (please don't take this as a rebuttal, just pondering)..

Given that the connection speed is not the bottle neck and "Incremental Backups" are ~ 3 GB of data.  Then with the FreeNas Volume having separate vdevs (2 x 6 Drives) in a RAIDZ2 (Similar to Raid 6) as well as 24 GB of ECC Ram; I am failing to see how copying ~ 3GB of data could take ~ 13 minutes.  Later today (if I can RDP to a machine on the network) I will do a simple file copy (will vary the sizes) from a Windows machine directly to the FreeNas CIFs share (will be over a 1 GB connection) to see what speeds I get via that route.  Not being facetious, but I would be willing to bet that it does not take as long; which would still leave me to think that the bottle-neck is with DAR...

If my assumptions are incorrect, please feel free to let me know.  I won't get upset.   

[Daniel B.]

I am failing to see how copying ~ 3GB of data could take ~ 13 minutes

Transfering the data is the easy and fast part. It's not taking 13 minutes in itself. Checking which files should be part of the incremential backup is the costy part, requiring a lot of metadata reading. Note that I'm not talking especially about dar, this will be true for nearly every file based backup method (tar, rsync etc...)

Backups at the block layer level are different, they will be super fast because there's no metadata reading or anything to compare: just read raw data and send it on the other side. Modern filesystem like ZFS and btrfs push this to the next level because of their COW way of working, they can transfert efficiently deltas between 2 snapshots, allowing for incremential block level backups

[Mirfster]

well.. if you have a backup tool at virtualization level, to restore easily a single file I suggest you to use backuppc

Looked into it a little bit, but I was thinking that this is more geared towards backing up PCs as opposed to backing up the iBays/Data on the SME Server?  As far as User's PCs, I am not too concerned with that. They should be saving data to the Network Shares which are backed up.

Of course, I could have this all wrong.

[Stefano]

in some SME of mine, I install backuppc and let it backup itself.. I can restore single files from anywhere, via web GUI

[Mirfster]

Interesting, will put it on my VM SME.  Man I got a lot of testing to do. 

[ReetP]

Man I got a lot of testing to do. 

 

[PlusPlus]

I use vBackup to Backup the full vm on esxi free:
http://www.thinware.net/Products/ThinwarevBackup/tabid/202/Default.aspx

If i want to restore a single mail (1x last 6 years), i have to run the backup of the saved vm, give them an other ip adress und connect via ftp to get the mail(s).

The Backup of 329GB takes 4 hours, 22 minutes and 17 seconds,  Transfer Rate: 21,5 MB/s
Local USB 3.0 Disk