Topic: Affa 0.9.0 performs full backup every time?

[dmcguire]

Hi,

First I am new here, although I have been using SME Server for about 1 year now (Linux approx 2 years). I find SME excellent and I'd like to say a big thank you to all who helped to make it what it is.

I am having an issue with Affa, but it could be something silly that I'm doing wrong as this is also the first time I have used a contrib outside the standard SME Server install, so apologies in advance if this is a stupid question, but maybe someone can help...

Scenario: I have two Windows Server 2003 boxes (fileserver1 & Exch-Server) running Cygwin rsyncd, and one SME Server 7.3 with Affa for backup. SME is doing only backup (server only mode) nothing else.

I have set up & scheduled two jobs which when run copy the D$ of each windows box. Affa --list-archives shows;
Affa version 0.9.0 on backup.company.lan (192.168.1.250)
+------------------------------------------------------------------------------+
| Job: exch-server-d                                                           |
| Description: Exch-server d drive backup                                      |
| Directory: /var/affa/exch-server-d/                                          |
| Hostname: 192.168.1.21                                                       |
| Email: admin@company.lan                                             |
+-----------------------+----------------+--------------+--------+-------------+
| Date                  | Archive        |        Files |   Size |  Disk usage |
+-----------------------+----------------+--------------+--------+-------------+
| Fri 2008 Mar 07 21:58 | daily.2        |       202026 |  106GB |   318GB/73% |
| Sat 2008 Mar 08 02:31 | daily.1        |       202122 |  106GB |   370GB/85% |
| Sun 2008 Mar 09 02:31 | daily.0        |       202173 |  106GB |   418GB/96% |
+-----------------------+----------------+--------------+--------+-------------+
Affa version 0.9.0 on backup.company.lan (192.168.1.250)
+------------------------------------------------------------------------------+
| Job: fs1d                                                                    |
| Description: fileserver1 d drive backup                                      |
| Directory: /var/affa/fs1d/                                                   |
| Hostname: 192.168.1.181                                                      |
| Email: admin@company.lan                                             |
+-----------------------+----------------+--------------+--------+-------------+
| Date                  | Archive        |        Files |   Size |  Disk usage |
+-----------------------+----------------+--------------+--------+-------------+
| Fri 2008 Mar 07 14:40 | daily.2        |       639911 |  284GB |   223GB/51% |
| Sat 2008 Mar 08 00:36 | daily.1        |       640570 |  287GB |   369GB/85% |
| Sun 2008 Mar 09 00:28 | daily.0        |       640572 |  287GB |   417GB/96% |
+-----------------------+----------------+--------------+--------+-------------+

As you can see, the disk simply fills up with a full backup every time, but I need full first, then incremental (or differential) as required. Only 6 Gig of data gets changed every day.

In /root/exch-server-backup-job.pl (fileserver-backup.pl is similar setting) I have the following;

'Description'=>'Exch-server d drive backup'
'scheduledKeep'=>1
'dailyKeep'=>7
'weeklyKeep'=>4
'monthlyKeep'=>12
'yearlyKeep'=>1

So affa works OK (full backup), but I maybe have it configured wrong. Can anybody offer a few pointers here?

Thanks in advance,
Dec

[m]

On Sat 2008 Mar 08 02:31 after the exch-server-d job your disk usage increased by 1G from 369GB to 370GB. This doesn't look like a full backup as the backed up data size is 106G.

On Sun 2008 Mar 09 00:28 after the fs1d job your disk usage increased by 47G from 370GB to 417GB. This also doesn't look like a full backup as the backed up data size is 287GB here.

Either your 6G estimation is wrong and there was more data modified or you have the clocks on the Affa server and the fileserver1 not in sync. If the latter is the case, try the 'rsync--modify-window' property and have a look at this bug report: http://bugs.contribs.org/show_bug.cgi?id=3462

[dmcguire]

Well spotted, you are correct, the clocks were out of sync by several hours. I will make suggested modifications & post results here. Thanks for the tips, much appreciated!

[m]

and you should consider adding larger disks to your Affa server. Just a simple estimation:
The initial data volume to backup is roughly 400G.
After a year you will have 1+7+4+12+1=25 archives.
Expected new/changed data every day is 6G:
25*6G=150G
Estimated archive overhead is 3%:
25*400G*3%=300G
makes 400G+150G+300G=850G after a year, which is almost two times more than your current disk capacity.

[kevinb]

Just some friendly advice dmcguire,

We tried a restore from a backup on a Windows share and I struggled with permissions problems. Getting the failed server fixed (hard drive), installing SME, installing updates, installing the contribs and restoring the backup cost us about four hours of down time (this did not include troubleshooting, running out and getting a new hard drive or messing with permissions).

We then grabbed an old computer (Pentium 800, 256 Mb RAM) installed SME on the 10 Gb hard drive. Added two more 500 Gb hard drives configured as software RAID 0 (for 1 Tb of disk space) and a second NIC. We then use this computer for backups of our main server. The RAID 0 is mounted to an ibay and shared to domain admins only. DA's can pull individual files from the historical backups.

As a restore test we did an "affa rise" on the backup box, reconfigured the NIC cards and in less than 10 minutes the backup server was running as our main server (much slower but it did the job). We then reversed the process to get the main server up to date and back on line.

We now have a second backup server offsite doing the same thing.

Affa is a really great contrib! Thanks mweinber!

Kevin

[dmcguire]

and you should consider adding larger disks to your Affa server.

Thanks mweinber, I've decided to run only one backup job to the SME Affa box. It's a Dell Optiplex PC with 2x500 Gb IDE drives mirroring eachother, so I cannot extend capacity. If successful, I'll apply for funds to build a 2 Tb box. I'm going to remove the existing jobs/archives and start over on Friday, then let it run for the weekend. Will post my results here.

We tried a restore from a backup on a Windows share and I struggled with permissions problems.

Thanks kevinb, I note what you say and I recall reading this also in the Wiki somewhere. In this case almost all of the data is accessible to everyone on the LAN so resetting the ACL's should not be a major issue. I'm purely concerned with the data. Should either Win2k3 server fail it would be replaced with new anyway. Management accepts that there would be significant downtime were this to happen. Unfortunately we need to run Win2k3 for the finance department software, otherwise I'd run on SME.

Mostly I want to locate the SME Affa box(es) in a separate building nearby so as to have geographical diversity if there is a fire etc. in the main building.

I like what you did with your 10 minute fail-over though, and may try something similar simply as a learning exercise if I can get the time. Ideally a cluster running SME!

Thanks again to both of you for advice & I'll post my results next week.

Belated question: What units does the  rsync--modify-window property use? seconds, milli-seconds?

[kevinb]

Sorry dmcguire,

I did not read your post thoroughly. We were backing up a SME server to a Windows box (which did not work well) and not the other way around.

Our setup:
- SME    -    Domain controller, gateway and file server
- Centos 5.1    -    Scalix server (Exchange replacement for Outlook), Authenticates against the SME server
- Windows 2000 Server    -    For Accounting software, Authenticates against the SME server
- SME    -    Backup server for all of the above


BTW ... does anyone know how many users can mount the same share on Windows XP at the same time?


Kevin

[raem]

kevinb

... does anyone know how many users can mount the same share on Windows XP at the same time?

10

[dmcguire]

OK, I removed the two jobs, all the archives, and created a new job backing up only the one box "mailserver".

At first, I had a problem on the Win2k3 server running Rsyncd on Cygwin. It wouldn't start because I had a windows program called "Delta Copy" installed, which was using an older version of the Cygwin DLL. Once this was removed, and the Win2k3 box rebooted, the rsync daemon worked perfectly.

With the time correctly synced between the SME/Affa box & the "mailserver" box, the Affa backup worked nicely.

Affa --list-archives now shows;

Affa version 0.9.0 on backup.company.lan (192.168.1.250)
+------------------------------------------------------------------------------+
| Job: mailserver                                                              |
| Description: mailserver d drive backup                                       |
| Directory: /var/affa/mailserver/                                             |
| Hostname: 192.168.1.21                                                       |
| Email: admin@company.lan                                                     |
+-----------------------+----------------+--------------+--------+-------------+
| Date                  | Archive        |        Files |   Size |  Disk usage |
+-----------------------+----------------+--------------+--------+-------------+
| Sat 2008 Mar 15 03:11 | daily.2        |       205220 |  109GB |    98GB/22% |
| Sun 2008 Mar 16 02:31 | daily.1        |       205268 |  109GB |    98GB/22% |
| Mon 2008 Mar 17 02:30 | daily.0        |       205314 |  109GB |    98GB/22% |
+-----------------------+----------------+--------------+--------+-------------+
| Tue 2008 Mar 18 02:30 | scheduled.0    |       205401 |  109GB |    99GB/22% |
+-----------------------+----------------+--------------+--------+-------------+

As there was nobody in the office over the weekend (Monday was a holiday here), almost no data went through mailserver, but the backup is running well.

I'm really happy with this now, it's doing exactly what I need, and the Affa box is located in a different building so I have a little geographic diversity on the data also.

Thanks for this contrib mweinber, it's a fantastic piece of code which solves a tricky problem for me. It's also the first contrib I have used, so a learning exercise too