Koozali.org: home of the SME Server

[Announce] backup2usb alpha-release

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[Announce] backup2usb alpha-release
« on: December 09, 2005, 11:32:50 PM »
[Announce] backup2usb alpha-release

Will show up in my contrib area shortly:

http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/alpha/smeserver-backup2usb-0.0.1-a2dmay.noarch.rpm

This is an alpha release and is highly experimental. USE WITH CAUTION!! I expect this may not work for everyone. Please install on a test server only.

This contrib is identical in theory to backup2ws except instead of backing up to a workstation, it backs up to an attached USB hard drive. However, it is up to you to configure your own USB hardware and make sure your drive works.

For myself, here is what I had to do on my test server:

Add the USB drive to your server and see whether it gets recognized and on what device:

# tail -100 /var/log/messages

Dec  8 10:07:10 test kernel: Attached scsi disk sda at scsi1, channel 0, id 0, lun 0

# fdisk /dev/sda1

Press p
Press n
Press w

# mkfs.ext3 /dev/sda1

# reboot

# tail -100 /var/log/messages

Dec  8 10:49:07 test fstab-sync[3298]: added mount point /media/usbdisk for /dev/sda1

Test the USB drive

# mount /media/usbdisk
# ll /media/usbdisk
total 20
drwx------  2 root root 16384 Dec  8 10:42 lost+found

Finally once your USB drive is working from the command line, try the Backup to USB panel and report your results.

Darrell

Offline pickles

  • 10
  • +0/-0
[Announce] backup2usb alpha-release
« Reply #1 on: December 12, 2005, 12:10:18 PM »
Darrell,

I followed your instructions and everything has worked.  The only hiccup was the example text in the web panel states "/media/usbdisc".  It should read "/media/usbdisk".

I have so far performed a 911 backup with level 3 compression and a selective restore of one ibay.

I have disconnected/reconnected the USB drive and am now performing a full restore.

I'm running this on a pretty standard SME 7beta8.

Justin

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[Announce] backup2usb alpha-release
« Reply #2 on: December 12, 2005, 07:42:09 PM »
Quote from: "pickles"
Darrell,

I followed your instructions and everything has worked.  The only hiccup was the example text in the web panel states "/media/usbdisc".  It should read "/media/usbdisk".

Actually the panel reads:
Code: [Select]
Enter the mount point of your USB hard drive. Example: [/media/usbdrive]
I agree this should be changed to "/media/usbdisk". Will do so in my next build. Thanks for the testing and feedback :->

Also, I found a neat utility for mounting ext3 file system formated drives under Windows XP:

http://www.fs-driver.org/index.html

Tip. Make sure the drive is formated ext3 only. If it had a previous NTFS partition you have to zero the drive to remove all NTFS remnants, prior to partitioning and formating under SME. As my test server used /dev/sda for my USB drive, I had to do this (carefully entering the correct device!):

dd if=/dev/zero of=/dev/sda bs=512 count=63
dd if=/dev/zero of=/dev/sda1 bs=512 count=63

Then repeat the fdisk, mkfs.ext3 process mentioned above. After that the drive worked great under SME and XP with the above driver installed.

Darrell

Offline wjhobbs

  • ****
  • 171
  • +0/-0
    • http://www.chryxus.ca
[Announce] backup2usb alpha-release
« Reply #3 on: December 12, 2005, 10:31:12 PM »
Darrell,

That's great!

I note that this targets SME7. Do you know if it will work on 6.0.1?

John
...

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[Announce] backup2usb alpha-release
« Reply #4 on: December 13, 2005, 12:14:47 AM »
Quote from: "wjhobbs"
Do you know if it will work on 6.0.1?

Quick answer is Yes. I've confirmed it works on my test 6.0.1 server environment.

The harder part is getting your USB drive set up. It is a more complicated manual process in 6.x. You need to create your own mount point directory and add the correct entries to /etc/fstab. However, if you can get your hardware to work from the command line, the contrib will work.

Offline pickles

  • 10
  • +0/-0
[Announce] backup2usb alpha-release
« Reply #5 on: December 15, 2005, 11:25:31 AM »
Darrell,

I've just tried smeserver-backup2usb-0.0.1-a4dmay.noarch.rpm

Unfortunately I didn't get around to testing scheduled backups before upgrading from 0.0.1-a2.

So, after over-coming a small hiccup with the 'Save' button(I forgot to click it), everything is working fine.

BTW - Thanks for the tip on IFS Drive Control, that works great.

Justin

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[Announce] backup2usb alpha-release
« Reply #6 on: December 15, 2005, 04:01:55 PM »
Thanks for the continued feedback.

I'd also like to hear from unique USB hardware users. Particularly hardware such as hot-swap usb drive bays, Iomega REV drives, Storcase DE50, just to name a few. Anyone with this type of hardware, pease test and report.

I'm also thinking of merging backup2usb/backup2ws together into one rpm. Of course the new challenge will be what to call the single rpm. :-?

HarryM

backup2usb testing
« Reply #7 on: December 17, 2005, 04:51:03 PM »
Hay Darrell

I have test your backup2usb rpm. Works fine for 6.01 and 6.5.
Under 7beta8 i have a error in my messages log file:
Dec 17 16:29:46 test e-smith-bg: cannot handle file 'libc.so.6' with TLS data

I have no idea somewhat is wrong?
Did you have any idea. It was install on a clean install system.

Greet
Harry

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
Re: backup2usb testing
« Reply #8 on: December 18, 2005, 09:53:06 PM »
Quote from: "HarryM"
Hay Darrell

I have test your backup2usb rpm. Works fine for 6.01 and 6.5.
Under 7beta8 i have a error in my messages log file:
Dec 17 16:29:46 test e-smith-bg: cannot handle file 'libc.so.6' with TLS data

I have no idea somewhat is wrong?
Did you have any idea. It was install on a clean install system.

Greet
Harry


7.x requires a different rar version:

http://mirror.contribs.org/smeserver/contribs/dmay/smeserver/7.x/rar-3.5.1-7dmay.i386.rpm

Make sure you download then install with yum to get the required dependancy rpms:

yum install rar-3.5.1-7dmay.i386.rpm

Offline smitti

  • *
  • 27
  • +0/-0
    • http://smitti.mine.nu
backup2usb testing
« Reply #9 on: December 21, 2005, 02:48:58 PM »
Hi,

Thanx for this contrib.
I am testing it on 6.5 and works like a charm......

The drive i use a MobileDisk (thats the name on it)
It's a cheap usb 2.0 drive 3.5GB

Thanx again

Peter

Offline jester

  • *
  • 496
  • +1/-0
[Announce] backup2usb alpha-release
« Reply #10 on: December 22, 2005, 03:54:00 PM »
Hi Darrel,

Installation in SME7b8 went fine. Problem is i've got a second disk symlinked to an ibay as described by: http://no.longer.valid/phpwiki/index.php/extra%20hard%20disk

The files on this extra disk are NOT backed up! Hope this can be solved because i'd really love to use this contrib.


Kind regards,
jester.

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[Announce] backup2usb alpha-release
« Reply #11 on: December 22, 2005, 05:17:20 PM »
Quote from: "jester"
Hi Darrel,

Installation in SME7b8 went fine. Problem is i've got a second disk symlinked to an ibay as described by: http://no.longer.valid/phpwiki/index.php/extra%20hard%20disk

The files on this extra disk are NOT backed up! Hope this can be solved because i'd really love to use this contrib.


Kind regards,
jester.

Hi Jester,

The default rar backup does correctly back up symlinked files & directories. If your symlink to a second disk is correct and the disk is mounted, it should back up just fine. Easy to test and view results via the command line. Simply initiate a small rar backup and watch the results:

# rar a /root/test.rar /your_symlink_here

As an example, lets test a standard symlink we backup during the 911 job:

Quote
[root@test ~]# ll /home/e-smith/files/primary
lrwxrwxrwx  1 root root 33 Dec 14 15:39 /home/e-smith/files/primary -> /home/e-smith/files/ibays/Primary

[root@test ~]# rar a /root/test.rar /home/e-smith/files/primary

Updating archive /root/test.rar

Adding /home/e-smith/files/primary/files OK
Adding /home/e-smith/files/primary/cgi-bin OK
Adding /home/e-smith/files/primary/html/index.htm OK
Adding /home/e-smith/files/primary/html OK
Adding /home/e-smith/files/primary OK
Done

Here is a test backing up a symlink going to a second mounted drive, in this case an external usb drive:

Quote
[root@test ~]# ll /home/e-smith/files/mp3
lrwxrwxrwx  1 root root 16 Dec 22 09:30 /home/e-smith/files/mp3 -> /mnt/usbdisk/mp3

[root@test ~]# rar a /root/test.rar /home/e-smith/files/mp3

Updating archive /root/test.rar

Adding    /home/e-smith/files/mp3/file1.mp3                           OK
Adding    /home/e-smith/files/mp3/file2.mp3                           OK
Adding    /home/e-smith/files/mp3/file3.mp3                           OK
Adding    /home/e-smith/files/mp3/file4.mp3                           OK
Adding    /home/e-smith/files/mp3/file5.mp3                           OK
Adding    /home/e-smith/files/mp3                                     OK
Done

Offline jester

  • *
  • 496
  • +1/-0
[Announce] backup2usb alpha-release
« Reply #12 on: December 23, 2005, 03:15:42 PM »
Hi Darrel,

Using your commandline method i works like it should, but if i use the server-manger panel it does not work... Below are the relevant lines of my backup2usb log:

Quote
Adding    /home/e-smith/files/users/admin/home/.AppleDB/db_errlog         OK
Adding    /home/e-smith/files/users/admin/home/.AppleDB/lock              OK
Adding    /home/e-smith/files/users/admin/home/.AppleDB/cnid2.db          OK
Adding    /home/e-smith/files/users/admin/home/Temporary Items/.AppleDouble/.Parent      OK
Adding    /home/e-smith/files/users/admin/home/.AppleDouble/.Parent       OK
Adding    /home/e-smith/files/users/admin/home/.AppleDesktop/.volinfo      OK
Adding    /home/e-smith/files/users/admin/home/Network Trash Folder/.AppleDouble/.Parent      OK
Adding    /home/e-smith/files/ibays/projects                              OK
Adding    /home/e-smith/files/ibays/Primary/.AppleDB/db_errlog            OK
Adding    /home/e-smith/files/ibays/Primary/.AppleDB/lock                 OK
Adding    /home/e-smith/files/ibays/Primary/.AppleDB/cnid2.db             OK
Adding    /home/e-smith/files/ibays/Primary/html/index.htm                OK
Adding    /home/e-smith/files/server-resources/regedit/win98pwdcache.reg      OK
Adding    /home/e-smith/files/server-resources/regedit/winxplogon.reg      OK


My problem is with the 'projects' ibay and as you can see it sais that it adds: /home/e-smith/files/ibays/projects but none of the 2Gb data in that ibay. The rights on this symlink look ok:

Quote
[root@tiger ~]# ll /home/e-smith/files/ibays/
total 4
drwxr-xr-x  6 root root 4096 Dec 17 13:24 Primary
lrwxrwxrwx  1 root root   26 Dec 19 13:27 projects -> /mnt/array2/ibays/projects
[root@tiger ~]#
[root@tiger ~]# mount
/dev/mapper/vg_primary-lv_root on / type ext3 (rw,usrquota,grpquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/md1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
/dev/ida/c0d1p1 on /mnt/array2 type ext3 (rw,usrquota,grpquota)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
[root@tiger ~]#


Or im i missing something?

Kind regards,
jester.

Offline dmay

  • *
  • 450
  • +0/-0
    • http://myezserver.com
[Announce] backup2usb alpha-release
« Reply #13 on: December 23, 2005, 05:44:11 PM »
My mistake. I forget the rar options used in the action scripts and the reasons I used them. I had to take a look myself to remember. :-o

In the script backup2usb-backup or backup2ws-backup I use the option -ol

# man rar
-ol    Save symbolic links as the link instead of the file.

One example of why this is used instead of following links is due to the primary ibay:

[root@test usbdisk]# ll /home/e-smith/files
lrwxrwxrwx  1 root root   33 Dec 14 15:39 primary -> /home/e-smith/files/ibays/Primary

Both the link and the source are under the directory /home/e-smith. If we have /home/e-smith as a source for backup, and we followed symlinks, we would back up the primary ibay twice:

/home/e-smith/files/primary -> /home/e-smith/files/ibays/Primary
/home/e-smith/files/ibays/Primary

I could remove the option and set an exclusion, however, I believe for the majority of SME Server users the current settings are the better default.

In your case you have added a custom link and mount point. The default options will back up and retain the link. All that is remaining is for you to add the custom mount point to the backup sources to have this custom store backed up. In your case you would append /mnt/array2/ibays/projects or to back up the entire array, /mnt/array2.

Regards,

Darrell

Offline wjhobbs

  • ****
  • 171
  • +0/-0
    • http://www.chryxus.ca
[Announce] backup2usb alpha-release
« Reply #14 on: December 28, 2005, 06:36:34 PM »
Darrell,

First, the feedback....

I tried it on a fresh install of 6.0.1-01 using an ULTRA Mini Portable Disk case (purchased at TigerDirect) and a 300GB Maxtor drive installed and formatted as ext3. Tried on a test server and the 911 job worked well.

Update: On reboot e2fsck gets called and can't deal with the drive. The boot hangs. The message suggests I get a more recent version of e2fsck. I have to comment out the fstab entry to get boot to complete -- but the fstab entry is needed for the backup job. Fedora Legacy for 7.3 shows the same version as on this machine. Any suggestions for getting around this? The backup job has no problem with the drive.

Next the question(s)...

In setting up the system I had several false starts. For example, since I was just testing, I did a manual mount; but the job failed. It was only after I added the fstab entry that things went well. Then I went to look at the result and the drive appeared to be missing -- I presume you do an unmount at the end.

Also, for doing a 911 restore (I really want to test this before putting it on a production box) what is the restore process? Presumably fresh install, install backup2usb, and then restore -- but should default versions of the other contribs on the box be installed before the restore?

Is there a comprehensive set of documentation for setting up and using backup2ws/backup2usb? This would help a noob like myself to understand what is going on.

If there is a single place that has this kind of guidance, could you please point me to it?

This is a great piece of work!

Thanks, Darrell.

John
...