Koozali.org: home of the SME Server

Rsync 5.1.2 question.

Kees

Rsync 5.1.2 question.
« on: February 01, 2002, 02:06:46 PM »
Hi All,

I'm trying to rsync the new version. since I live in the country where you are disconnected after 2 hours online, I thought I'd use the --partial parameter in rsync.
( as the man suggests) to recover from an incomplete download.

However, it does not seem to work. (as in: I dont' know how I can make it work)

This is what I do:

rsync -avv --statts -P rsync:ftp...........
rsync -avv --statts --partial --progress rsync: .....

-P stands for ( man rsync)  --partial --progress, and is a valid shortcut says the rsync author. Neither version seems to work.
I get a partial file on my disk, no problems there. Do I miss something obvious?

Any suggestions are welcome..

kees

Charlie Brady

Re: Rsync 5.1.2 question.
« Reply #1 on: February 02, 2002, 01:31:53 AM »
Kees wrote:

> This is what I do:
>
> rsync -avv --statts -P rsync:ftp...........
> rsync -avv --statts --partial --progress rsync: .....
>
> -P stands for ( man rsync)  --partial --progress, and is a
> valid shortcut says the rsync author. Neither version seems
> to work.
> I get a partial file on my disk, no problems there. Do I miss
> something obvious?
>
> Any suggestions are welcome..

Try again. We've had some problems with our rsync daemon - it's been sacked and we've hired a new one.

[ But if you were using a local mirror site (as you should do), then that can't be the answer, can it?]

Mail me directly if you continue to have problems.

Regards

Charlie

Kees

Re: Rsync 5.1.2 question.
« Reply #2 on: February 03, 2002, 01:54:17 AM »
Micro-HOWTO download a smeserver.iso when your isp keeps disconnecting you.

If you have an unreliable connection to the net, than you might find the following useful. (Do read the information about rsync on the Mitel website first!)

If you do have (even if you do not*) a previous version of the sme server, use that as your seed to start an rsync session. I use the following magic words in my rsync-script. ( putting it in a one-line script beats retyping)

# my download script, filename: download.sh
#
rsync -avv --stats -P --timeout=0 rsync://ftp.e-smith.com/e-smith/e-smith/current/iso/smeserver-5.1.2.iso  /mnt/optical/sme512.iso

# end of download script
# usage:   sh download.sh

The rsync bit is all one line. My downloaded file is called sme512.iso, use your own names and directories here, and of course, use your local mirror!
 
The -P is an important change from the standard one on the Mitel website. It stands for '--partial --progress'. Adding it keeps the (a) temporary download file, if the connection is dropped. Adding the --timeout=0 seemed a good idea too, it sets the timeout to infinite.
(note: I'm pretty sure the -P option is responsible for keeping my partially downloaded file. In the past I had an terrible time keeping a partially downloaded file. Every time I would restart rsync it would start at zero bytes, hence the trick with the automagic copying of files, (see below). I'm downloading at the moment, so I'm not going to experiment!)

Sooner or later your connection is dropped. Now DON'T PANIC.

You will find a hidden file in the format .sme512.iso.xxxx in the same directory as you were downloading into, where xxxx is a random string of characters. This extension will be different every time you start a new download session. ( Every time your ISP hangs up, and you reconnect, you get a new IP address. So as far as the Mitel daemon is concerned you are a new user, and you get a new identity. That's why normal ftp clients seldom work (the ones I use anyway), when you try to restart the download, after being disconnected. You are presenting yourself with a new IP address, while trying to download stuff belonging to the old IP address. The daemons on the sending side, rightfully conclude you are an imposter!

Do a:
>mv .sme512.iso.xxxx  sme512.iso

If your script is still waiting, ctrl-C it. Restart it, and see, to you amazement, that after a (few)minute(s)or so it will continue where it left off!

Since I am a very suspicious person I also use another little script to make copies of the downloaded bits of file, in case something decides to delete it.

# my tiny copy script rev 0.0.1

cnt=100
while :; do
   cp /mnt/optical/.sme512.iso.*  /home/smeserver-5.1.2.iso.$cnt
   cnt= expr $cnt +1
   sleep 600
done
# end of my tiny copy script

Of course you get loads of downloaded files, so you might want to delete the ones you do not need every time your connection is dropped. This is very much a hands on affair, but since I cannot restart the connection automagically, that does not matter to me. Below is an alternative.

# my tiny copy script rev 0.0.2
# this one will keep writing to the same file. If your system crashes you loose all!
while :; do
   cp /mnt/optical/.sme512.iso.*  /home/smeserver-5.1.2.iso
   sleep 600
done
# end of my tiny copy script
 
As the saying goes, YMMV and I'm sure one day I'll develop have a fully automagical script. Right now, this works, 20 odd interrupts and 74% downloaded!

(*if you have nothing to start with, do:
> touch sme512.iso  to create a file without contents.)

I know rsync is not intended to do a download form scratch, but since it works, why worry?
Your comments and/or suggestions are of course always welcome!

kees

Kees

Re: Rsync 5.1.2 question.
« Reply #3 on: February 03, 2002, 02:05:32 AM »
Talk about reinventing the wheel!

Thought I just check what the e-smith page is so I can include a link in my micro Howto.....

http://www.e-smith.org/docs/howto/rsync-mini-howto.php3

Right at the bottom it says:

Additional command-line options to consider:
  -z (compress):
  -P --partial: The --partial option instructs rsync to leave any partial file(s) that might have been created during an interrupted download.


Ah well, at least I can claim I re-invented the wheel myself   ;-)
(and last time I checked (V 4.1.x) it was NOT THERE)
As they say: Always check for updates, it just might save you some time, then again, at least I learned something....

77% and going strong....