Topic: How to syncronise 2 sme servers

[arne]

A small adjustment ..

This will recorsively copy all the content of the whole directory /arne from the remote PC to the directory /arne at the local PC.

rsync --archive --rsh=/usr/bin/ssh --verbose root@domene.com:/arne /

Looks like one ibay can be done using only one command.

Ok, I think there should be a script first dumping the approciate databases and the transfering files with rsync.

Arne

[Brenno]

arne,

Sounds like you're making progress there!  I just discovered why sync2nas won't work - it doesn't support SSH and therefore SME blocks the connections at the firewall level (rsync uses port 873.)  Not sure why I was under the impression that sync2nas would work

[arne]

The question: "how to open one port in the firewall" were up here int he forum for some days ago.

I answered it out and "someone" tested it out and confirmed that it worked. (I could't myself becaus I did not have the right equipement just now. You could try to search for it ..

[gelcube]

Then I started to develop a solution based on the use of the principle of making bacup using a number of cheap USB harddisks for full data backups.

The tasks were automated in such a way that you just had to plug in the usb harddisk for instance in the evening and pick up the full backup in the morning.

This solution were improved until I could not imagine how to improve it more, and then the whole project were packed down, because I really do not use such a backup capacity for my homeserver. The whole project were made mostly for "fun" and the exitement to see if it could work.

I don't know if usb harrdiks is considered to be a "safe enough" medium for backup, but it seems to work rather good. By the way for making it I hacked into the backupfunctions of the sme server to make bacups of the identical same files including dumps from the databases.

What about starting up such a project "backup to the usb harddisk" and then this time with someone with the knowledge of making a server-manager front end ?! (in my project it was just a collection of shell scripts.)

Is there anyway you could post these scripts and a how-to?  I'd be very interested in it...even without a server manager frontend.

[arne]

The question would rather be how and where to find it again. It was developed trough a discussion here at the contibs.org for a year or two ago, so it should be here as well, if one search for it.

Anyhow it should not be a to big task to redevelop it.

Actually a "backup to usb harddisk" and "syncronice two sme servers" could be two wery close projects, actually so close that both cluld be developed in one new project.

The difference could be that the one method mounts a usb harddisk while the other etsablish a rsync/ssh connection.

Most of the stuff using rsync to copy/syncronizing files, database bacup, restore etc would be common.

By the way used rsync for the usb hd as well as I were not able to find out how to use the rsync over the net at that time.

I have two servers at the moment, one sme 6.0.1 in my friends house (he got a lot bether internet line than me) and a centos server in my home.

I am thinking a little bit now if I should try to set up a sme server in my home as well and then try to make some automatic routines to syncronize these two servers.

If "somebody else" took the "check out of the usb part" of the project, I could eventuelly think about doing the "syncronize the servers project".

Actually 95 % of those two "project halves" could be identical with only minor "adjusments" for passing the datas to the usb harddisk or to the other server.

"Passing the datas to the other server" was that part of it I had to give up the last time, so it cluld be something ..

By the way another discussion about storing the datas to the usb harddisk is "should the datas be stored just as individual files reflecting an existing datastructure or should they be for instanse collected to one compressed file.

There is one thing about this rsync function, at least as it worked for me. It is not "bidirectional" in that way, it syncronize the one way, but not the other. That will say it will not delete all old garbage files on the usb backup if they are deleted on the server. All reviced files at the server will be copied over to the usb disc via rsync, but old files that are deleted on the server will not be deleted on the "rsync image". So if you have to do a restore you will restore all the newest files pluss all the old garbage that should be deleted.

Don't know what's the best way to handle this "delete the old files on the backup" problem.

Arne

[arne]

By the way the "get rid of the garbage problem" could be just a question of using the right parameters for the rsync .. dont know, but I can se that the "man rsync" has a lot of info.

Arne.

[arne]

Think I found something about the "garbage problem" here:

To delete files that are on the destination system but no longer on the source system, use the --delete option:

e.g.:

rsync -e ssh -avz --delete /lib/ root@myserver.com:/lib/

http://wiki.linuxquestions.org/wiki/Rsync

[blockbuster]

Arne you wrote:

This will recorsively copy all the content of the whole directory /arne from the remote PC to the directory /arne at the local PC.

rsync --archive --rsh=/usr/bin/ssh --verbose root@domene.com:/arne /

If you do this from a windows machine via putty you will copy all documents of /arne into the / directory on both the remote machine.

I don't see any code to get it to the local machine but would be happy to have it. I mean you need something to define as an ip adres to backup to?! or two ip's one to the internet router, and 1 to the internal network?!

[arne]

I think this one apeared to be the best variant (event though I havent tested it out to much yet - both works but  i believe that the last one work "best" as it also delete old files that should be deleted).

rsync -e ssh -avz --delete /lib/ root@myserver.com:/lib/

the pronsipple is very easy. the souce is allways to the left and destignation to the right in the adress. Ip is resolved to myserver.com automatic.

From local to remote:

rsync -e ssh -avz --delete /lib/ root@myserver.com:/lib/

From remote to local:
rsync -e ssh -avz --delete root@myserver.com:/lib/ /lib/

Syncronising the directory /lib/ recursively.

It might sount rather stupid, but I had never thought about the rsync as something that can be used at a Windows work station. I have only used it between Linux machines.

You will need a rsync and a ssh client ? How to install ?

Arne.

[cydonia]

It might sount rather stupid, but I had never thought about the rsync as something that can be used at a Windows work station. I have only used it between Linux machines.

You will need a rsync and a ssh client ? How to install ?

Arne.

Isn't it possible to rsync to a Samba share?

[blockbuster]

Installation (Windows 95/98/NT4/2k/XP)
To use rsync, you first have to have ssh running. Don't even think about using rsh, and you probably don't want to rely on the built-in rsync transport.

   1. Unpack rsync_full e.g. as c:\rsync.

   2. Edit the rsetup.bat file. This is a simple batch file that defines your username and home directory and path; modify the appropriate two lines.

   3. Open a DOS or NT command window and go to the appropriate directoy, e.g. cd \rsync

   4. Set the path etc. by running rsetup.


For complete how to goto:
http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html


According to the ssh client:

Wouldn't Putty, WinSCP3 or Terre Term Pro do the job?

[blockbuster]

Isn't it possible to rsync to a Samba share?

I think it's possible but the nice thing from ssh is that you can do it over an insecure network like the internet. The good thing about Rsync is that you do incremental backup so you don't have to push to many mb's over the internet which is taking to much time and too much bandwith.  

In my situation I live part time in Europe and part time in South  America, I am keen on doing backups over the two continents. Since my adsl in South America isn't that strong I am looking in this direction to backup.

Other way to follow via openVPN and SambaShares, but I don't like to depent on that too much since I haven't configured it that stable at the moment.

[Brenno]

I've almost got this working... almost.  I setup OpenSSH on XP Pro, then simply wrote a .bat file which contains the following:

Code: [Select]

@ECHO OFF
ECHO **********************************************************
ECHO rsync.bat - Batch file to start your rsync command (s).
ECHO **********************************************************

SET CWRSYNCHOME="c:\progra~1\OpenSSH\bin"

REM ** CUSTOMIZE ** Enter your rsync command(s) here

SET RSYNCCMD=rsync -e %CWRSYNCHOME%\ssh -av --delete
SET REMOTE=user@domain.com::user

c:
cd \
SET DIRS=testing
echo Backing up from C: drive: %DIRS%
%RSYNCCMD% %DIRS% %REMOTE%/c

set CWRSYNCHOME=

pause

(The "pause" at the end is just a thingy to keep the window from closing immediately.)

The idea is that you can then use Windows Scheduler to execute the .bat file whenever you want.

Trouble is, I get a password prompt for user, but once I enter the password, I get the following errors:

Code: [Select]

TERM environment variable not set.
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/tmp/rysnc-2.6.3/io.c(515)

This is where I get stuck.  No matter what I do, I can't get this to complete the transaction.  Anyone have any ideas?

[arne]

I did not even know it was a OpenSh server for windows, but I can now see that there is.

If you install the Windows OpenSSH server will you then also got a client ?

I found some info here:
http://freessh.org/windows.html

Best reg Arne.

[arne]

Brenno ->

No, it was not a openssh client (??) on that link (or is it ??)

Is there some place to download a openssh client ? (Or how do you set up the ssh trafic from windows ?)

Arne.