Topic: can rclone be used on SMEserver?

[brianr]

I am interested in using rclone (or similar) to backup ibay data to the cloud (Dropbox in my case).
I am sucessfully running rclone from my Fedora 27 desktop, and wonder if we could run rclone in the server?

I am not looking for a full implementation, just an indication whether someone out there has tried this already and perhaps knows whether it WON'T work, or that it might?

If the latter I'll have a go.

[mmccarn]

I don't know about rclone, but you could backup ibays and user folders directly with the dropbox linux client.

These instructions were written for SME 8; SME 9 comes with python 2.7 so you can ignore anything related to setting up or using python 2.7:
https://wiki.contribs.org/Dropbox

Here are some abbreviated instructions (I tested this in 5 - 10 minutes on my server):
1. Download and install the dropbox client (by default into into /root/.dropbox):

Code: [Select]

cd ~ && uname -a |grep "x86_64" && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf - || wget -O - "https://www.dropbox.com/download?plat=lnx.x86" | tar xzf -

2. Download the dropbox control script

Code: [Select]

mkdir -p ~/bin
cd ~/bin
wget -O dropbox.py https://www.dropbox.com/download?dl=packages/dropbox.py
chmod 700 dropbox.py

3. Start the dropbox client - you'll get a URL on your SME server that you need to visit from an authenticated browser on another system:

Code: [Select]

dropbox.py start
4. Once you've 'authenticated' the SME server with Dropbox, your existing dropbox data will sync to ~/Dropbox.  Create a symlink in that folder to your Ibay and dropbox will back it up:

Code: [Select]

cd ~/Dropbox
ln -s /home/e-smith/files/ibays/<ibayname> .

Finally, if it all looks like it's working OK, create a crontab entry to restart the dropbox cli at reboot.
* run "crontab -e" as root (or the user you've been using for all of this)
* add this line to it:

Code: [Select]

@reboot /root/bin/dropbox.py start

Over time you may want to upgrade the dropbox client.  I didn't test these today, but I did when I set it up back on SME 8:

Code: [Select]

cd ~
dropbox.py stop
uname -a |grep "x86_64" \
&& wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf - \
|| wget -O - "https://www.dropbox.com/download?plat=lnx.x86" | tar xzf -
dropbox.py start

Notes:
* The "uname" bit in the client download command is just about picking the correct 32-bit or 64-bit client for download
* You can include sql backups by symlinking /home/e-smith/db/mysql/ and running "signal-event pre-backup" from time to time (mysql55 via SCL backs up to /home/e-smith/db/mysql55)


[edit]
You can prevent your entire dropbox from syncing back to the SME server like this:
- start the dropbox client and wait for it to start syncing
dropbox.py start

- configure dropbox on SME to exclude the files and folders currently stored in Dropbox:
cd ~/Dropbox
dropbox.py exclude add *


- *then* add the symlinks to the folders you want to backup
cd ~/Dropbox
mkdir smeserver
cd smeserver
ln -s /home/e-smith/files/ibays/<ibayname>
ln -s /home/e-smith/db


Alternatively (if you've already created and synced the symlinked bay):
cd ~/Dropbox
dropbox.py exclude add *
dropbox.py exclude remove <symlinked-folder-name>

[brianr]

That is very interesting, I wonder why my google for "SMEServer Dropbox" did not pull that up?

The problem I have with the dropbox command program is that as far as I can see it is tricky to control what it writes and where (it talks about syncing which is not what I want). I can see that use soft or hardlinks might allow some control as to what is synced, but it all seems a bit obscure.

I have a very large amount of data on my dropbox account (backups from wordpress websites among other things), which I do not want synced to my server.

rclone gives me a very straighforward interface to write data to the account without any messing around.

[mmccarn]

I added a note to my original post on how to exclude all the other data already in Dropbox -- but I see your challenge.  You'd have to monitor the dropbox folder on your sme server and manually remove any new files or folders that appear because you added a new folder to the root of your Dropbox account...

As for googling "SMEServer dropbox", I tend to use "site:contribs.org dropbox" instead, which gets the wiki, forums, and bugzilla:
https://www.google.com/search?q=site:contribs.org+dropbox

[mmccarn]

Looking at the rclone faq, it appears that rclone does not support rsync-style diff-based backups.

The bandwidth savings in dropbox might be worth the extra headaches of keeping unwanted dropbox data off of your SME server. 

You could automate the exclusion of new dropbox data using something like this, scheduled to run hourly, daily, or whatever (in my testing, I've created a folder in ~/Dropbox/ named "office9" where I symlinked the folders I want backed up):

Code: [Select]

DSYNC=office9
cd ~/Dropbox
find . -maxdepth 1 ! -name "\.dropbox*" ! -name "\." ! -name "$DSYNC" -exec dropbox.py exclude add $(basename "{}") \;

The dropbox client does not preserve file or folder ownership or permissions; that could be a problem, too...

[edit]
Added "cd ~/Dropbox" to the code snippet

[brianr]

Looking at the rclone faq, it appears that rclone does not support rsync-style diff-based backups.

True, but clearly it checks dates etc and only writes up the changed / new files.

The bandwidth savings in dropbox might be worth the extra headaches of keeping unwanted dropbox data off of your SME server. 

The amount of extra data on dropbox would fill up my server in one sync.

[DanB35]

AFAIK, rclone is a statically-compiled binary, so it should run just fine under SME--though I haven't tried it myself (I've used it to back up several TB under FreeBSD, though).

[Jean-Philippe Pialasse]

There could be a limit with staticaly compiled binary. Sometime they still rely on your glib version. Which could be too old


Have a check on hubic.
They are cheaper than dropbox and have a backup function. Not only sync.

[brianr]

Have a check on hubic.
They are cheaper than dropbox and have a backup function. Not only sync.

I use dropbox for a number of other uses (with other people), i would not get them to move.

I've started to look at getting rclone to work (don't hold breath!!)

[DanB35]

There could be a limit with staticaly compiled binary. Sometime they still rely on your glib version.

Fair enough.  Downloaded rclone from rclone.org (the 64-bit Linux version), it runs (enough to show the usage page, anyway) without issue on a SME 9.2 test box.

[brianr]

Fair enough.  Downloaded rclone from rclone.org (the 64-bit Linux version), it runs (enough to show the usage page, anyway) without issue on a SME 9.2 test box.

That's good news, I'll do the same and configure it up and see what we get..

[brianr]

Have done some testing, and it certainly seems to work!

Have started on a wiki page:

https://wiki.contribs.org/Rclone

All contribs gratefully received.

[DanB35]

I see how it's useful to document rclone (and something addressing the encryption capability would be good as well), but I'm having trouble seeing the purpose of an smeserver-rclone contrib.  What would it accomplish?

[brianr]

I see how it's useful to document rclone (and something addressing the encryption capability would be good as well), but I'm having trouble seeing the purpose of an smeserver-rclone contrib.  What would it accomplish?

Actually I was not calling for a contrib rpm (or at least did not mean to), but I can see why you think I was.  I should have not used the abbreviation "contrib" but the full word - contribution. I agree that the jury is still out on whether it needs to be wrapped up in an rpm.

I'll try to look into encrypting the files as well.

[DanB35]

I should have not used the abbreviation "contrib" but the full word - contribution.

I was thinking more of this from the wiki page:

The latest version of smeserver-rclone is available in the SME repository, click on the version number(s) for more information.