I have changed the shell script, in order to avoid moving the remote directories. The script makes a remote copy of selected directories to Amazon S3, in the directory recent. When a file is changed in recent, the previous version is moved to one of a rotating set of remote directories. If you have a set of 7 rotating directories, and you run the script daily, you will be able to go a week back in history and find your missing file there. If you just want the most recent version of your files or directories, you can find them in recent.
The first time the script is run, it will most likely take a few hours. After this, it will be quite fast. 10 minutes for my server.
I have a small problem finding the right names for variables etc. The script is, of course, a backup-script. But in rclone-terms, what I do is a cloning with a "backup-dir", containing the previous versions of changed files.
I will be thankful if someone here can help me improve the script, and maybe turn it into a contrib.
The script now looks like this:
#!/bin/bash
SERVER=$(hostname -s)
REMOTE="amazonS3:${SERVER}" # Name of the remote service
MYDIR="/home/e-smith/files/users/holck/dev/rclone/" # Home-directory of this script
FILTER="filter.txt" # Filter, naming local directories to in- and exclude in backup
BACKUPLIST="bak.list" # Local file with names of a rotating set of remote directories to use for older file-versions
VERBOSE="" # Add more v's for more verbosity: -v = info, -vv = debug
LOG="/tmp/rclone_log.txt" # Temporary log file
cd $MYDIR
# Rotate the BACKUPLIST-file, deleting the top entry and adding a new entry at the bottom
OLDEST=$(head -1 $BACKUPLIST | sed 's/ .*//') # The oldest, extra backup-version
COUNT=$(cat $BACKUPLIST | wc -l) # Number of files in list
((COUNT=$COUNT-1))
tail -$COUNT $BACKUPLIST > $BACKUPLIST.tmp
echo $OLDEST $(date) >> $BACKUPLIST.tmp # Add a new line to the bottom, including today's date
# Purge oldest remote version
date
echo "Purging oldest version ($OLDEST)..."
nice rclone purge ${REMOTE}/${OLDEST} ${VERBOSE} &>> ${LOG}
echo "...done"
# Perform actual backup
date
echo "Performing backup to ${REMOTE}..."
nice rclone sync / ${REMOTE}/recent --backup-dir ${REMOTE}/${OLDEST} --filter-from ${FILTER} ${VERBOSE} --skip-links --update --use-server-modtime &
echo "...done"
date
mv -f $BACKUPLIST.tmp $BACKUPLIST # Update the BACKUPLIST
cat ${LOG} >> log.txt
Here is a sample filter:
+ /etc/e-smith/templates-custom/**
+ /etc/e-smith/templates-user-custom/**
+ /etc/group/**
+ /etc/gshadow/**
+ /etc/passwd/**
+ /etc/samba/secrets.tdb/**
+ /etc/samba/smbpasswd/**
+ /etc/shadow/**
+ /etc/ssh/**
+ /etc/sudoers/**
+ /home/e-smith/**
+ /opt/search/**
+ /root/**
+ /usr/lib/mailman/aliases/**
+ /usr/share/wordpress/wp-content/**
+ /var/lib/mailman/**
+ /var/log/**
- *
And a sample baklist-file:
old3
old2
old1
old7 Wed Dec 12 13:42:40 CET 2018
old6 Thu Dec 13 10:00:01 CET 2018
old5 Thu Dec 13 10:05:01 CET 2018
old4 Fri Dec 14 01:30:02 CET 2018