ari-backup is a lightweight wrapper around rdiff-backup. The goal is to provide a way to manage many rdiff-backup jobs consistently and easily with features like:
- centralzed configuration
- support for backing up local and remote hosts
- configurable job parallelization
- ability to run arbitrary commands locally or remotely before and/or after backup jobs (something especially handy for preparing databases pre-backup)
- logging to syslog
This application is lightweight thanks mostly to leveraging common system tools to provide most of the facility necessary to run a backup system. cron is used to schedule the backup jobs, xargs is used to optionally run jobs in parallel, run-parts is used to execute individual backup jobs, and ssh is used for authentication and secure data transport.
This README and the ari-backup documentation expect that the reader has a basic understanding of Linux, file system semantics, how to install a system package, and how to install a Python package. The typical audience for this software is the system administrator that wants to backup several systems with rdiff-backup.
ari-backup was developed on and written for Linux. But there have been reports of its use on Windows using cygwin.
Before you install ari-backup, consider installing the following packages from your Linux distribution.
- python-yaml
- rdiff-backup
To install the ari_backup package to your system, run this as root:
pip install git+git://github.com/jpwoodbu/ari-backup.git
Before you can execute a backup job there are a few files and directories that need to be setup. At this time, the configuration file for ari-backup is always read from /etc/ari-backup/ari-backup.conf.yaml. For this demo put this into the ari-backup.conf.yaml file:
backup_store_path: /backup-storeNow create the /backup-store directory.
Our demo will use the most basic example of a backup job. Our job will backup our /music directory to /backup-store/my_backup. Put the following into a file named ari-backup-local-demo:
#!/usr/bin/env python
from ari_backup import ARIBackup
backup = ARIBackup(label='my_backup', source_hostname='localhost')
backup.include_dir_list.append('/music')
backup.run_backup()
Make ari-backup-local-demo executable and run it. Make sure the user you're using to execute this script has the permission to read the /etc/ari-backup/ari-backup.conf.yaml file, to read the /music directory and has permission to write to the /backup-store/my_backup directory. If all goes well, you should see no output to the console. But take a look at your syslog and you might see lines like this:
Nov 17 23:21:56 morbo ARIBackup (my_backup) [INFO] started Nov 17 23:21:56 morbo ARIBackup (my_backup) [INFO] processing pre-job hooks... Nov 17 23:21:56 morbo ARIBackup (my_backup) [INFO] data backup started... Nov 17 23:21:59 morbo ARIBackup (my_backup) [INFO] data backup complete Nov 17 23:21:59 morbo ARIBackup (my_backup) [INFO] processing post-job hooks... Nov 17 23:21:59 morbo ARIBackup (my_backup) [INFO] stopped
Your /backup-store directory should now have a my_backup directory. And inside that directory you should see a mirror of your /music/ directory as well as a rdiff-backup-data directory. The rdiff-backup-data is where rdiff-backup stores its own data like the reverse increments, statistics, and file metadata.
For a more exciting demo, let's backup a remote host. We'll be using ssh to authenticate to the remote host and public key authentication is the only method supported by ari-backup. Be sure to have your keys setup for both the user that will run ari-backup and the user that we'll use to connect to the remote host. For this demo, we're going to use the user backups.
The remote system requires very little setup. Once you've got your SSH key installed, the only other step is to install rdiff-backup. ari-backup does not need to be installed on the remote system. Isn't that great!
Make sure that the user that's running your backup script has the remote host's host key in its known_hosts file. The best way to ensure that it is, is to test your public key authentication works by logging in to the remote system manually.
We'll need to add the remote_user setting to our /etc/ari-backup/ari-backup.conf.yaml file. It should now look like:
backup_store_path: /backup-store remote_user: backups
Let's assume that your remote host is named kif. Make a new backup job file named ari-backup-remote-demo with this content:
#!/usr/bin/env python
from ari_backup import ARIBackup
backup = ARIBackup(label='kif_backup', source_hostname='kif')
backup.include_dir_list.append('/music')
backup.run_backup()
Make ari-backup-remote-demo executable and run it. Again, no output to the console means everthing worked. Check the syslog and your /backup-store/kif_backup directory to see the results. Once you've got your ssh keys setup, the only thing different about remote backups is the value you put in the source_hostname parameter.
See include/cron/ari-backup for an example script you can use with cron. By default, this script will look for backup jobs in /etc/ari-backup/jobs.d. And by default, this script will only execute one backup job at a time. You can edit the JOBS_DIR and CONCURRENT_JOBS variables in the script to tweak those settings to taste.
To put this altogether with an example, let's use the two backup job scripts you made from before, ari-backup-local-demo and ari-backup-remote-demo. Place them into the /etc/ari-backup/jobs.d directory. Now copy include/cron/ari-backup to /etc/cron.daily (or an equivalent directory on your system). You can now wait for cron to run the script in /etc/cron.daily, or better yet, execute it yourself to test it out.
You can again look at your syslog to see that the backups ran. But you'll also notice that when running our cron script you will actually get some console output as it reports how long the entire selection of jobs took to run. You may see something like this:
real 3m44.318s user 0m45.595s sys 0m8.253s
If you have cron setup to email you when there's output like this, then you'll have a handy (or annoying) email reporting whether your backups ran successfully each time.
Be sure that the names of your backup job scripts are compatible with what run-parts expects. See the run-parts man page for more on their filename restrictions.
Pro tip: since run-parts will ignore file names with dots, a simple way to disable a backup job is to prefix a dot to its filename.
Let's add LVM into the mix so that we can achieve crash-consistent backups. This is done using the LVMBackup extension. We'll need to add the snapshot_mount_root and snapshot_suffix settings to our existing /etc/ari-backup/ari-backup.conf.yaml file:
snapshot_mount_root: /tmp snapshot_suffix: -ari-backup
snapshot_mount_root defines where the temporary snapshots are mounted during the backup (snapshots are automatically removed after the backup is completed). snapshot_suffix determines the suffix of the name of the snapshot. This is useful when debugging so that it's clear where the snapshot came from.
Let's assume that your remote host is named db-server. You want rdiff-backup to remove increments older than one month, so you set remove_older_than_timespec='1M'. You specify the LVM volumes and their mountpoints on the remote system (you may add more than one LVM volume by adding multiple backup.lv_list.append statements). Finally, specify the directories to be backed up with backup.include_dir_list.append. Make a new backup job file named ari-backup-remote-lvm-demo with this content:
#!/usr/bin/env python
from ari_backup import LVMBackup
backup = LVMBackup(
label='mybackup',
source_hostname='db-server',
remove_older_than_timespec='1M'
)
backup.lv_list.append(('vg0/root', '/'))
backup.include_dir_list.append('/etc')
backup.run_backup()
TODO write up examples of how to use the ZFSLVMBackup extension.
ari-backup gets its name from the American Research Institute where it was originally written in bash. As rdiff-backup was our software of choice to backup our Linux systems, we needed some sort of scripting around running rdiff-backup on a schedule. We could write a script that performed all our backups and just place it in /etc/cron.daily, but that didn't seem scalable and was especially monolithic since we were backing up about 50 machines.
We liked the idea of seperate backup scripts for each backup job. In our case, each job was backing up a host. But we didn't want to overcrowd the /etc/cron.daily directory. So we put all our backup scripts in their own directory and put a single file in /etc/cron.daily that called our backups using run-parts. We later cooked in the xargs part that made it easy to run backup jobs concurrently.
When we started to add the LVM snapshot features we decided that porting it to Python was going to make working on this project much easier.
In 2011, ARI graciously open-sourced this software.