Incremental Backup

From btrfs Wiki
Revision as of 15:50, 7 February 2019 by 3coma3 (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

BTRFS has some key capabilities which enable efficient backup solutions:

Instant, Atomic COW Snapshots
Since the snapshots are atomic, when a snapshot is restored it appears to applications as if a power loss had occurred (and the filesystem has gone back to an earlier state). Thus it is possible to backup databases without stopping them beforehand.
Incremental Snapshot Transfer
Efficiently determining and streaming the differences between two snapshots if they are either snapshots of the same underlying subvolume, or have a parent-child relationship. This is far quicker than e.g. rsync could, especially on large file systems. (For instance, rsync cannot be aware of mere metadata changes like filename, location etc but the FS itself is certainly aware of it.)

This page presents some approaches to leverage these capabilites.



Prerequisite is a fairly modern btrfs toolsuite, version 0.20rc1 seems to be good enough. Earlier versions may fail to list subvolumes when cleaner process is operating, and this could cause random failures during backup.

That said, many fixes have gone in btrfs send/receive and it is recommended to use at least a 3.14, or 3.15 kernel to have later patches. Similarly, you will want at least btrfs-tools 3.12, or later if available. Note that 32bit btrfs send will not work with a 64bit kernel as of 3.13.

Available Backup Tools

(See also the list in Use Cases).

btrfs-subvolume-backup [1] by Marc MERLIN.
Script taking a snapshot and transferring it to another location via SSH. Keeps an configurable number of past snapshots
Backup tool with configurable snapshot retention (for example 1 per week for a month, one per month for a year), restore functionality, synchronization to and from other hosts
btrfs-sxbackup [3] by Marco.
Another more featureful backup script in python that lets you initiate backups from a server
btrbk [4] by Axel Burri.
Backup tool using a configuration file, allows creation of backups from multiple sources to multiple destinations at once, with ssh and configurable retention support (daily/weekly/monthly)
snazzer [5]
Minimal-dependency btrfs snapshotting and backup system offering sha512sum/PGP snapshot measurement, ssh or local filesystem transport and non-linear pruning.
Just backup btrfs [6] by Nazar Mokrynskyi
Snapshots creation within the same btrfs partition, backups to another partition, snapshots rotation separately for subvolumes within partition and for external on another partition (you can keep few snapshots on source partition and more on external bigger one); single PHP script without external dependencies, uses simple configuration file in JSON format
Btrfs sync subvolumes [7] by Nazar Mokrynskyi
This is a pretty naive implementation of replication exact set of subvolumes from one location to another, plays nicely with Just backup btrfs
snapbtr [8]
Simple btrfs snapshoting. Does not require root permissions to create a snapshot, so it is possible to run it from user applications's hooks.
snapbtrex [9]
Snapshot creation and transfer to remote systems via ssh, no config files just shell options. The only dependency is python. An extension of the SnapBtr script. This is a fork of the original version of snapbtr.
btrfs-backup [10]
Another, very small (50 lines), rsync script intended as base for extensions and ideas. It implements a custom 5-level rotation scheme based on snapshot date and time.

Doing it by hand, step by step

When performing incremental backups, we will be working with two snapshots, one of them representing the time of the earlier backup, and the other representing the current backup. When the backup run completes, we can discard the earlier of the snapshots to prepare for the next incremental run.

Assuming that /home is the BTRFS volume you wish to backup, and backups are to be stored on another BTRFS volume called /backup and directory /backup/home, the procedure goes as follows.

Initial Bootstrapping

We will need to create a read-only snapshot of the volume that serves as the reference for the first backup. I will call this subvolume BACKUP. The subvolume is read-only because "btrfs send" requires read-only subvolumes to operate on. NB: there is currently an issue that the snapshots to be used with "btrfs send" must be physically on the disk, or you may receive a "stale NFS file handle" error. This is accomplished by "sync" after the snapshot:

   btrfs subvolume snapshot -r /home /home/BACKUP

Once created, we can distribute the initial copy into existing directory or subvolume /backup/home. The subvolume appears as /backup/home/BACKUP:

   btrfs send /home/BACKUP | btrfs receive /backup/home

Bootstrapping is now done. The subvolume /home/BACKUP is kept around to serve as local reference for the data that has been backed up, and it is needed for constructing the incremental backup for the next step.

Incremental Operation

During incremental backup, we make a new snapshot:

   btrfs subvolume snapshot -r /home /home/BACKUP-new

We can now send the difference between the old and new backup to the backup volume:

   btrfs send -p /home/BACKUP /home/BACKUP-new | btrfs receive /backup/home

Once this command completes, we should have these 4 subvolumes: /home/BACKUP, /home/BACKUP-new, /backup/home/BACKUP and /backup/home/BACKUP-new. We will now need to migrate the new backup as the old one, and do something for the old one. We could keep it around, maybe timestamped with the date of that backup, or just straight out delete it. Here, I am deleting it:

   btrfs subvolume delete /home/BACKUP
   mv /home/BACKUP-new /home/BACKUP
   btrfs subvolume delete /backup/home/BACKUP
   mv /backup/home/BACKUP-new /backup/home/BACKUP

But for instance, if you did want to keep a history of backups, perhaps you would snapshot one of the snapshot directories with something like:

   btrfs subvolume snapshot -r /backup/home/BACKUP /backup/home.$(date +%Y-%m-%d)

This concludes the incremental backup step.

See also

The FAQ on the difference between -p and -c when sending

Personal tools