For years I’m doing backups using rsync. Especially the variant using hardlinks, in order to get backups that only consume the disk space needed by an incremental backup, but being being able to access each daily backup as if it was a full one. The whole call looks like rsync -a --delete $OTHER_OPTIONS --link-dest=27/ $SRC/ 28/ (for the 28th of the month). Prior to doing each backup, I’m removing the backup from the previous month - a bit unsure why I exactly do this, the –delete option of rsync should exactly do the job)

Some time ago, I’ve replaced the drive used for backup by a new one, and in this context I’ve switched from ext4 to btrfs, in order to take advantage of its data protection features (btrfs scrub)

While this should not be an issue at all, I’ve noticed that the whole process is taking ages. Example from today :

1
2
3

Removal starting at  Sun 28 Mar 2021 05:00:08 AM CEST
Backup starting at  Sun 28 Mar 2021 07:17:02 AM CEST
Backup finished at  Sun 28 Mar 2021 09:04:03 AM CEST

So… 2h for removing xxx files, followed by 2h for the backup (OK, maybe today there were a bit more files than usual to backup)

Google is your friend… btrfs has terrific performance when it comes to deleting hundred thousands of files :-( In combination with rsync, the prefered solution seems to be using btrfs snapshots, which can be deleted a light speed. Some references :

• https://www.sanitarium.net/golug/rsync+btrfs_backups_2011.html
• https://www.funtoo.org/BTRFS_Fun
• https://github.com/oxplot/rsyncbtrfs/blob/master/rsyncbtrfs

The last one, rsyncbtrfs, is a quite complete all-in-one scripted solution that combines creating a subvolume, performing the sync and creating a snapshot. It also includes a few safeguards in case problems arise.

In the end I’ve adapted my scripts with some inspiration from the rsyncbtrfs script :

1
2
3
4
5
6

mkdir backup && btrfs subvolume create backup && cd backup
mkdir current
rsync -a --delete --inplace --delete-before $OTHER_OPTIONS $SRC/ current/
btrfs subvolume delete $(date +%d)
btrfs subvolume snapshot current $(date +%d)
btrfs subvolume list .

Update: here the backup time with the snapshot system

1
2
3

Removal starting at  Mon 29 Mar 2021 05:15:01 AM CEST
Backup starting at  Mon 29 Mar 2021 05:15:01 AM CEST
Backup finished at  Mon 29 Mar 2021 05:19:12 AM CEST