Feature Request: Subvolume Expiration
Hi everyone,
I'd like to propose "subvolume expiration dates" as a new feature for btrfs.
I want to use btrfs for archiving and backup solutions for which snapshots are very convenient. However, the rotation and cleanup of snapshots is always based on some scripting and often reimplemented for each usecase.
I know at least some proprietary storage solutions which allow to set a expiration date on snapshots so I wonder if it would be possible to add a similar functionality to btrfs. The advantage would be that one doesn't have to encode the expiry information in the subvolume name or parse it from an extended file attribute but it would be an (optional) attribute of the subvolume itself.
Therefore it would be easy to implement a subcommand in btrfs-progs which checks for expired subvolumes and deletes them (eg. btrfs subvolume prune or similar).
This would make btrfs a very easy to use solution for archiving data for a specific timespan as well as for backup snapshot rotations.
As I understand it I'd say the expiration date could be implemented in the root_item struct by squeezing in a additional btrfs_timespec into the reserved space.
But since I'm not really familiar with btrfs code I wanted to ask for more opinions before I spend time on a proof-of-concept.
So is this feature interesting, absurd, nice to have or completely impossible? Any feedback welcome :-)
Regards,
Beni
Quoted 4 lines · On Mon, Dec 05, 2022 at 01:28:42PM +0000, [email protected] wrote:
Hi everyone,
I'd like to propose "subvolume expiration dates" as a new feature for btrfs.
Quoted 8 lines
I want to use btrfs for archiving and backup solutions for which snapshots are very convenient. However, the rotation and cleanup of snapshots is always based on some scripting and often reimplemented for each usecase.
I know at least some proprietary storage solutions which allow to set a expiration date on snapshots so I wonder if it would be possible to add a similar functionality to btrfs. The advantage would be that one doesn't have to encode the expiry information in the subvolume name or parse it from an extended file attribute but it would be an (optional) attribute of the subvolume itself.
I can understand not wanting to encode it in the filename, but this is exactly the sort of use case that extended attributes are for:
attaching new bits of information to btrfs objects that are useful for userspace tools but not useful for the kernel.
Quoted 7 lines
Therefore it would be easy to implement a subcommand in btrfs-progs which checks for expired subvolumes and deletes them (eg.
btrfs subvolume pruneor similar).
This would make btrfs a very easy to use solution for archiving data for a specific timespan as well as for backup snapshot rotations.As I understand it I'd say the expiration date could be implemented in the
root_itemstruct by squeezing in a additionalbtrfs_timespecinto thereservedspace.
These fields (and there will always be more fields for these kinds of use cases) can be stored in xattrs on the subvol root dir. Nothing in the kernel needs to know anything about them. Userspace can parse the expiration date xattr, compare to the current date, and issue subvol delete commands for subvols where one is less than the other.
The gotcha is that it's not easy to store xattrs on subvol roots. They can be read-only, and making them read-write so you can set an xattr (e.g. change the expiration date) breaks btrfs incremental sends.
You can avoid the read-only problems by setting an expiration date xattr on the origin subvol before making a snapshot. You'd have to find some way to remove the risk that the origin subvol ends up with an expiration date that another tool notices before you can remove the expiration date from the origin subvol. Also the xattrs will be transmitted by btrfs send, rsync -X, and possibly other tools, and it's not always the case that the received subvol should have the same attribute values as the origin subvol. This isn't ideal--you wouldn't want your short-lived origin subvols to move to an archive server and be promptly deleted for being too old.
So we'd need some way to store xattrs for a subvol, but somehow avoid to storing them in the subvol itself. It looks like the root tree is ideally set up for that--just add XATTR items to the tree alongside the existing root_items, and add some way for userspace to get and set them. They would be outside of the subvol, so they are not affected by subvol read-only state, and not copied or snapshotted with the subvol contents, but still uniquely associated with one instance of the subvol on one filesystem, and still follow the subvol around if it is renamed.
Without a sane place to store xattrs and without modifying the kernel, it makes the most sense to simply store expiration times in a tool-specific external database, and let that tool manage the whole lifecycle of the subvols it knows about. That way there's no possibility of cross-tool confusion, and no need to implement new kernel-space interfaces to access xattrs in a previously unused part of the filesystem. This approach basically reduces to "install a subvol management tool to solve this problem in an appropriate way for your use case" and likely contributes to why there are so many fine subvol management tools available today.
Quoted 8 lines
But since I'm not really familiar with btrfs code I wanted to ask for more opinions before I spend time on a proof-of-concept.
So is this feature interesting, absurd, nice to have or completely impossible? Any feedback welcome :-)
Regards,
Beni