Possible to wait for snapshot deletion?
Hi!
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
This would fit some purposes I'm planning to implement:
* In a backup scenario have a subprocess which deletes snapshots one by one, starting with the oldest. If free space raises above a certain threshold, pause the subprocess. If number of kept snapshots falls below a certain threshold, exit the subprocess. When the backup job finished, it joins the subprocess to wait for a pending subvolume deletion if any, then syncs the filesystem and waits some grace time for uncommitted writes, then shuts the system down or hibernates it.
* Wait for pending background subvolume deletion before putting the system to sleep.
* Get better control of cron jobs working with subvolumes so jobs either do not overlap or do not put too much parallel work on the file system.
Scrubbing does allow that. Why don't snapshots do? At least I found no options for it.
Background information: If btrfs is shut down improperly, already deleted subvolume entries may reappear in their directories. The deletion is not gracefully resumed. Since deletion of subvolumes/snapshots can take hours, this is a problem for systems that are not guaranteed to be up all the time or suffer from disconnected power, especially if multiple snapshots are being deleted at once. With an option to wait, a script managing snapshot deletion could more gracefully resume its job.
Signature · 1 line
Replies to list only preferred.
Quoted 8 lines · Kai Krakow
Hi!
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
This would fit some purposes I'm planning to implement:
* In a backup scenario
I have a similar use-case for this also involving backups. In my case I have a script that uses a btrfs filesystem for the backup store using snapshots. At the end of each run, if diskspace usage is below a predefined threshold, it will delete old snapshots until the diskspace usage is below that threshold again.
Of course, the first time I added the automatic deletion, it deleted far more than was necessary due to the fact that the actual freeing of diskspace is asynchronous from the command completion. I ended up setting a small delay (of about 60 seconds) between each iteration and also set it to monitor system load. If load is not low enough after the delay then it waits another 60 seconds.
This complicated (frankly broken) workaround would be completely unnecessary with a --wait switch.
Alternatively, perhaps a knob where we can see if a subvolume deletion is in progress could help.
Signature · 4 lines
__________ Brendan Hide http://swiftspirit.co.za/ http://www.webafrica.co.za/?AFF1E97
Quoted 5 lines · Brendan Hide
Quoted 5 lines
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
This would fit some purposes I'm planning to implement:
Quoted 7 lines
Quoted 1 line
* In a backup scenario
I have a similar use-case for this also involving backups. In my case I have a script that uses a btrfs filesystem for the backup store using snapshots. At the end of each run, if diskspace usage is below a predefined threshold, it will delete old snapshots until the diskspace usage is below that threshold again.
Yeah, I thought of that approach first, too... But:
Quoted 6 lines
Of course, the first time I added the automatic deletion, it deleted far more than was necessary due to the fact that the actual freeing of diskspace is asynchronous from the command completion. I ended up setting a small delay (of about 60 seconds) between each iteration and also set it to monitor system load. If load is not low enough after the delay then it waits another 60 seconds.
Due to btrfs' behavior that cannot reliably work as you also figured out. It will need quirky work-arounds totally dependent on system load.
Quoted 2 lines
This complicated (frankly broken) workaround would be completely unnecessary with a --wait switch.
That's why I had the idea. ;-)
Quoted 2 lines
Alternatively, perhaps a knob where we can see if a subvolume deletion is in progress could help.
Like "btrfs scrub status"...
If you're feeling curious, here's what I've implemented yet (snapshot deletion still on the todo list):
https://gist.github.com/kakra/5520370
My idea is to fork-off a background process which constantly removes old snapshots (within sane bounds yet to be defined) and then exits. Some mechanism has to be found to not run into a deadlock situation here. Then, at the end issue a bash "wait" to join the processes again and run a final sync. However, I don't like to issue explicit syncs from within the subprocess as the filesystem is busy with rsync at the same time.
PS: Yes, I know there's btrfs send/receive - but it doesn't seem ready for the big show yet because it still has many strange quirks and should not be run unattended yet.
Signature · 1 line
Replies to list only preferred.
Quoted 4 lines · Kai Krakow
Hi!
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
I'm sure there's been some patches for that around here somewhere,
some time ago. I guess they didn't get merged.
Hugo.
Quoted 27 lines
This would fit some purposes I'm planning to implement:
* In a backup scenario have a subprocess which deletes snapshots one by one, starting with the oldest. If free space raises above a certain threshold, pause the subprocess. If number of kept snapshots falls below a certain threshold, exit the subprocess. When the backup job finished, it joins the subprocess to wait for a pending subvolume deletion if any, then syncs the filesystem and waits some grace time for uncommitted writes, then shuts the system down or hibernates it.
* Wait for pending background subvolume deletion before putting the system to sleep.
* Get better control of cron jobs working with subvolumes so jobs either do not overlap or do not put too much parallel work on the file system.
Scrubbing does allow that. Why don't snapshots do? At least I found no options for it.
Background information: If btrfs is shut down improperly, already deleted subvolume entries may reappear in their directories. The deletion is not gracefully resumed. Since deletion of subvolumes/snapshots can take hours, this is a problem for systems that are not guaranteed to be up all the time or suffer from disconnected power, especially if multiple snapshots are being deleted at once. With an option to wait, a script managing snapshot deletion could more gracefully resume its job.
Signature · 3 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Gentlemen! You can't fight here! This is the War Room! --- Quoted 3 lines · On 2-13-14 20:02:43 Kai Krakow wrote:
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
This may be what you're looking for:
http://www.spinics.net/lists/linux-btrfs/msg29833.html
Signature · 1 line
Garry T. Williams
Quoted 9 lines · Garry T. Williams
Quoted 3 lines · On 2-13-14 20:02:43 Kai Krakow wrote:
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
This may be what you're looking for:
http://www.spinics.net/lists/linux-btrfs/msg29833.html
Looks fantastic. Can I vote to get it into official sources? ;-)
Signature · 1 line
Replies to list only preferred.
Quoted 7 lines · Kai Krakow
Quoted 5 lines · Garry T. Williams
Quoted 3 lines · On 2-13-14 20:02:43 Kai Krakow wrote:
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
Quoted 6 lines
Quoted 4 lines
This may be what you're looking for:
http://www.spinics.net/lists/linux-btrfs/msg29833.htmlLooks fantastic. Can I vote to get it into official sources? ;-)
This has been in the development integration branch for some time and is present in the integration branch pending for the next release.
Quoted 2 lines · Kai Krakow
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
That would indeed be sweet (see pointer to patch by Garry Williams).
In the meantime have you tried issuing a simple "sync" in your script after every snapshot deletion? I've observed that this kicks the cleaner into action.
-h
Quoted 10 lines · Holger Hoffstätte
Quoted 2 lines · Kai Krakow
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
That would indeed be sweet (see pointer to patch by Garry Williams).
In the meantime have you tried issuing a simple "sync" in your script after every snapshot deletion? I've observed that this kicks the cleaner into action.
I was under the impression that a sync command just doesn't do the trick. It kicks in the cleaner but returns to early. But I din't try that again lately.
But I'm actually currently using "btrfs fs sync" to do it because a normal "sync" command would affect all my filesystems and I'd like to keep it isolated.
I'll post a link to my backup script for reference in another post in this thread.
Signature · 1 line
Replies to list only preferred.
Quoted 2 lines · Kai Krakow
Is it technically possible to wait for a snapshot completely purged from disk? I imagine an option like "--wait" for btrfs delete subvolume.
I have the patch WIP, will look at it again.
Quoted 10 lines
This would fit some purposes I'm planning to implement:
* In a backup scenario have a subprocess which deletes snapshots one by one, starting with the oldest. If free space raises above a certain threshold, pause the subprocess. If number of kept snapshots falls below a certain threshold, exit the subprocess. When the backup job finished, it joins the subprocess to wait for a pending subvolume deletion if any, then syncs the filesystem and waits some grace time for uncommitted writes, then shuts the system down or hibernates it.
Quoted 5 lines
* Wait for pending background subvolume deletion before putting the system to sleep.
* Get better control of cron jobs working with subvolumes so jobs either do not overlap or do not put too much parallel work on the file system.
These usecases should be covered by the 'wait' command.
Quoted 7 lines
If btrfs is shut down improperly, already deleted
subvolume entries may reappear in their directories. The deletion is not gracefully resumed. Since deletion of subvolumes/snapshots can take hours, this is a problem for systems that are not guaranteed to be up all the time or suffer from disconnected power, especially if multiple snapshots are being deleted at once. With an option to wait, a script managing snapshot deletion could more gracefully resume its job.
As mentioned down the thread, next progs release will be able to force a transaction commit when the subvoulme is deleted.