Extents for a particular subvolume
Are there any btrfs commands (or APIs) to allow a script to create a list of all the extents referred to within a particular (mounted) subvolume? And is it a reasonably efficient process (i.e. doesn't involve backrefs and, preferably, doesn't involve following directory trees)?
I am not looking to relate the extents to files/inodes/paths. My particular need, at the moment, is to work out how much of two snapshots is shared data, but I can think of other uses for the information.
Graham
Quoted 5 lines · Graham Cobb
Are there any btrfs commands (or APIs) to allow a script to create a list of all the extents referred to within a particular (mounted) subvolume? And is it a reasonably efficient process (i.e. doesn't involve backrefs and, preferably, doesn't involve following directory trees)?
Since the size of your output is linear to the number of extents which is between the number of files and sum of their sizes, I see no gain in trying to avoid following the directory tree.
And that can be done with FIEMAP on any filesystem, not just btrfs.
Quoted 3 lines
I am not looking to relate the extents to files/inodes/paths. My particular need, at the moment, is to work out how much of two snapshots is shared data, but I can think of other uses for the information.
Thus, unlike the question you asked above, you're not interested in _all_ extents, merely those which changed.
You may want to look at "btrfs subv find-new" and "btrfs send --no-data".
Meow!
Signature · 1 line
An imaginary friend squared is a real enemy.
Quoted 10 lines · Adam Borowski
Quoted 5 lines · Graham Cobb
Are there any btrfs commands (or APIs) to allow a script to create a list of all the extents referred to within a particular (mounted) subvolume? And is it a reasonably efficient process (i.e. doesn't involve backrefs and, preferably, doesn't involve following directory trees)?
Since the size of your output is linear to the number of extents which is between the number of files and sum of their sizes, I see no gain in trying to avoid following the directory tree.
Thanks for the help, Adam. There are a lot of files and a lot of directories - find, "ls -R" and similar operations take a very long time. I was hoping that I could query some sort of extent tree for the subvolume and get the answer back in seconds instead of multiple minutes.
But I can follow the directory tree if I need to.
Quoted 8 lines
Quoted 3 lines
I am not looking to relate the extents to files/inodes/paths. My particular need, at the moment, is to work out how much of two snapshots is shared data, but I can think of other uses for the information.
Thus, unlike the question you asked above, you're not interested in _all_ extents, merely those which changed.
You may want to look at "btrfs subv find-new" and "btrfs send --no-data".
Unfortunately, the subvolumes do not have an ancestor-descendent relationship (although they do have some common ancestors), so I don't think find-new is much help (as far as I can see).
But just looking at the size of the output from "send -c" would work well enough for the particular problem I am trying to solve tonight! Although I will need to take read-only snapshots of the subvolumes to allow send to work. Thanks for the suggestion.
I would still be interested in the extent list, though. The main problem with find-new and send is that they don't tell me how much has been deleted, only added. I am thinking about using the extents to get a much better handle on what is using up space and what I could recover if I removed (or moved to another volume) various groups of related subvolumes.
Thanks again for the help.
Quoted 28 lines · Graham Cobb
Quoted 10 lines · Adam Borowski
Quoted 5 lines · Graham Cobb
Are there any btrfs commands (or APIs) to allow a script to create a list of all the extents referred to within a particular (mounted) subvolume? And is it a reasonably efficient process (i.e. doesn't involve backrefs and, preferably, doesn't involve following directory trees)?
Since the size of your output is linear to the number of extents which is between the number of files and sum of their sizes, I see no gain in trying to avoid following the directory tree.
Thanks for the help, Adam. There are a lot of files and a lot of directories - find, "ls -R" and similar operations take a very long time. I was hoping that I could query some sort of extent tree for the subvolume and get the answer back in seconds instead of multiple minutes.
But I can follow the directory tree if I need to.
Quoted 8 lines
Quoted 3 lines
I am not looking to relate the extents to files/inodes/paths. My particular need, at the moment, is to work out how much of two snapshots is shared data, but I can think of other uses for the information.
Thus, unlike the question you asked above, you're not interested in _all_ extents, merely those which changed.
You may want to look at "btrfs subv find-new" and "btrfs send --no-data".
Quoted 8 lines
Unfortunately, the subvolumes do not have an ancestor-descendent relationship (although they do have some common ancestors), so I don't think find-new is much help (as far as I can see).
But just looking at the size of the output from "send -c" would work well enough for the particular problem I am trying to solve tonight! Although I will need to take read-only snapshots of the subvolumes to allow send to work. Thanks for the suggestion.
FWIW, if you're not using any files in the subvolumes, you can run:
btrfs property set <subvolume> ro true
to mark them read-only so you don't need the snapshots, and then run the same command with 'false' at the end instead of true to mark them writable again.
Quoted 7 lines
I would still be interested in the extent list, though. The main problem with find-new and send is that they don't tell me how much has been deleted, only added. I am thinking about using the extents to get a much better handle on what is using up space and what I could recover if I removed (or moved to another volume) various groups of related subvolumes.
You may want to look into 'btrfs filesystem usage' and 'btrfs filesystem du' commands. I'm not sure if they'll cover what you need, but they can show info about how much is shared.
Quoted 7 lines · Graham Cobb
Quoted 6 lines · Adam Borowski
Quoted 5 lines · Graham Cobb
Are there any btrfs commands (or APIs) to allow a script to create a list of all the extents referred to within a particular (mounted) subvolume? And is it a reasonably efficient process (i.e. doesn't involve backrefs and, preferably, doesn't involve following directory trees)?
In case anyone else is interested in this, I ended up creating some simple scripts to allow me to do this. They are slow because they walk the directory tree and they use filefrag to get the extent data, but they do let me answer questions like:
* How much space am I wasting by keeping historical snapshots?
* How much data is being shared between two subvolumes
* How much of the data in my latest snapshot is unique to that snapshot?
* How much data would I actually free up if I removed (just) these particular subvolumes?
If they are useful to anyone else you can find them at:
https://github.com/GrahamCobb/extents-lists
If anyone knows of more efficient ways to get this information please let me know. And, of course, feel free to suggest improvements/bugfixes!