How to find (out if) files sharing content?
Hi,
How could one find out if 2 files share any extents on a btrfs file system?
A more generic variation of the above: How to list files on the same file system/subvolume sharing content?
Thanks,
Gábor
Quoted 6 lines · Gábor Nyers
Hi,
How could one find out if 2 files share any extents on a btrfs file system?
A more generic variation of the above: How to list files on the same file system/subvolume sharing content?
You have direct (read-only) access to the metadata trees through
the TREE_SEARCH ioctl. It should be possible to walk through the extents of a given file, and (I think) follow back-refs from the extent back to the other files that share it.
There's no simple code to do that right now, though.
Hugo.
Signature · 4 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- And what rough beast, its hour come round at last / slouches ---
towards Bethlehem, to be born? Quoted 3 lines · Hugo Mills
It should be possible to walk through the
extents of a given file, and (I think) follow back-refs from the extent back to the other files that share it.
You wish :-) Backrefs are not made to walk them while the file system is online. However "btrfs inspect logical" manages quite well, at least I haven't heard otherwise so far. You still need to get the logical block numbers, either by TREE_SEARCH ioctl or by filefrag.
-Jan
Quoted 7 lines · Gábor Nyers
Hi,
How could one find out if 2 files share any extents on a btrfs file system?
A more generic variation of the above: How to list files on the same file system/subvolume sharing content?
Indeed ocfs2 already has the feature where you can get shared parts via 'du', we're planning to support this in btrfs, too.
thanks,
liubo
Quoted 7 lines
Thanks,
Gábor
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Liu Bo
Quoted 6 lines · Gábor Nyers
Hi,
How could one find out if 2 files share any extents on a btrfs file system?
A more generic variation of the above: How to list files on the same file system/subvolume sharing content?
One idea is to mark those cloned extents as FIEMAP_EXTENT_SHARED so that we can go through a file to figure out how many extents are shared through fiemap(2), and calculate the real storage(fs/subvolume) footprint in the end.
Thanks,
-Jeff
Quoted 16 lines
Quoted 1 line
Indeed ocfs2 already has the feature where you can get shared parts via 'du', we're planning to support this in btrfs, too.
thanks,
liuboQuoted 7 lines
Thanks,
Gábor
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Jeff Liu
One idea is to mark those cloned extents as
FIEMAP_EXTENT_SHAREDso that we can go through a file to figure out how many extents are shared through fiemap(2), and calculate the real storage(fs/subvolume) footprint in the end.
This will cost at least one more seek per extent to find out that the extent is shared, could be quite expensive. And without any possibility to turn this off, I'm afraid this will render FIEMAP unusable in practice.
david
Quoted 8 lines · David Sterba
Quoted 4 lines · Jeff Liu
One idea is to mark those cloned extents as
FIEMAP_EXTENT_SHAREDso that we can go through a file to figure out how many extents are shared through fiemap(2), and calculate the real storage(fs/subvolume) footprint in the end.This will cost at least one more seek per extent to find out that the extent is shared, could be quite expensive.
I propose this because OCFS2 report shared space in this way combine with du(1).
An old patch set to teach du(1) aware of reflinked file:
https://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007293.html
Do you means that the costs is very expensive for userland extent status checkup per file? If yes, I have once tested an 50Gb OCFS2 partition filled with reflinked files on an old laptop, it spent around 4 minutes to show the totally results if I recalled correct, but this definitely depending on the real world scenarios.
Quoted 1 line
And without any possibility to turn this off,I'm afraid this will render FIEMAP unusable in practice.
For OCFS2, the FIEMAP_EXTENT_SHARED flag will be set upon fiemap ioctl(2) if an extent is OCFS2_EXT_REFCOUNTED(i.e. reflinked or cloned), which means that FIEMAP_EXTENT_SHARED is not a persistent flag, but I have no idea how Btrfs would be in this point. :(
Thanks,
-Jeff
Quoted 7 lines
david
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Jeff Liu
I propose this because OCFS2 report shared space in this way combine with du(1).
An old patch set to teach du(1) aware of reflinked file:
https://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007293.html
Patch looks ok, the shared size is requested by an option.
Quoted 1 line
Do you means that the costs is very expensive for userland extent status checkup per file?
The most expensive part is IMO not in userspace, it does in-memory lookups.
Quoted 4 lines
Quoted 1 line
And without any possibility to turn this off,I'm afraid this will render FIEMAP unusable in practice.
For OCFS2, the
FIEMAP_EXTENT_SHAREDflag will be set upon fiemap ioctl(2) if an extent is OCFS2_EXT_REFCOUNTED(i.e. reflinked or cloned), which means thatFIEMAP_EXTENT_SHAREDis not a persistent flag, but I have no idea how Btrfs would be in this point. :(
After some research, I think this could work for btrfs without unwanted performance penalties.
There's the fiemap::fm_flags field that can be extended to request the shared extent info from fiemap, so the information is not computed unconditionally (that was my concern before). The rest is only implementation details how to speed up the file extent -> refcount info lookups.
david
Quoted 17 lines · David Sterba
Quoted 4 lines · Jeff Liu
I propose this because OCFS2 report shared space in this way combine with du(1).
An old patch set to teach du(1) aware of reflinked file:
https://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007293.htmlPatch looks ok, the shared size is requested by an option.
Quoted 1 line
Do you means that the costs is very expensive for userland extent status checkup per file?
The most expensive part is IMO not in userspace, it does in-memory lookups.
Quoted 4 lines
Quoted 1 line
And without any possibility to turn this off,I'm afraid this will render FIEMAP unusable in practice.
For OCFS2, the
FIEMAP_EXTENT_SHAREDflag will be set upon fiemap ioctl(2) if an extent is OCFS2_EXT_REFCOUNTED(i.e. reflinked or cloned), which means thatFIEMAP_EXTENT_SHAREDis not a persistent flag, but I have no idea how Btrfs would be in this point. :(
Quoted 8 lines
After some research, I think this could work for btrfs without unwanted performance penalties.
There's the
fiemap::fm_flagsfield that can be extended to request the shared extent info from fiemap, so the information is not computed unconditionally (that was my concern before). The rest is only implementation details how to speed up the file extent -> refcount info lookups.
Thanks for your confirmation.
-Jeff
Quoted 7 lines
david
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html