fiemap extension to add physical extent length
Subsystems (11)
cover fiemap extension to add physical extent length
For many years, various btrfs users have written programs to discover the actual disk space used by files, using root-only interfaces. However, this information is a great fit for fiemap: it is inherently tied to extent information, all filesystems can use it, and the capabilities required for FIEMAP make sense for this additional information also.
Hence, this patchset adds physical extent length information to fiemap, and extends btrfs to return it. This uses some of the reserved padding in the fiemap extent structure, so programs unaware of the new field will be unaffected by its presence.
This is based on next-20240307. I've tested the btrfs part of this with the standard btrfs testing matrix locally, and verified that the physical extent information returned there is correct, but I'm still waiting on more tests. Please let me know what you think of the general idea!
Sweet Tea Dorminy (3):
fs: add physical_length field to fiemap extents
fs: update fiemap_fill_next_extent() signature
btrfs: fiemap: return extent physical size
Documentation/filesystems/fiemap.rst | 29 +++++++++---- fs/bcachefs/fs.c | 6 ++- fs/btrfs/extent_io.c | 63 +++++++++++++++++----------- fs/ext4/extents.c | 1 + fs/f2fs/data.c | 8 ++-- fs/f2fs/inline.c | 3 +- fs/ioctl.c | 8 ++-- fs/iomap/fiemap.c | 2 +- fs/nilfs2/inode.c | 8 ++-- fs/ntfs3/frecord.c | 6 ++- fs/ocfs2/extent_map.c | 4 +- fs/smb/client/smb2ops.c | 1 + include/linux/fiemap.h | 2 +- include/uapi/linux/fiemap.h | 24 +++++++---- 14 files changed, 108 insertions(+), 57 deletions(-)
base-commit: 1843e16d2df9d98427ef8045589571749d627cf7
Signature · 1 line
2.44.0
PATCH 1/3 fs: add physical_length field to fiemap extents
Some filesystems support compressed extents which have a larger logical size than physical, and for those filesystems, it can be useful for userspace to know how much space those extents actually use. For instance, the compsize [1] tool for btrfs currently uses btrfs-internal, root-only ioctl to find the actual disk space used by a file; it would be better and more useful for this information to require fewer privileges and to be usable on more filesystems. Therefore, use one of the padding u64s in the fiemap extent structure to return the actual physical length; and, for now, return this as equal to the logical length.
[1] https://github.com/kilobyte/compsize
--- Documentation/filesystems/fiemap.rst | 26 ++++++++++++++++++-------- fs/ioctl.c | 1 + include/uapi/linux/fiemap.h | 24 +++++++++++++++++------- 3 files changed, 36 insertions(+), 15 deletions(-)
Documentation/filesystems/fiemap.rst +18 -8
@@ -80,14 +80,24 @@ Each extent is described by a single fiemap_extent structure as returned in fm_extents:: struct fiemap_extent { - __u64 fe_logical; /* logical offset in bytes for the start of - * the extent */ - __u64 fe_physical; /* physical offset in bytes for the start - * of the extent */ - __u64 fe_length; /* length in bytes for the extent */ - __u64 fe_reserved64[2]; - __u32 fe_flags; /* FIEMAP_EXTENT_* flags for this extent */ - __u32 fe_reserved[3]; + /* + * logical offset in bytes for the start of + * the extent from the beginning of the file + */ + __u64 fe_logical; + /* + * physical offset in bytes for the start + * of the extent from the beginning of the disk + */ + __u64 fe_physical; + /* length in bytes for this extent */ + __u64 fe_length; + /* physical length in bytes for this extent */ + __u64 fe_physical_length; + __u64 fe_reserved64[1]; + /* FIEMAP_EXTENT_* flags for this extent */ + __u32 fe_flags; + __u32 fe_reserved[3]; }; All offsets and lengths are in bytes and mirror those on disk. It is valid
fs/ioctl.c +1 -0
@@ -139,6 +139,7 @@ int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical, extent.fe_logical = logical; extent.fe_physical = phys; extent.fe_length = len; + extent.fe_physical_length = len; extent.fe_flags = flags; dest += fieinfo->fi_extents_mapped;
include/uapi/linux/fiemap.h +17 -7
@@ -15,13 +15,23 @@#include <linux/types.h> struct fiemap_extent { -__u64 fe_logical;/* logical offset in bytes for the start of -* the extent from the beginning of the file */ -__u64 fe_physical;/* physical offset in bytes for the start -* of the extent from the beginning of the disk */ -__u64 fe_length;/* length in bytes for this extent */ -__u64 fe_reserved64[2 ]; -__u32 fe_flags;/* FIEMAP_EXTENT_* flags for this extent */ +/* +* logical offset in bytes for the start of +* the extent from the beginning of the file +*/ +__u64 fe_logical; +/* +* physical offset in bytes for the start +* of the extent from the beginning of the disk +*/ +__u64 fe_physical; +/* length in bytes for this extent */ +__u64 fe_length; +/* physical length in bytes for this extent */ +__u64 fe_physical_length; +__u64 fe_reserved64[1 ]; +/* FIEMAP_EXTENT_* flags for this extent */ +__u32 fe_flags;__u32 fe_reserved[3 ]; };
Signature · 1 line
2.44.0
Re: [PATCH 1/3] add physical_length field to fiemap extents
Quoted 11 lines · Sweet Tea Dorminy
Some filesystems support compressed extents which have a larger logical size than physical, and for those filesystems, it can be useful for userspace to know how much space those extents actually use. For instance, the compsize [1] tool for btrfs currently uses btrfs-internal, root-only ioctl to find the actual disk space used by a file; it would be better and more useful for this information to require fewer privileges and to be usable on more filesystems. Therefore, use one of the padding u64s in the fiemap extent structure to return the actual physical length; and, for now, return this as equal to the logical length.
Thank you for working on this patch. Note that there was a patch from David Sterba and a lengthy discussion about exactly this functionality several years ago. If you haven't already read the details, it would be useful to do so. I think the thread had mostly come to good conclusions, but the patch never made it into the kernel.
One of those conclusions was that the kernel should always fill in the fe_physical_length field in the returned extent, and set a flag:
#define FIEMAP_EXTENT_PHYS_LENGTH 0x00000010
to indicate to userspace that the physical length field is valid.
There should also be a separate flag for extents that are compressed:
#define FIEMAP_EXTENT_DATA_COMPRESSED 0x00000040
Rename fe_length to fe_logical_length and #define fe_length fe_logical_length so that it is more clear which field is which in the data structure, but does not break compatibility.
I think this patch gets most of this right, except the presence of the flags to indicate the PHYS_LENGTH and DATA_COMPRESSED state in the extent.
Cheers, Andreas
Quoted 94 lines
[1] https://github.com/kilobyte/compsize
Signed-off-by Sweet Tea Dorminy <[email protected]>---
Documentation/filesystems/fiemap.rst | 26 ++++++++++++++++++-------- fs/ioctl.c | 1 + include/uapi/linux/fiemap.h | 24 +++++++++++++++++------- 3 files changed, 36 insertions(+), 15 deletions(-)Documentation/filesystems/fiemap.rst +0
-0@@ -80,14 +80,24 @@ Each extent is described by a single fiemap_extent structure asreturned in
fm_extents::struct fiemap_extent {Documentation/filesystems/fiemap.rst +18
-8- __u64 fe_logical; /* logical offset in bytes for the start of - * the extent */ - __u64 fe_physical; /* physical offset in bytes for the start - * of the extent */ - __u64 fe_length; /* length in bytes for the extent */ - __u64 fe_reserved64[2]; - __u32 fe_flags; /* FIEMAP_EXTENT_* flags for this extent */ - __u32 fe_reserved[3]; + /* + * logical offset in bytes for the start of + * the extent from the beginning of the file + */ + __u64 fe_logical; + /* + * physical offset in bytes for the start + * of the extent from the beginning of the disk + */ + __u64 fe_physical; + /* length in bytes for this extent */ + __u64 fe_length; + /* physical length in bytes for this extent */ + __u64 fe_physical_length; + __u64 fe_reserved64[1]; + /* FIEMAP_EXTENT_* flags for this extent */ + __u32 fe_flags; + __u32 fe_reserved[3]; };All offsets and lengths are in bytes and mirror those on disk. It is valid
fs/ioctl.c +1
-0@@ -139,6 +139,7 @@ int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical, extent.fe_logical = logical; extent.fe_physical = phys; extent.fe_length = len; + extent.fe_physical_length = len; extent.fe_flags = flags; dest += fieinfo->fi_extents_mapped;include/uapi/linux/fiemap.h +17
-8@@ -15,13 +15,23 @@#include <linux/types.h> struct fiemap_extent { -__u64 fe_logical;/* logical offset in bytes for the start of -* the extent from the beginning of the file */ -__u64 fe_physical;/* physical offset in bytes for the start -* of the extent from the beginning of the disk */ -__u64 fe_length;/* length in bytes for this extent */ -__u64 fe_reserved64[2 ]; -__u32 fe_flags;/* FIEMAP_EXTENT_* flags for this extent */ +/* +* logical offset in bytes for the start of +* the extent from the beginning of the file +*/ +__u64 fe_logical; +/* +* physical offset in bytes for the start +* of the extent from the beginning of the disk +*/ +__u64 fe_physical; +/* length in bytes for this extent */ +__u64 fe_length; +/* physical length in bytes for this extent */ +__u64 fe_physical_length; +__u64 fe_reserved64[1 ]; +/* FIEMAP_EXTENT_* flags for this extent */ +__u32 fe_flags;__u32 fe_reserved[3 ]; }; --
Quoted 3 lines
2.44.0
Cheers, Andreas
Re: [PATCH 1/3] add physical_length field to fiemap extents
Quoted 32 lines · Andreas Dilger
Quoted 11 lines · Sweet Tea Dorminy
Some filesystems support compressed extents which have a larger logical size than physical, and for those filesystems, it can be useful for userspace to know how much space those extents actually use. For instance, the compsize [1] tool for btrfs currently uses btrfs-internal, root-only ioctl to find the actual disk space used by a file; it would be better and more useful for this information to require fewer privileges and to be usable on more filesystems. Therefore, use one of the padding u64s in the fiemap extent structure to return the actual physical length; and, for now, return this as equal to the logical length.
Thank you for working on this patch. Note that there was a patch from David Sterba and a lengthy discussion about exactly this functionality several years ago. If you haven't already read the details, it would be useful to do so. I think the thread had mostly come to good conclusions, but the patch never made it into the kernel.
One of those conclusions was that the kernel should always fill in the
fe_physical_lengthfield in the returned extent, and set a flag:#define
FIEMAP_EXTENT_PHYS_LENGTH0x00000010to indicate to userspace that the physical length field is valid.
There should also be a separate flag for extents that are compressed:
#define
FIEMAP_EXTENT_DATA_COMPRESSED0x00000040
Quoted 8 lines
Rename
fe_lengthtofe_logical_lengthand #definefe_lengthfe_logical_lengthso that it is more clear which field is which in the data structure, but does not break compatibility.I think this patch gets most of this right, except the presence of the flags to indicate the
PHYS_LENGTHandDATA_COMPRESSEDstate in the extent.Cheers, Andreas
Thanks for resurrecting this. Andreas's suggestions sound good to me. And yes, please try to search for any past discussions on this topic.
It may be a good idea to Cc the f2fs mailing list ([email protected]), since this will be useful for f2fs too, since f2fs supports compression.
One use case is that this will make testing the combination of compression+encryption (e.g. as xfstest f2fs/002 tries to do) easier.
- Eric
Re: [PATCH 1/3] add physical_length field to fiemap extents
In reply to Andreas DilgerQuoted 33 lines · Andreas Dilger
Quoted 11 lines · Sweet Tea Dorminy
Some filesystems support compressed extents which have a larger logical size than physical, and for those filesystems, it can be useful for userspace to know how much space those extents actually use. For instance, the compsize [1] tool for btrfs currently uses btrfs-internal, root-only ioctl to find the actual disk space used by a file; it would be better and more useful for this information to require fewer privileges and to be usable on more filesystems. Therefore, use one of the padding u64s in the fiemap extent structure to return the actual physical length; and, for now, return this as equal to the logical length.
Thank you for working on this patch. Note that there was a patch from David Sterba and a lengthy discussion about exactly this functionality several years ago. If you haven't already read the details, it would be useful to do so. I think the thread had mostly come to good conclusions, but the patch never made it into the kernel.
One of those conclusions was that the kernel should always fill in the
fe_physical_lengthfield in the returned extent, and set a flag:#define
FIEMAP_EXTENT_PHYS_LENGTH0x00000010to indicate to userspace that the physical length field is valid.
There should also be a separate flag for extents that are compressed:
#define
FIEMAP_EXTENT_DATA_COMPRESSED0x00000040
Quoted 9 lines
Rename
fe_lengthtofe_logical_lengthand #definefe_lengthfe_logical_lengthso that it is more clear which field is which in the data structure, but does not break compatibility.I think this patch gets most of this right, except the presence of the flags to indicate the
PHYS_LENGTHandDATA_COMPRESSEDstate in the extent.Cheers, Andreas
I had not seen that; thank you for the pointers, I'll add the flags in v2.
PATCH 2/3 fs: update fiemap_fill_next_extent() signature
In reply to Sweet Tea DorminyUpdate the signature of fiemap_fill_next_extent() to allow passing a physical length, and update all callers to pass the same logical and physical lengths.
--- Documentation/filesystems/fiemap.rst | 3 ++- fs/bcachefs/fs.c | 6 +++++- fs/btrfs/extent_io.c | 2 +- fs/ext4/extents.c | 1 + fs/f2fs/data.c | 8 +++++--- fs/f2fs/inline.c | 3 ++- fs/ioctl.c | 9 +++++---- fs/iomap/fiemap.c | 2 +- fs/nilfs2/inode.c | 8 +++++--- fs/ntfs3/frecord.c | 6 ++++-- fs/ocfs2/extent_map.c | 4 ++-- fs/smb/client/smb2ops.c | 1 + include/linux/fiemap.h | 2 +- 13 files changed, 35 insertions(+), 20 deletions(-)
Documentation/filesystems/fiemap.rst +2 -1
@@ -234,7 +234,8 @@ For each extent in the request range, the file system should call the helper function, fiemap_fill_next_extent():: int fiemap_fill_next_extent(struct fiemap_extent_info *info, u64 logical, - u64 phys, u64 len, u32 flags, u32 dev); + u64 phys, u64 log_len, u64 phys_len, u32 flags, + u32 dev); fiemap_fill_next_extent() will use the passed values to populate the next free extent in the fm_extents array. 'General' extent flags will
fs/bcachefs/fs.c +5 -1
@@ -927,7 +927,9 @@ static int bch2_fill_extent(struct bch_fs *c, ret =fiemap_fill_next_extent (info,bkey_start_offset (k.k) <<9 , offset <<9 , - k.k->size <<9 , flags|flags2); + k.k->size <<9 , + k.k->size <<9 , + flags|flags2);if (ret)return ret; }
@@ -937,12 +939,14 @@ static int bch2_fill_extent(struct bch_fs *c,return fiemap_fill_next_extent (info,bkey_start_offset (k.k) <<9 ,0 , k.k->size <<9 , + k.k->size <<9 , flags|FIEMAP_EXTENT_DATA_INLINE ); }else if (k.k->type == KEY_TYPE_reservation) {return fiemap_fill_next_extent (info,bkey_start_offset (k.k) <<9 ,0 , k.k->size <<9 , + k.k->size <<9 , flags|FIEMAP_EXTENT_DELALLOC |FIEMAP_EXTENT_UNWRITTEN );
fs/btrfs/extent_io.c +1 -1
@@ -2743,7 +2743,7 @@ static int emit_last_fiemap_cache(struct fiemap_extent_info *fieinfo,return 0 ; ret =fiemap_fill_next_extent (fieinfo, cache->offset, cache->phys, - cache->len, cache->flags); + cache->len, cache->len, cache->flags); cache->cached = false;if (ret >0 ) ret =0 ;
fs/ext4/extents.c +1 -0
@@ -2215,6 +2215,7 @@ static int ext4_fill_es_cache_info(struct inode *inode, (__u64 )es.es_lblk << blksize_bits, (__u64 )es.es_pblk << blksize_bits, (__u64 )es.es_len << blksize_bits, + (__u64 )es.es_len << blksize_bits, flags);if (next ==0 )break ;
fs/f2fs/data.c +5 -3
@@ -1843,7 +1843,8 @@ static int f2fs_xattr_fiemap(struct inode *inode,if (!xnid) flags |=FIEMAP_EXTENT_LAST ; - err =fiemap_fill_next_extent (fieinfo,0 , phys, len, flags); + err =fiemap_fill_next_extent ( + fieinfo,0 , phys, len, len, flags);trace_f2fs_fiemap (inode,0 , phys, len, flags, err);if (err)return err;
@@ -1869,7 +1870,8 @@ static int f2fs_xattr_fiemap(struct inode *inode, }if (phys) { - err =fiemap_fill_next_extent (fieinfo,0 , phys, len, flags); + err =fiemap_fill_next_extent ( + fieinfo,0 , phys, len, len, flags);trace_f2fs_fiemap (inode,0 , phys, len, flags, err); }
@@ -1988,7 +1990,7 @@ int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, flags |=FIEMAP_EXTENT_DATA_ENCRYPTED ; ret =fiemap_fill_next_extent (fieinfo, logical, - phys, size, flags); + phys, size, size, flags);trace_f2fs_fiemap (inode, logical, phys, size, flags, ret);if (ret) goto out;
fs/f2fs/inline.c +2 -1
@@ -806,7 +806,8 @@ int f2fs_inline_data_fiemap(struct inode *inode, byteaddr = (__u64 )ni.blk_addr << inode->i_sb->s_blocksize_bits; byteaddr += (char *)inline_data_addr (inode, ipage) - (char *)F2FS_INODE (ipage); - err =fiemap_fill_next_extent (fieinfo, start, byteaddr, ilen, flags); + err =fiemap_fill_next_extent ( + fieinfo, start, byteaddr, ilen, ilen, flags);trace_f2fs_fiemap (inode, start, byteaddr, ilen, flags, err); out:f2fs_put_page (ipage,1 );
fs/ioctl.c +5 -4
@@ -99,7 +99,8 @@ static int ioctl_fibmap(struct file *filp, int __user *p)* @fieinfo: Fiemap context passed into ->fiemap * @logical: Extent logical start offset, in bytes * @phys: Extent physical start offset, in bytes -* @len: Extent length, in bytes +* @log_len: Extent logical length, in bytes +* @phys_len: Extent physical length, in bytes * @flags: FIEMAP_EXTENT flags that describe this extent * * Called from file system ->fiemap callback. Will populate extent
@@ -110,7 +111,7 @@ static int ioctl_fibmap(struct file *filp, int __user *p)* extent that will fit in user array. */ int fiemap_fill_next_extent (struct fiemap_extent_info *fieinfo,u64 logical, -u64 phys,u64 len,u32 flags) +u64 phys,u64 log_len,u64 phys_len,u32 flags) {struct fiemap_extent extent;struct fiemap_extent __user *dest = fieinfo->fi_extents_start;
@@ -138,8 +139,8 @@ int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical,memset (&extent,0 ,sizeof (extent)); extent.fe_logical = logical; extent.fe_physical = phys; - extent.fe_length = len; - extent.fe_physical_length = len; + extent.fe_length = log_len; + extent.fe_physical_length = phys_len; extent.fe_flags = flags; dest += fieinfo->fi_extents_mapped;
fs/iomap/fiemap.c +1 -1
@@ -36,7 +36,7 @@ static int iomap_to_fiemap(struct fiemap_extent_info *fi,return fiemap_fill_next_extent (fi, iomap->offset, iomap->addr !=IOMAP_NULL_ADDR ? iomap->addr :0 , - iomap->length, flags); + iomap->length, iomap->length, flags); } static loff_t iomap_fiemap_iter(const struct iomap_iter *iter,
fs/nilfs2/inode.c +5 -3
@@ -1190,7 +1190,8 @@ int nilfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,if (size) {/* End of the current extent */ ret =fiemap_fill_next_extent ( - fieinfo, logical, phys, size, flags); + fieinfo, logical, phys, size, size, + flags);if (ret)break ; }
@@ -1240,7 +1241,8 @@ int nilfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, flags |=FIEMAP_EXTENT_LAST ; ret =fiemap_fill_next_extent ( - fieinfo, logical, phys, size, flags); + fieinfo, logical, phys, size, size, + flags);if (ret)break ; size =0 ;
@@ -1256,7 +1258,7 @@ int nilfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,/* Terminate the current extent */ ret =fiemap_fill_next_extent ( fieinfo, logical, phys, size, - flags); + size, flags);if (ret || blkoff > end_blkoff)break ;
fs/ntfs3/frecord.c +4 -2
@@ -1948,6 +1948,7 @@ int ni_fiemap(struct ntfs_inode *ni, struct fiemap_extent_info *fieinfo, err =fiemap_fill_next_extent ( fieinfo,0 ,0 , attr ?le32_to_cpu (attr->res.data_size) :0 , + attr ?le32_to_cpu (attr->res.data_size) :0 ,FIEMAP_EXTENT_DATA_INLINE |FIEMAP_EXTENT_LAST |FIEMAP_EXTENT_MERGED ); goto out;
@@ -2042,7 +2043,7 @@ int ni_fiemap(struct ntfs_inode *ni, struct fiemap_extent_info *fieinfo, flags |=FIEMAP_EXTENT_LAST ; err =fiemap_fill_next_extent (fieinfo, vbo, lbo, dlen, - flags); + dlen, flags);if (err <0 )break ;if (err ==1 ) {
@@ -2062,7 +2063,8 @@ int ni_fiemap(struct ntfs_inode *ni, struct fiemap_extent_info *fieinfo,if (vbo + bytes >= end) flags |=FIEMAP_EXTENT_LAST ; - err =fiemap_fill_next_extent (fieinfo, vbo, lbo, bytes, flags); + err =fiemap_fill_next_extent (fieinfo, vbo, lbo, bytes, bytes, + flags);if (err <0 )break ;if (err ==1 ) {
fs/ocfs2/extent_map.c +2 -2
@@ -723,7 +723,7 @@ static int ocfs2_fiemap_inline(struct inode *inode, struct buffer_head *di_bh, id2.i_data.id_data); ret =fiemap_fill_next_extent (fieinfo,0 , phys, id_count, - flags); + id_count, flags);if (ret <0 )return ret; }
@@ -794,7 +794,7 @@ int ocfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, virt_bytes = (u64 )le32_to_cpu (rec.e_cpos) << osb->s_clustersize_bits; ret =fiemap_fill_next_extent (fieinfo, virt_bytes, phys_bytes, - len_bytes, fe_flags); + len_bytes, len_bytes, fe_flags);if (ret)break ;
fs/smb/client/smb2ops.c +1 -0
@@ -3778,6 +3778,7 @@ static int smb3_fiemap(struct cifs_tcon *tcon,le64_to_cpu (out_data[i].file_offset),le64_to_cpu (out_data[i].file_offset),le64_to_cpu (out_data[i].length), +le64_to_cpu (out_data[i].length), flags);if (rc <0 ) goto out;
include/linux/fiemap.h +1 -1
@@ -16,6 +16,6 @@ struct fiemap_extent_info {int fiemap_prep (struct inode *inode,struct fiemap_extent_info *fieinfo,u64 start,u64 *len,u32 supported_flags);int fiemap_fill_next_extent (struct fiemap_extent_info *info,u64 logical, -u64 phys,u64 len,u32 flags); +u64 phys,u64 log_len,u64 phys_len,u32 flags);#endif /* _LINUX_FIEMAP_H 1 */
Signature · 1 line
2.44.0
PATCH 3/3 btrfs: fiemap: return extent physical size
In reply to Sweet Tea DorminyNow that fiemap allows returning extent physical size, make btrfs return the appropriate extent's actual disk size.
--- fs/btrfs/extent_io.c | 63 +++++++++++++++++++++++++++----------------- 1 file changed, 39 insertions(+), 24 deletions(-)
fs/btrfs/extent_io.c +39 -24
@@ -2456,7 +2456,8 @@ int try_release_extent_mapping(struct page *page, gfp_t mask)struct btrfs_fiemap_entry {u64 offset;u64 phys; -u64 len; +u64 log_len; +u64 phys_len;u32 flags; };
@@ -2514,7 +2515,8 @@ struct fiemap_cache {/* Fields for the cached extent (unsubmitted, not ready, extent). */ u64 offset;u64 phys; -u64 len; +u64 log_len; +u64 phys_len;u32 flags;bool cached; };
@@ -2527,8 +2529,8 @@ static int flush_fiemap_cache(struct fiemap_extent_info *fieinfo,int ret; ret =fiemap_fill_next_extent (fieinfo, entry->offset, - entry->phys, entry->len, - entry->flags); + entry->phys, entry->log_len, + entry->phys_len, entry->flags);/* * Ignore 1 (reached max entries) because we keep track of that * ourselves in emit_fiemap_extent().
@@ -2553,7 +2555,8 @@ static int flush_fiemap_cache(struct fiemap_extent_info *fieinfo,*/ static int emit_fiemap_extent (struct fiemap_extent_info *fieinfo,struct fiemap_cache *cache, -u64 offset,u64 phys,u64 len,u32 flags) +u64 offset,u64 phys,u64 log_len, +u64 phys_len,u32 flags) {struct btrfs_fiemap_entry *entry;u64 cache_end;
@@ -2596,7 +2599,7 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,* or equals to what we have in cache->offset. We deal with this as * described below. */ - cache_end = cache->offset + cache->len; + cache_end = cache->offset + cache->log_len;if (cache_end > offset) {if (offset == cache->offset) {/*
@@ -2620,10 +2623,10 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,* where a previously found file extent item was split * due to an ordered extent completing. */ - cache->len = offset - cache->offset; + cache->log_len = offset - cache->offset; goto emit; }else { -const u64 range_end = offset + len; +const u64 range_end = offset + log_len;/* * The offset of the file extent item we have just found
@@ -2660,7 +2663,7 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo, phys += cache_end - offset; offset = cache_end; - len = range_end - cache_end; + log_len = range_end - cache_end; goto emit; } }
@@ -2670,15 +2673,17 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,* 1) Their logical addresses are continuous * * 2) Their physical addresses are continuous -* So truly compressed (physical size smaller than logical size) -* extents won't get merged with each other * * 3) Share same flags +* +* 4) Not compressed */ -if (cache->offset + cache->len == offset && - cache->phys + cache->len == phys && - cache->flags == flags) { - cache->len += len; +if (cache->offset + cache->log_len == offset && + cache->phys + cache->log_len == phys && + cache->flags == flags && + !(flags &FIEMAP_EXTENT_ENCODED )) { + cache->log_len += log_len; + cache->phys_len += phys_len;return 0 ; }
@@ -2695,7 +2700,7 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,* to miss it. */ entry = &cache->entries[cache->entries_size -1 ]; - cache->next_search_offset = entry->offset + entry->len; + cache->next_search_offset = entry->offset + entry->log_len; cache->cached = false;return BTRFS_FIEMAP_FLUSH_CACHE ;
@@ -2704,7 +2709,8 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo, entry = &cache->entries[cache->entries_pos]; entry->offset = cache->offset; entry->phys = cache->phys; - entry->len = cache->len; + entry->log_len = cache->log_len; + entry->phys_len = cache->phys_len; entry->flags = cache->flags; cache->entries_pos++; cache->extents_mapped++;
@@ -2717,7 +2723,8 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo, cache->cached = true; cache->offset = offset; cache->phys = phys; - cache->len = len; + cache->log_len = log_len; + cache->phys_len = phys_len; cache->flags = flags;return 0 ;
@@ -2743,7 +2750,8 @@ static int emit_last_fiemap_cache(struct fiemap_extent_info *fieinfo,return 0 ; ret =fiemap_fill_next_extent (fieinfo, cache->offset, cache->phys, - cache->len, cache->len, cache->flags); + cache->log_len, cache->phys_len, + cache->flags); cache->cached = false;if (ret >0 ) ret =0 ;
@@ -2937,13 +2945,15 @@ static int fiemap_process_hole(struct btrfs_inode *inode, } ret =emit_fiemap_extent (fieinfo, cache, prealloc_start, disk_bytenr + extent_offset, - prealloc_len, prealloc_flags); + prealloc_len, prealloc_len, + prealloc_flags);if (ret)return ret; extent_offset += prealloc_len; } ret =emit_fiemap_extent (fieinfo, cache, delalloc_start,0 , + delalloc_end +1 - delalloc_start, delalloc_end +1 - delalloc_start,FIEMAP_EXTENT_DELALLOC |FIEMAP_EXTENT_UNKNOWN );
@@ -2984,7 +2994,8 @@ static int fiemap_process_hole(struct btrfs_inode *inode, } ret =emit_fiemap_extent (fieinfo, cache, prealloc_start, disk_bytenr + extent_offset, - prealloc_len, prealloc_flags); + prealloc_len, prealloc_len, + prealloc_flags);if (ret)return ret; }
@@ -3130,6 +3141,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,u64 extent_offset =0 ;u64 extent_gen;u64 disk_bytenr =0 ; +u64 disk_size =0 ;u64 flags =0 ;int extent_type;u8 compression;
@@ -3192,7 +3204,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo, flags |=FIEMAP_EXTENT_DATA_INLINE ; flags |=FIEMAP_EXTENT_NOT_ALIGNED ; ret =emit_fiemap_extent (fieinfo, &cache, key.offset,0 , - extent_len, flags); + extent_len, extent_len, flags); }else if (extent_type ==BTRFS_FILE_EXTENT_PREALLOC ) { ret = fiemap_process_hole(inode, fieinfo, &cache, &delalloc_cached_state,
@@ -3207,6 +3219,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo, backref_ctx,0 ,0 ,0 , key.offset, extent_end - 1); }else { + disk_size =btrfs_file_extent_disk_num_bytes (leaf, ei);/* We have a regular extent. */ if (fieinfo->fi_extents_max) { ret = btrfs_is_data_extent_shared(inode,
@@ -3221,7 +3234,9 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo, ret =emit_fiemap_extent (fieinfo, &cache, key.offset, disk_bytenr + extent_offset, - extent_len, flags); + extent_len, + disk_size - extent_offset, + flags); }if (ret <0 ) {
@@ -3259,7 +3274,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo, prev_extent_end = range_end; } -if (cache.cached && cache.offset + cache.len >= last_extent_end) { +if (cache.cached && cache.offset + cache.log_len >= last_extent_end) {const u64 i_size =i_size_read (&inode->vfs_inode);if (prev_extent_end < i_size) {
Signature · 1 line
2.44.0
Quoted 12 lines · Sweet Tea Dorminy
For many years, various btrfs users have written programs to discover the actual disk space used by files, using root-only interfaces. However, this information is a great fit for fiemap: it is inherently tied to extent information, all filesystems can use it, and the capabilities required for FIEMAP make sense for this additional information also.
Hence, this patchset adds physical extent length information to fiemap, and extends btrfs to return it. This uses some of the reserved padding in the fiemap extent structure, so programs unaware of the new field will be unaffected by its presence.
Quoted 4 lines
This is based on next-20240307. I've tested the btrfs part of this with the standard btrfs testing matrix locally, and verified that the physical extent information returned there is correct, but I'm still waiting on more tests. Please let me know what you think of the general idea!
Seems useful! Any chance you'd be willing to pick up this old proposal to report the dev_t through iomap? iirc the iomap wrappers for fiemap can export that pretty easily.
https://lore.kernel.org/linux-fsdevel/[email protected]/
(Not sure what we do for pmem filesystems)
--D
Quoted 27 lines
Sweet Tea Dorminy (3):
fs: addphysical_lengthfield to fiemap extents
fs: updatefiemap_fill_next_extent()signature
btrfs: fiemap: return extent physical sizeDocumentation/filesystems/fiemap.rst | 29 +++++++++---- fs/bcachefs/fs.c | 6 ++- fs/btrfs/extent_io.c | 63 +++++++++++++++++----------- fs/ext4/extents.c | 1 + fs/f2fs/data.c | 8 ++-- fs/f2fs/inline.c | 3 +- fs/ioctl.c | 8 ++-- fs/iomap/fiemap.c | 2 +- fs/nilfs2/inode.c | 8 ++-- fs/ntfs3/frecord.c | 6 ++- fs/ocfs2/extent_map.c | 4 +- fs/smb/client/smb2ops.c | 1 + include/linux/fiemap.h | 2 +- include/uapi/linux/fiemap.h | 24 +++++++---- 14 files changed, 108 insertions(+), 57 deletions(-)base-commit:
1843e16d2df9d98427ef8045589571749d627cf7Signature · 1 line
2.44.0
Quoted 18 lines · Darrick J. Wong
Quoted 12 lines · Sweet Tea Dorminy
For many years, various btrfs users have written programs to discover the actual disk space used by files, using root-only interfaces. However, this information is a great fit for fiemap: it is inherently tied to extent information, all filesystems can use it, and the capabilities required for FIEMAP make sense for this additional information also.
Hence, this patchset adds physical extent length information to fiemap, and extends btrfs to return it. This uses some of the reserved padding in the fiemap extent structure, so programs unaware of the new field will be unaffected by its presence.
Quoted 4 lines
This is based on next-20240307. I've tested the btrfs part of this with the standard btrfs testing matrix locally, and verified that the physical extent information returned there is correct, but I'm still waiting on more tests. Please let me know what you think of the general idea!
Quoted 5 lines
Seems useful! Any chance you'd be willing to pick up this old proposal to report the
dev_tthrough iomap? iirc the iomap wrappers for fiemap can export that pretty easily.
I think this is not too useful for btrfs (in general) due to the block group profiles that store copies on multiple devices, we'd need more than one device identifier per extent.
Quoted 24 lines · David Sterba
Quoted 22 lines · Darrick J. Wong
Quoted 12 lines · Sweet Tea Dorminy
For many years, various btrfs users have written programs to discover the actual disk space used by files, using root-only interfaces. However, this information is a great fit for fiemap: it is inherently tied to extent information, all filesystems can use it, and the capabilities required for FIEMAP make sense for this additional information also.
Hence, this patchset adds physical extent length information to fiemap, and extends btrfs to return it. This uses some of the reserved padding in the fiemap extent structure, so programs unaware of the new field will be unaffected by its presence.
Quoted 4 lines
This is based on next-20240307. I've tested the btrfs part of this with the standard btrfs testing matrix locally, and verified that the physical extent information returned there is correct, but I'm still waiting on more tests. Please let me know what you think of the general idea!
Seems useful! Any chance you'd be willing to pick up this old proposal to report the
dev_tthrough iomap? iirc the iomap wrappers for fiemap can export that pretty easily.
Quoted 5 lines
I think this is not too useful for btrfs (in general) due to the block group profiles that store copies on multiple devices, we'd need more than one device identifier per extent.
My thought would be that there are multiple overlapping extents with the same logical offset returned in this case. It wouldn't just be the device that would be different in this case, but also the physical offset may be different on each device (depending on how allocation is done), and maybe even the length and flags are different if one device stores compressed data and another one does not.
Having multiple overlapping extents for files with built-in mirrors allows freedom for all of the extent parameters to be different, doesn't have any limits on the number of copies/devices that could fit in one extent, etc.
Cheers, Andreas