fiemap: introduce EXTENT_DATA_COMPRESSED flag
Subsystems (9)
v4 cover fiemap: introduce EXTENT_DATA_COMPRESSED flag
The original FIEMAP patch did not define this bit, btrfs will make use of it. The defined constant maintains the same value as originally proposed.
Currently, the 'filefrag' utility has no way to recognize and denote a compressed extent. As implemented in btrfs right now, the compression step splits a big extent into smaller chunks and this is reported as a heavily fragmented file. Adding the flag to filefrag will at least give some explanation why, this has been confusing users for some time already.
fiemap_fill_next_extent is extended and takes argument to fill the physical length.
V4:
The physical length is always set and equal to logical, or different and then sets the COMPRESSED flag.fiemap_extent::fe_length renamed to fe_logi_length
V3:
Based on feedback from Andreas, implement #1 from V2, current users of fiemap_fill_next_extent (fs/, ext4, gfs2, ocfs2, nilfs2, xfs) updated accordingly, no functional change.
V2:
Based on feedback from Andreas, the fiemap_extent is now able to hold the physical extent length, to be filled by the filesystem callback.
1) extend fiemap_fill_next_extent to take phys_length and update all users (ext4, gfs2, ocfs2, nilfs2, xfs)
David Sterba (5):
fiemap: fix comment at EXTENT_DATA_ENCRYPTED
fiemap: add EXTENT_DATA_COMPRESSED flag
btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents Documentation/fiemap: Document the DATA_COMPRESSED flag
fiemap: rename fe_length to fe_logi_length
Documentation/filesystems/fiemap.txt | 19 +++++++++++++++---- fs/btrfs/extent_io.c | 8 ++++++-- fs/ext4/extents.c | 3 ++- fs/ext4/inline.c | 2 +- fs/gfs2/inode.c | 2 +- fs/ioctl.c | 29 ++++++++++++++++++++++------- fs/nilfs2/inode.c | 8 +++++--- fs/ocfs2/extent_map.c | 4 ++-- fs/xfs/xfs_iops.c | 2 +- include/linux/fs.h | 2 +- include/uapi/linux/fiemap.h | 13 ++++++++++--- 11 files changed, 66 insertions(+), 26 deletions(-)
Signature · 1 line
1.8.4.5
PATCH v4 1/5 fiemap: fix comment at EXTENT_DATA_ENCRYPTED
The flag was named EXTENT_NO_BYPASS in the original fiemap proposal[1], but renamed to EXTENT_ENCODED afterwards.
[1] http://article.gmane.org/gmane.comp.file-systems.ext4/8871
Signed-off-by David Sterba <[email protected]>
--- include/uapi/linux/fiemap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/uapi/linux/fiemap.h +1 -1
@@ -51,7 +51,7 @@ struct fiemap {#define FIEMAP_EXTENT_ENCODED 0x00000008/* Data can not be read * while fs is unmounted */ #define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080/* Data is encrypted by fs. -* Sets EXTENT_NO_BYPASS. */ +* Sets EXTENT_ENCODED */ #define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100/* Extent offsets may not be * block aligned. */ #define FIEMAP_EXTENT_DATA_INLINE 0x00000200/* Data mixed with metadata.
Signature · 1 line
1.8.4.5
PATCH v4 2/5 fiemap: add EXTENT_DATA_COMPRESSED flag
In reply to David SterbaThis flag was not accepted when fiemap was proposed [2] due to lack of in-kernel users. Btrfs has compression for a long time and we'd like to see that an extent is compressed in the output of 'filefrag' utility once it's taught about it.
For that purpose, a reserved field from fiemap_extent is used to let the filesystem store along the physcial extent length when the flag is set. This keeps compatibility with applications that use FIEMAP.
Extend arguments of fiemap_fill_next_extent and update all users.
[1] http://article.gmane.org/gmane.comp.file-systems.ext4/8871
[2] http://thread.gmane.org/gmane.comp.file-systems.ext4/8870
[3] http://thread.gmane.org/gmane.linux.file-systems/77632 (v1)
[4] http://www.spinics.net/lists/linux-fsdevel/msg69078.html (v2)
Cc: Al Viro <[email protected]>
CC: Andreas Dilger <[email protected]>
CC: Chris Mason <[email protected]>
CC: Christoph Hellwig <[email protected]> CC KONISHI Ryusuke <[email protected]>
CC Mark Fasheh <[email protected]>
CC Steven Whitehouse <[email protected]>
CC "Theodore Ts'o" <[email protected]>
CC Ben Myers <[email protected]>
Signed-off-by David Sterba <[email protected]>
--- fs/btrfs/extent_io.c | 2 +- fs/ext4/extents.c | 3 ++- fs/ext4/inline.c | 2 +- fs/gfs2/inode.c | 2 +- fs/ioctl.c | 27 +++++++++++++++++++++------ fs/nilfs2/inode.c | 8 +++++--- fs/ocfs2/extent_map.c | 4 ++-- fs/xfs/xfs_iops.c | 2 +- include/linux/fs.h | 2 +- include/uapi/linux/fiemap.h | 8 +++++++- 10 files changed, 42 insertions(+), 18 deletions(-)
fs/btrfs/extent_io.c +1 -1
@@ -4357,7 +4357,7 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, end =1 ; } ret =fiemap_fill_next_extent (fieinfo, em_start, disko, - em_len, flags); + em_len, em_len, flags);if (ret) goto out_free; }
fs/ext4/extents.c +2 -1
@@ -2253,6 +2253,7 @@ static int ext4_fill_fiemap_extents(struct inode *inode, (__u64 )es.es_lblk << blksize_bits, (__u64 )es.es_pblk << blksize_bits, (__u64 )es.es_len << blksize_bits, + (__u64 )es.es_len << blksize_bits, flags);if (err <0 )break ;
@@ -5125,7 +5126,7 @@ static int ext4_xattr_fiemap(struct inode *inode,if (physical) error =fiemap_fill_next_extent (fieinfo,0 , physical, - length, flags); + length, length, flags);return (error <0 ? error :0 ); }
fs/ext4/inline.c +1 -1
@@ -1825,7 +1825,7 @@ int ext4_inline_data_fiemap(struct inode *inode,if (physical) error =fiemap_fill_next_extent (fieinfo,0 , physical, - length, flags); + length, length, flags);brelse (iloc.bh); out:up_read (&EXT4_I (inode)->xattr_sem);
fs/gfs2/inode.c +1 -1
@@ -1931,7 +1931,7 @@ static int gfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, len = size - start;if (start < size) ret =fiemap_fill_next_extent (fieinfo, start, phys, - len, flags); + len, len, flags);if (ret ==1 ) ret =0 ; }else {
fs/ioctl.c +21 -6
@@ -70,20 +70,26 @@ static int ioctl_fibmap(struct file *filp, int __user *p)* @logical: Extent logical start offset, in bytes * @phys: Extent physical start offset, in bytes * @len: Extent length, in bytes +* @phys_len: Physical extent length in bytes * @flags: FIEMAP_EXTENT flags that describe this extent * * Called from file system ->fiemap callback. Will populate extent * info as passed in via arguments and copy to user memory. On * success, extent count on fieinfo is incremented. * +* Extents without any encoding must set the physical and logical length +* to the same value. Otherwise, set flags to FIEMAP_EXTENT_ENCODED +* and possibly specify encoding type. +* * Returns 0 on success, -errno on error, 1 if this was the last * extent that will fit in user array. */ #define SET_UNKNOWN_FLAGS (FIEMAP_EXTENT_DELALLOC) -#define SET_NO_UNMOUNTED_IO_FLAGS (FIEMAP_EXTENT_DATA_ENCRYPTED) +#define SET_NO_UNMOUNTED_IO_FLAGS (FIEMAP_EXTENT_DATA_ENCRYPTED | \ + FIEMAP_EXTENT_DATA_COMPRESSED)#define SET_NOT_ALIGNED_FLAGS (FIEMAP_EXTENT_DATA_TAIL|FIEMAP_EXTENT_DATA_INLINE)int fiemap_fill_next_extent (struct fiemap_extent_info *fieinfo,u64 logical, -u64 phys,u64 len,u32 flags) +u64 phys,u64 len,u64 phys_len,u32 flags) {struct fiemap_extent extent;struct fiemap_extent __user *dest = fieinfo->fi_extents_start;
@@ -110,6 +116,14 @@ int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical, extent.fe_length = len; extent.fe_flags = flags; +WARN_ONCE ((flags &FIEMAP_EXTENT_DATA_COMPRESSED ) + && !(flags &FIEMAP_EXTENT_ENCODED )); +WARN_ONCE (phys_len != len && !(flags &FIEMAP_EXTENT_DATA_COMPRESSED ), +"physical length %llu != logical length %llu without = DATA_COMPRESSED\n" , + phys_len, len); + + extent.fe_phys_length = phys_len; + dest += fieinfo->fi_extents_mapped;if (copy_to_user (dest, &extent,sizeof (extent)))return -EFAULT ;
@@ -318,10 +332,11 @@ int __generic_block_fiemap(struct inode *inode, flags =FIEMAP_EXTENT_MERGED |FIEMAP_EXTENT_LAST ; ret =fiemap_fill_next_extent (fieinfo, logical, phys, size, - flags); + size, flags); }else if (size) { ret =fiemap_fill_next_extent (fieinfo, logical, - phys, size, flags); + phys, size, + size, flags); size =0 ; }
@@ -347,7 +362,7 @@ int __generic_block_fiemap(struct inode *inode,if (start_blk > last_blk && !whole_file) { ret =fiemap_fill_next_extent (fieinfo, logical, phys, size, - flags); + size, flags);break ; }
@@ -358,7 +373,7 @@ int __generic_block_fiemap(struct inode *inode,if (size) { ret =fiemap_fill_next_extent (fieinfo, logical, phys, size, - flags); + size, flags);if (ret)break ; }
fs/nilfs2/inode.c +5 -3
@@ -1017,7 +1017,8 @@ int nilfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,if (size) {/* End of the current extent */ ret =fiemap_fill_next_extent ( - fieinfo, logical, phys, size, flags); + fieinfo, logical, phys, size, size, + flags);if (ret)break ; }
@@ -1067,7 +1068,8 @@ int nilfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, flags |=FIEMAP_EXTENT_LAST ; ret =fiemap_fill_next_extent ( - fieinfo, logical, phys, size, flags); + fieinfo, logical, phys, size, + size, flags);if (ret)break ; size =0 ;
@@ -1083,7 +1085,7 @@ int nilfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,/* Terminate the current extent */ ret =fiemap_fill_next_extent ( fieinfo, logical, phys, size, - flags); + size, flags);if (ret || blkoff > end_blkoff)break ;
fs/ocfs2/extent_map.c +2 -2
@@ -736,7 +736,7 @@ static int ocfs2_fiemap_inline(struct inode *inode, struct buffer_head *di_bh, id2.i_data.id_data); ret =fiemap_fill_next_extent (fieinfo,0 , phys, id_count, - flags); + id_count, flags);if (ret <0 )return ret; }
@@ -809,7 +809,7 @@ int ocfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, virt_bytes = (u64 )le32_to_cpu (rec.e_cpos) << osb->s_clustersize_bits; ret =fiemap_fill_next_extent (fieinfo, virt_bytes, phys_bytes, - len_bytes, fe_flags); + len_bytes, len_bytes, fe_flags);if (ret)break ;
fs/xfs/xfs_iops.c +1 -1
@@ -1030,7 +1030,7 @@ xfs_fiemap_format( fiemap_flags |=FIEMAP_EXTENT_LAST ; error =fiemap_fill_next_extent (fieinfo, logical, physical, - length, fiemap_flags); + length, length, fiemap_flags);if (error >0 ) { error =0 ; *full =1 ;/* user array now full */
include/linux/fs.h +1 -1
@@ -1412,7 +1412,7 @@ struct fiemap_extent_info {fiemap_extent array */ };int fiemap_fill_next_extent (struct fiemap_extent_info *info,u64 logical, -u64 phys,u64 len,u32 flags); +u64 phys,u64 len,u64 phys_len,u32 flags);int fiemap_check_flags (struct fiemap_extent_info *fieinfo,u32 fs_flags);/*
include/uapi/linux/fiemap.h +7 -1
@@ -19,7 +19,9 @@ struct fiemap_extent {__u64 fe_physical;/* physical offset in bytes for the start * of the extent from the beginning of the disk */ __u64 fe_length;/* length in bytes for this extent */ -__u64 fe_reserved64[2 ]; +__u64 fe_phys_length;/* physical length in bytes, may be different from +* fe_length and sets additional extent flags */ +__u64 fe_reserved64;__u32 fe_flags;/* FIEMAP_EXTENT_* flags for this extent */ __u32 fe_reserved[3 ]; };
@@ -50,6 +52,10 @@ struct fiemap {* Sets EXTENT_UNKNOWN. */ #define FIEMAP_EXTENT_ENCODED 0x00000008/* Data can not be read * while fs is unmounted */ +#define FIEMAP_EXTENT_DATA_COMPRESSED 0x00000040/* Data is compressed by fs. +* Sets EXTENT_ENCODED and +* the compressed size is +* stored in fe_phys_length */ #define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080/* Data is encrypted by fs. * Sets EXTENT_ENCODED */ #define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100/* Extent offsets may not be
Signature · 6 lines
1.8.4.5 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH v4 3/5 btrfs: set FIEMAP_EXTENT_DATA_COMPRESSED for compressed extents
In reply to David SterbaSet the EXTENT_DATA_COMPRESSED flag together with EXTENT_ENCODED as defined by fiemap spec.
Signed-off-by David Sterba <[email protected]>
--- fs/btrfs/extent_io.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
fs/btrfs/extent_io.c +6 -2
@@ -4272,6 +4272,7 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,while (!end) {u64 offset_in_extent =0 ; +u64 em_phys_len;/* break if the extent we found is outside the range */ if (em->start >= max ||extent_map_end (em) < off)
@@ -4295,6 +4296,7 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, offset_in_extent = em_start - em->start; em_end =extent_map_end (em); em_len = em_end - em_start; + em_phys_len = em->len; disko =0 ; flags =0 ;
@@ -4333,9 +4335,11 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,if (ref_cnt >1 ) flags |=FIEMAP_EXTENT_SHARED ; + em_phys_len = em->block_len; }if (test_bit (EXTENT_FLAG_COMPRESSED , &em->flags)) - flags |=FIEMAP_EXTENT_ENCODED ; + flags |=FIEMAP_EXTENT_ENCODED | +FIEMAP_EXTENT_DATA_COMPRESSED ;free_extent_map (em); em =NULL ;
@@ -4357,7 +4361,7 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, end =1 ; } ret =fiemap_fill_next_extent (fieinfo, em_start, disko, - em_len, em_len, flags); + em_len, em_phys_len, flags);if (ret) goto out_free; }
Signature · 1 line
1.8.4.5
PATCH v4 4/5 Documentation/fiemap: Document the DATA_COMPRESSED flag
In reply to David SterbaCC Rob Landley <[email protected]>
Signed-off-by David Sterba <[email protected]>
--- Documentation/filesystems/fiemap.txt | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
Documentation/filesystems/fiemap.txt +13 -2
@@ -83,7 +83,9 @@ struct fiemap_extent { __u64 fe_physical; /* physical offset in bytes for the start * of the extent */ __u64 fe_length; /* length in bytes for the extent */ - __u64 fe_reserved64[2]; + __u64 fe_phys_length; /* physical length in bytes, may be different from + * fe_length and sets additional extent flags */ + __u64 fe_reserved64; __u32 fe_flags; /* FIEMAP_EXTENT_* flags for this extent */ __u32 fe_reserved[3]; };
@@ -93,7 +95,10 @@ for an extents logical offset to start before the request or its logical length to extend past the request. Unless FIEMAP_EXTENT_NOT_ALIGNED is returned, fe_logical, fe_physical, and fe_length will be aligned to the block size of the file system. With the exception of extents flagged as -FIEMAP_EXTENT_MERGED, adjacent extents will not be merged. +FIEMAP_EXTENT_MERGED, adjacent extents will not be merged. The value of +fe_phys_length is always set. It may be different from fe_length and sets +additional extent flags that indicate the reason, eg. for compression it's +FIEMAP_EXTENT_DATA_COMPRESSED. The fe_flags field contains flags which describe the extent returned. A special flag, FIEMAP_EXTENT_LAST is always set on the last extent in
@@ -143,6 +148,12 @@ unmounted, and then only if the FIEMAP_EXTENT_ENCODED flag is clear; user applications must not try reading or writing to the filesystem via the block device under any other circumstances. +* FIEMAP_EXTENT_DATA_COMPRESSED + - This will also set FIEMAP_EXTENT_ENCODED +The data in this extent has been compressed by the file system. When +set, fe_phys_length contains the physical extent length in bytes. The +value may be rounded depending on the filesystem implementation. + * FIEMAP_EXTENT_DATA_ENCRYPTED - This will also set FIEMAP_EXTENT_ENCODED The data in this extent has been encrypted by the file system.
Signature · 1 line
1.8.4.5
PATCH v4 5/5 fiemap: rename fe_length to fe_logi_length
In reply to David SterbaPick a more descriptive name for the logical length.
Signed-off-by David Sterba <[email protected]>
--- Documentation/filesystems/fiemap.txt | 10 +++++----- fs/ioctl.c | 2 +- include/uapi/linux/fiemap.h | 3 ++- 3 files changed, 8 insertions(+), 7 deletions(-)
Documentation/filesystems/fiemap.txt +5 -5
@@ -82,7 +82,7 @@ struct fiemap_extent { * the extent */ __u64 fe_physical; /* physical offset in bytes for the start * of the extent */ - __u64 fe_length; /* length in bytes for the extent */ + __u64 fe_logi_length; /* logical length in bytes for the extent */ __u64 fe_phys_length; /* physical length in bytes, may be different from * fe_length and sets additional extent flags */ __u64 fe_reserved64;
@@ -93,12 +93,12 @@ struct fiemap_extent { All offsets and lengths are in bytes and mirror those on disk. It is valid for an extents logical offset to start before the request or its logical length to extend past the request. Unless FIEMAP_EXTENT_NOT_ALIGNED is -returned, fe_logical, fe_physical, and fe_length will be aligned to the +returned, fe_logical, fe_physical, and fe_logi_length will be aligned to the block size of the file system. With the exception of extents flagged as FIEMAP_EXTENT_MERGED, adjacent extents will not be merged. The value of -fe_phys_length is always set. It may be different from fe_length and sets -additional extent flags that indicate the reason, eg. for compression it's -FIEMAP_EXTENT_DATA_COMPRESSED. +fe_phys_length is always set. It may be different from fe_logi_length and +sets additional extent flags that indicate the reason, eg. for compression +it's FIEMAP_EXTENT_DATA_COMPRESSED. The fe_flags field contains flags which describe the extent returned. A special flag, FIEMAP_EXTENT_LAST is always set on the last extent in
fs/ioctl.c +1 -1
@@ -113,7 +113,7 @@ int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical,memset (&extent,0 ,sizeof (extent)); extent.fe_logical = logical; extent.fe_physical = phys; - extent.fe_length = len; + extent.fe_logi_length = len; extent.fe_flags = flags; WARN_ONCE((flags & FIEMAP_EXTENT_DATA_COMPRESSED)
include/uapi/linux/fiemap.h +2 -1
@@ -18,7 +18,8 @@ struct fiemap_extent {* the extent from the beginning of the file */ __u64 fe_physical;/* physical offset in bytes for the start * of the extent from the beginning of the disk */ -__u64 fe_length;/* length in bytes for this extent */ +#define fe_length fe_logi_length +__u64 fe_logi_length;/* logical length in bytes for this extent */ __u64 fe_phys_length;/* physical length in bytes, may be different from * fe_length and sets additional extent flags */ __u64 fe_reserved64;
Signature · 1 line
1.8.4.5