ext4_fiemap gives 0 extents for files smaller than a block (patch included)
Fiemap (ioctl) does not return any extents for small files on ext4. (fm_start=0, fm_length=filesize)
File affected: fs/ext4/extents.c
I found the reason of the bug: wrong rounding. It will not only affect small files, but any request that overlaps an extent boundary by less that blocksize.
The attached patch is against 2.6.32-rc5.
Leonard Michlmayr
fs/ext4/extents.c +3 -1
@@ -3685,6 +3685,7 @@ __u64 start, __u64 len) {ext4_lblk_t start_blk; +ext4_lblk_t end_blk;ext4_lblk_t len_blks;int error =0 ;
@@ -3700,7 +3701,8 @@ error =ext4_xattr_fiemap (inode, fieinfo); }else { start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + end_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = end_blk - start_blk +1 ;/* * Walk the extent tree gathering extent information.
Quoted 11 lines · Leonard Michlmayr
Fiemap (ioctl) does not return any extents for small files on ext4. (fm_start=0, fm_length=filesize)
File affected:
fs/ext4/extents.cI found the reason of the bug: wrong rounding. It will not only affect small files, but any request that overlaps an extent boundary by less that blocksize.
fs/ext4/extents.c line 3703
Quoted 4 lines
fs/ext4/extents.c +2
-1@@ -3700,7 +3701,8 @@ start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + end_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = end_blk - start_blk +1 ;
I don't think this is quite correct either. For example, if blocksize is 1024 and start is 1023 (start_blk = 0) and len is 2 (end = 1024, end_blk = 1) then len_blks = 2 which is too much.
I think the right calculation here is:
end_blk = (start + len + inode->i_sb->s_blocksize -
1) >>
inode->i_sb->s_blocksize_bits;
len_blks = end_blk - start_blk;I'm also wondering (unrelated to this bug) why inode->i_sb->s_blocksize_bits is used instead of inode->i_blkbits? That is probably worth a separate cleanup patch.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Thank you for your reply.
Quoted 12 lines
fs/ext4/extents.c line 3703
Quoted 6 lines
fs/ext4/extents.c +2
-1@@ -3700,7 +3701,8 @@ start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + end_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = end_blk - start_blk +1 ;I don't think this is quite correct either. For example, if blocksize is 1024
and start is 1023 (start_blk= 0) and len is 2 (end = 1024,end_blk=
1) thenlen_blks= 2 which is too much.
I think that len_blks = 2 is the correct value, because the requested region extends into 2 blocks (namely 0 and 1). If both blocks are in two separate extents, then ext4_ext_walk_space should report 2 extents. (If it's the same extent, only 1 will be reported anyways)
Quoted 6 lines
I think the right calculation here is:
end_blk = (start + len + inode->i_sb->s_blocksize - 1) >> inode->i_sb->s_blocksize_bits; len_blks = end_blk - start_blk;
This is exactly the same (provided that len > 0). You can convince yourself easily that ((blocksize + x) >> blocksize_bits == x >> blocksize_bits + 1) for any positive x, because the lower bits of blocksize are all 0. (Your calculation would handle the case len == 0 right, if that was allowed.)
Regards
Leonard
Quoted 25 lines · Leonard Michlmayr
Thank you for your reply.
Quoted 10 lines
fs/ext4/extents.c line 3703
Quoted 5 lines
fs/ext4/extents.c +2
-1@@ -3700,7 +3701,8 @@ start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + end_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = end_blk - start_blk +1 ;I don't think this is quite correct either. For example, if blocksize is 1024
and start is 1023 (start_blk= 0) and len is 2 (end = 1024,end_blk=
1) thenlen_blks= 2 which is too much.I think that
len_blks= 2 is the correct value, because the requested region extends into 2 blocks (namely 0 and 1). If both blocks are in two separate extents, thenext4_ext_walk_spaceshould report 2 extents. (If it's the same extent, only 1 will be reported anyways)Quoted 6 lines
I think the right calculation here is:
end_blk = (start + len + inode->i_sb->s_blocksize - 1) >> inode->i_sb->s_blocksize_bits; len_blks = end_blk - start_blk;
Quoted 8 lines
This is exactly the same (provided that len > 0). You can convince yourself easily that ((blocksize + x) >>
blocksize_bits== x >>blocksize_bits+ 1) for any positive x, because the lower bits of blocksize are all 0. (Your calculation would handle the case len == 0 right, if that was allowed.)Regards
Leonard
I was wondering if there is any update on the status of this patch. Thanks !
Warm Regards,
Surbhi.
PATCH 1/1 ext4: number of blocks for fiemap
Problem:
fs/ext4/extents.c:ext4_fiemap rounds the length of the requested range down to blocksize. This is not the true number of blocks that cover the requested region. This problem is especially impressive if the user requests only the first byte of a file: not a single extent will be reported.
Solution:
Calculate the last byte of the region and round to blocksize. Then get the number of blocks by subtracting last_blk - start_blk and adding 1 for the first block. (The variable last_blk is introduced just for easier reading.) This patch will fix this.
I already suggested this patch some time ago, this is the same patch for a more recent kernel version.
fs/ext4/extents.c +4 -1
@@ -3711,6 +3711,7 @@ int ext4_fiemap(struct inode *inode, str __u64 start, __u64 len) {ext4_lblk_t start_blk; +ext4_lblk_t last_blk;ext4_lblk_t len_blks;int error =0 ;
@@ -3726,7 +3727,9 @@ int ext4_fiemap(struct inode *inode, str error =ext4_xattr_fiemap (inode, fieinfo); }else { start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; +/* the last byte in the range is (start + len - 1) */ + last_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = last_blk - start_blk +1 ;/* * Walk the extent tree gathering extent information.
Re: [PATCH 2.6.32.7] ext4: number of blocks for fiemap
Quoted 31 lines · Leonard Michlmayr
fs/ext4/extents.c:ext4_fiemap rounds the length of the requested range down to blocksize. This is not the true number of blocks that cover the requested region. This problem is especially impressive if the user requests only the first byte of a file: not a single extent will be reported.
Solution:
Calculate the last byte of the region and round to blocksize. Then get the number of blocks by subtractinglast_blk-start_blkand adding 1 for the first block. (The variablelast_blkis introduced just for easier reading.) This patch will fix this.I already suggested this patch some time ago, this is the same patch for a more recent kernel version.
Signed-off-by Leonard Michlmayr <[email protected]>diff -rup
linux-2.6.32.7/fs/ext4/extents.clinux-2.6.32.7-lm/fs/ext4/extents.c
---linux-2.6.32.7/fs/ext4/extents.c2010-01-29 00:06:20.000000000
+0100
+++linux-2.6.32.7-lm/fs/ext4/extents.c2010-02-11
21:26:40.000000000 +0100fs/ext4/extents.c +1
-0@@ -3711,6 +3711,7 @@ int ext4_fiemap(struct inode *inode, str __u64 start, __u64 len) {ext4_lblk_t start_blk; +ext4_lblk_t last_blk;ext4_lblk_t len_blks;int error =0 ;
fs/ext4/extents.c int ext4_fiemap(struct inode *inode, str · line 3732
Quoted 7 lines
fs/ext4/extents.c +3
-1@@ -3726,7 +3727,9 @@ int ext4_fiemap(struct inode *inode, str error =ext4_xattr_fiemap (inode, fieinfo); }else { start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; +/* the last byte in the range is (start + len - 1) */ + last_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = last_blk - start_blk +1 ;
If "last_blk" is only used in this one place, please put the declaration inside the scope of the "else" clause. Looks fine otherwise.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
PATCH 1/1 ext4: number of blocks for fiemap
Quoted 3 lines · On 2010-02-12, at 14:49 -0700 Andreas Dilger wrote:
If "
last_blk" is only used in this one place, please put the declaration inside the scope of the "else" clause. Looks fine otherwise.
Done. Thanks.
---
Problem:
fs/ext4/extents.c:ext4_fiemap rounds the length of the requested range down to blocksize. This is not the true number of blocks that cover the requested region. This problem is especially impressive if the user requests only the first byte of a file: not a single extent will be reported.
Solution:
Calculate the last byte of the region and round to blocksize. Then get the number of blocks by subtracting last_blk - start_blk and adding 1 for the first block. (The variable last_blk is introduced just for easier reading.) This patch will fix this.
fs/ext4/extents.c +4 -1
@@ -3725,8 +3725,11 @@ int ext4_fiemap(struct inode *inode, strif (fieinfo->fi_flags &FIEMAP_FLAG_XATTR ) { error =ext4_xattr_fiemap (inode, fieinfo); }else { +ext4_lblk_t last_blk; start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; +/* the last byte in the range is (start + len - 1) */ + last_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = last_blk - start_blk +1 ;/* * Walk the extent tree gathering extent information.
[email protected]
maintainerRe: [PATCH 2.6.32.7] ext4: number of blocks for fiemap
Sorry for the delay in attending to this patch. This is what I've added to the ext4 patch queue.
- Ted
commit 7f9aeb212e41ea20f61061d51ad11ee2449853f1
Author: Theodore Ts'o <[email protected]> Date: Mon Feb 15 14:48:32 2010 -0500
ext4: correctly calculate number of blocks for fiemap
ext4_fiemap() rounds the length of the requested range down to blocksize, which is is not the true number of blocks that cover the requested region. This problem is especially impressive if the user requests only the first byte of a file: not a single extent will be reported.
We fix this by calculating the last block of the region and then subtract to find the number of blocks in the extents.
fs/ext4/extents.c +4 -2
@@ -3768,7 +3768,6 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, __u64 start, __u64 len) {ext4_lblk_t start_blk; -ext4_lblk_t len_blks;int error =0 ;/* fallback to generic here if not in extents fmt */
@@ -3782,8 +3781,11 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,if (fieinfo->fi_flags &FIEMAP_FLAG_XATTR ) { error =ext4_xattr_fiemap (inode, fieinfo); }else { +ext4_lblk_t last_blk, len_blks; + start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + last_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = last_blk - start_blk +1 ;/* * Walk the extent tree gathering extent information.
Re: [PATCH 2.6.32.7] ext4: number of blocks for fiemap
Quoted 8 lines · [email protected] wrote:
Sorry for the delay in attending to this patch. This is what I've added to the ext4 patch queue.
- Ted
commit
7f9aeb212e41ea20f61061d51ad11ee2449853f1Author: Theodore Ts'o <[email protected]> Date: Mon Feb 15 14:48:32 2010 -0500
Ted, it looks like perhaps you accidentally reassigned authorship for this patch?
-Eric
Quoted 43 lines
ext4: correctly calculate number of blocks for fiemap
ext4_fiemap()rounds the length of the requested range down to blocksize, which is is not the true number of blocks that cover the requested region. This problem is especially impressive if the user requests only the first byte of a file: not a single extent will be reported.We fix this by calculating the last block of the region and then subtract to find the number of blocks in the extents.
Signed-off-by Leonard Michlmayr <[email protected]>Signed-off-by "Theodore Ts'o" <[email protected]>fs/ext4/extents.c +4
-3@@ -3768,7 +3768,6 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, __u64 start, __u64 len) {ext4_lblk_t start_blk; -ext4_lblk_t len_blks;int error =0 ;/* fallback to generic here if not in extents fmt */ @@ -3782,8 +3781,11 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,if (fieinfo->fi_flags &FIEMAP_FLAG_XATTR ) { error =ext4_xattr_fiemap (inode, fieinfo); }else { +ext4_lblk_t last_blk, len_blks; + start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + last_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = last_blk - start_blk +1 ;/* * Walk the extent tree gathering extent information. -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[email protected]
maintainerRe: [PATCH 2.6.32.7] ext4: number of blocks for fiemap
Quoted 4 lines · Eric Sandeen
Quoted 3 lines · [email protected] wrote:
Sorry for the delay in attending to this patch. This is what I've added to the ext4 patch queue.
Quoted 8 lines
Quoted 5 lines
- Ted
commit
7f9aeb212e41ea20f61061d51ad11ee2449853f1Author: Theodore Ts'o <[email protected]> Date: Mon Feb 15 14:48:32 2010 -0500Ted, it looks like perhaps you accidentally reassigned authorship for this patch?
Oops, I forgot the fix the ownership when I applied the patch. Fixed.
- Ted
Do you have any news on this bug?
Will you apply the patch? Do you not plan to apply the patch? Do you need an update of the patch?
Regards
Leonard
Am Mittwoch, den 04.11.2009, 12:44 -0700 schrieb Andreas Dilger:
Quoted 35 lines · Leonard Michlmayr
Quoted 11 lines
Fiemap (ioctl) does not return any extents for small files on ext4. (fm_start=0, fm_length=filesize)
File affected:
fs/ext4/extents.cI found the reason of the bug: wrong rounding. It will not only affect small files, but any request that overlaps an extent boundary by less that blocksize.
fs/ext4/extents.c line 3703
Quoted 4 lines
fs/ext4/extents.c +2
-1@@ -3700,7 +3701,8 @@ start_blk = start >> inode->i_sb->s_blocksize_bits; - len_blks = len >> inode->i_sb->s_blocksize_bits; + end_blk = (start + len -1 ) >> inode->i_sb->s_blocksize_bits; + len_blks = end_blk - start_blk +1 ;I don't think this is quite correct either. For example, if blocksize is 1024
and start is 1023 (start_blk= 0) and len is 2 (end = 1024,end_blk=
1) thenlen_blks= 2 which is too much.I think the right calculation here is:
end_blk = (start + len + inode->i_sb->s_blocksize - 1) >> inode->i_sb->s_blocksize_bits; len_blks = end_blk - start_blk;I'm also wondering (unrelated to this bug) why inode->i_sb-
Quoted 1 line
s_blocksize_bitsis used instead of
inode->i_blkbits? That is probably worth a separate cleanup
patch.
...