ext4: Fix credit estimates for extent tree modifications
Jan Kara
reviewercover ext4: Fix credit estimates for extent tree modifications
Hello!
This patch set fixes credit estimates for extent tree modification during convertion of unwritten extents on writeback completion. Otherwise we are occasionally seeing assertion failures during extent conversion in end io completion callback due to insufficient transation credits.
Honza
Jan Kara
reviewerPATCH 1/3 ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
So far ext4_meta_trans_blocks() expects that each extent counted in @pextents will be allocated in the transaction we estimate credits for. This is correct for the use in ext4_chunk_trans_blocks() and ext4_chunk_trans_extent() however the use in atomic write path (ext4_convert_unwritten_extents_atomic() and ext4_iomap_alloc() for IOMAP_ATOMIC) unnecessarily overestimates the number of necessary credits as neither of them allocates any data. Add argument to ext4_meta_trans_blocks() for number of extents that are going to be allocated in the transaction.
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents.c | 2 +- fs/ext4/inode.c | 32 +++++++++++++++++--------------- 3 files changed, 19 insertions(+), 17 deletions(-)
fs/ext4/ext4.h +1 -1
@@ -3184,7 +3184,7 @@ extern int ext4_normal_submit_inode_data_buffers(struct jbd2_inode *jinode);extern int ext4_chunk_trans_blocks (struct inode *,int nrblocks);extern int ext4_chunk_trans_extent (struct inode *inode,int nrblocks);extern int ext4_meta_trans_blocks (struct inode *inode,int lblocks, -int pextents); +int pextents,int alloc_extents);extern int ext4_block_zero_eof (struct inode *inode,loff_t from,loff_t end);extern int ext4_zero_partial_blocks (struct inode *inode,loff_t lstart,loff_t length,bool *did_zero);
fs/ext4/extents.c +1 -1
@@ -4976,7 +4976,7 @@ int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode,* it can tell if the extent in the cache is a split extent. * But for now let's assume pextents as 2 always. */ - credits =ext4_meta_trans_blocks (inode, max_blocks,2 ); + credits =ext4_meta_trans_blocks (inode, max_blocks,2 ,0 ); }if (credits) {
fs/ext4/inode.c +17 -15
@@ -3693,9 +3693,11 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,if (ret <0 )return ret;if (map->m_len < orig_mlen) { +int bpg =EXT4_BLOCKS_PER_GROUP (inode->i_sb); + map->m_len = orig_mlen; - dio_credits =ext4_meta_trans_blocks (inode, orig_mlen, - map->m_len); + dio_credits =ext4_meta_trans_blocks (inode, map->m_len, + map->m_len,0 ); }else { dio_credits =ext4_chunk_trans_blocks (inode, map->m_len);
@@ -6307,17 +6309,17 @@ static int ext4_index_trans_blocks(struct inode *inode, int lblocks, }/* -* Account for index blocks, block groups bitmaps and block group -* descriptor blocks if modify datablocks and index blocks -* worse case, the indexs blocks spread over different block groups -* -* If datablocks are discontiguous, they are possible to spread over -* different block groups too. If they are contiguous, with flexbg, -* they could still across block group boundary. -* -* Also account for superblock, inode, quota and xattr blocks +* Calculate number of credits needed in a transaction to: +* * Allocate data blocks from @alloc_extents different groups - note that +* with flexbg a single physical extent can span multiple groups but +* single mballoc request only returns extent within one group. +* * Allocate metatadata (extent tree blocks, indirect blocks) to store +* pointers to @pextents data extents having @lblocks in total. +* * Modify extent tree / indirect block tree, inode, superblock, quota +* tracking, xattr blocks */ -int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents) +int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents, +int alloc_extents) {ext4_group_t groups, ngroups =ext4_get_groups_count (inode->i_sb);int gdpblocks;
@@ -6334,7 +6336,7 @@ int ext4_meta_trans_blocks(struct inode *inode, int lblocks, int pextents)* Now let's see how many group bitmaps and group descriptors need * to account */ - groups = idxblocks + pextents; + groups = idxblocks + alloc_extents; gdpblocks = groups;if (groups > ngroups) groups = ngroups;
@@ -6360,7 +6362,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks) {int ret; - ret =ext4_meta_trans_blocks (inode, nrblocks,1 ); + ret =ext4_meta_trans_blocks (inode, nrblocks,1 ,1 );/* Account for data blocks for journalled mode */ if (ext4_should_journal_data (inode)) ret += nrblocks;
@@ -6378,7 +6380,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks)*/ int ext4_chunk_trans_blocks (struct inode *inode,int nrblocks) { -return ext4_meta_trans_blocks (inode, nrblocks,1 ); +return ext4_meta_trans_blocks (inode, nrblocks,1 ,1 ); }/*
Signature · 1 line
2.51.0
Re: [PATCH 1/3] ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
Quoted 11 lines · Jan Kara
So far
ext4_meta_trans_blocks()expects that each extent counted in @pextents will be allocated in the transaction we estimate credits for. This is correct for the use inext4_chunk_trans_blocks()andext4_chunk_trans_extent()however the use in atomic write path (ext4_convert_unwritten_extents_atomic()andext4_iomap_alloc()forIOMAP_ATOMIC) unnecessarily overestimates the number of necessary credits as neither of them allocates any data. Add argument toext4_meta_trans_blocks()for number of extents that are going to be allocated in the transaction.Signed-off-by Jan Kara <[email protected]>
Hi Jan,
Looks mostly okay, feel free to add:
just a couple comments below:
Quoted 24 lines
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents.c | 2 +- fs/ext4/inode.c | 32 +++++++++++++++++--------------- 3 files changed, 19 insertions(+), 17 deletions(-)fs/ext4/ext4.h +1
-1@@ -3184,7 +3184,7 @@ extern int ext4_normal_submit_inode_data_buffers(struct jbd2_inode *jinode);extern int ext4_chunk_trans_blocks (struct inode *,int nrblocks);extern int ext4_chunk_trans_extent (struct inode *inode,int nrblocks);extern int ext4_meta_trans_blocks (struct inode *inode,int lblocks, -int pextents); +int pextents,int alloc_extents);extern int ext4_block_zero_eof (struct inode *inode,loff_t from,loff_t end);extern int ext4_zero_partial_blocks (struct inode *inode,loff_t lstart,loff_t length,bool *did_zero);fs/ext4/extents.c +0
-0
fs/ext4/extents.c int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode, · line 4979
Quoted 5 lines
fs/ext4/extents.c +1
-1@@ -4976,7 +4976,7 @@ int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode,* it can tell if the extent in the cache is a split extent. * But for now let's assume pextents as 2 always. */ - credits =ext4_meta_trans_blocks (inode, max_blocks,2 ); + credits =ext4_meta_trans_blocks (inode, max_blocks,2 ,0 );
Right, because we are not going to allocate anything but we might be modifying upto 2 extent entries. By that logic shall we also change the following:
@@ -5046,7 +5046,7 @@ int ext4_convert_unwritten_extents(handle_t *handle, struct inode *inode,
/*
* credits to insert 1 extent into extent tree
*/
- credits = ext4_chunk_trans_blocks(inode, max_blocks);
+ credits = ext4_meta_trans_blocks(inode, max_blocks, max_blocks, 0);
}
while (ret >= 0 && ret < max_blocks) {
map.m_lblk += ret;since we are not going to allocate anything.
Quoted 8 lines
fs/ext4/extents.c +0
-0}if (credits) {fs/ext4/inode.c +0
-0
fs/ext4/inode.c static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map, · line 3700
Quoted 10 lines
fs/ext4/inode.c +4
-2@@ -3693,9 +3693,11 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,if (ret <0 )return ret;if (map->m_len < orig_mlen) { +int bpg =EXT4_BLOCKS_PER_GROUP (inode->i_sb); + map->m_len = orig_mlen; - dio_credits =ext4_meta_trans_blocks (inode, orig_mlen, - map->m_len); + dio_credits =ext4_meta_trans_blocks (inode, map->m_len, + map->m_len,0 );
This makes sense because we always come here in the bigalloc case when the cluster is already allocated. If we ever support atomic writes without bigalloc, we'll have to remember to take care of this. Maybe we can add a small one line comment here?
/*
* we have an encountered mixed mapping in an already allocated bigalloc
* cluster hence credits for allocation are not needed
*/
Quoted 61 lines
fs/ext4/inode.c +13
-13}else { dio_credits =ext4_chunk_trans_blocks (inode, map->m_len);@@ -6307,17 +6309,17 @@ static int ext4_index_trans_blocks(struct inode *inode, int lblocks, }/* -* Account for index blocks, block groups bitmaps and block group -* descriptor blocks if modify datablocks and index blocks -* worse case, the indexs blocks spread over different block groups -* -* If datablocks are discontiguous, they are possible to spread over -* different block groups too. If they are contiguous, with flexbg, -* they could still across block group boundary. -* -* Also account for superblock, inode, quota and xattr blocks +* Calculate number of credits needed in a transaction to: +* * Allocate data blocks from @alloc_extents different groups - note that +* with flexbg a single physical extent can span multiple groups but +* single mballoc request only returns extent within one group. +* * Allocate metatadata (extent tree blocks, indirect blocks) to store +* pointers to @pextents data extents having @lblocks in total. +* * Modify extent tree / indirect block tree, inode, superblock, quota +* tracking, xattr blocks */ -int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents) +int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents, +int alloc_extents) {ext4_group_t groups, ngroups =ext4_get_groups_count (inode->i_sb);int gdpblocks;@@ -6334,7 +6336,7 @@ int ext4_meta_trans_blocks(struct inode *inode, int lblocks, int pextents)* Now let's see how many group bitmaps and group descriptors need * to account */ - groups = idxblocks + pextents; + groups = idxblocks + alloc_extents; gdpblocks = groups;if (groups > ngroups) groups = ngroups;@@ -6360,7 +6362,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks) {int ret; - ret =ext4_meta_trans_blocks (inode, nrblocks,1 ); + ret =ext4_meta_trans_blocks (inode, nrblocks,1 ,1 );/* Account for data blocks for journalled mode */ if (ext4_should_journal_data (inode)) ret += nrblocks;@@ -6378,7 +6380,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks)*/ int ext4_chunk_trans_blocks (struct inode *inode,int nrblocks) { -return ext4_meta_trans_blocks (inode, nrblocks,1 ); +return ext4_meta_trans_blocks (inode, nrblocks,1 ,1 ); }/* Signature · 1 line
2.51.0
Re: [PATCH 1/3] ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
In reply to Jan KaraQuoted 11 lines · Jan Kara
So far
ext4_meta_trans_blocks()expects that each extent counted in @pextents will be allocated in the transaction we estimate credits for. This is correct for the use inext4_chunk_trans_blocks()andext4_chunk_trans_extent()however the use in atomic write path (ext4_convert_unwritten_extents_atomic()andext4_iomap_alloc()forIOMAP_ATOMIC) unnecessarily overestimates the number of necessary credits as neither of them allocates any data. Add argument toext4_meta_trans_blocks()for number of extents that are going to be allocated in the transaction.Signed-off-by Jan Kara <[email protected]>
Hi Jan,
I have some questions below.
Quoted 37 lines
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents.c | 2 +- fs/ext4/inode.c | 32 +++++++++++++++++--------------- 3 files changed, 19 insertions(+), 17 deletions(-)fs/ext4/ext4.h +1
-1@@ -3184,7 +3184,7 @@ extern int ext4_normal_submit_inode_data_buffers(struct jbd2_inode *jinode);extern int ext4_chunk_trans_blocks (struct inode *,int nrblocks);extern int ext4_chunk_trans_extent (struct inode *inode,int nrblocks);extern int ext4_meta_trans_blocks (struct inode *inode,int lblocks, -int pextents); +int pextents,int alloc_extents);extern int ext4_block_zero_eof (struct inode *inode,loff_t from,loff_t end);extern int ext4_zero_partial_blocks (struct inode *inode,loff_t lstart,loff_t length,bool *did_zero);fs/ext4/extents.c +1
-1@@ -4976,7 +4976,7 @@ int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode,* it can tell if the extent in the cache is a split extent. * But for now let's assume pextents as 2 always. */ - credits =ext4_meta_trans_blocks (inode, max_blocks,2 ); + credits =ext4_meta_trans_blocks (inode, max_blocks,2 ,0 ); }if (credits) {fs/ext4/inode.c +0
-0
fs/ext4/inode.c static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map, · line 3700
Quoted 10 lines
fs/ext4/inode.c +4
-2@@ -3693,9 +3693,11 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,if (ret <0 )return ret;if (map->m_len < orig_mlen) { +int bpg =EXT4_BLOCKS_PER_GROUP (inode->i_sb); + map->m_len = orig_mlen; - dio_credits =ext4_meta_trans_blocks (inode, orig_mlen, - map->m_len); + dio_credits =ext4_meta_trans_blocks (inode, map->m_len, + map->m_len,0 );
This pertains to the mixed map case, where holes may exist in between. My understanding is that we cannot assume all blocks are already allocated. Is this right?
Quoted 18 lines
fs/ext4/inode.c +2
-9}else { dio_credits =ext4_chunk_trans_blocks (inode, map->m_len);@@ -6307,17 +6309,17 @@ static int ext4_index_trans_blocks(struct inode *inode, int lblocks, }/* -* Account for index blocks, block groups bitmaps and block group -* descriptor blocks if modify datablocks and index blocks -* worse case, the indexs blocks spread over different block groups -* -* If datablocks are discontiguous, they are possible to spread over -* different block groups too. If they are contiguous, with flexbg, -* they could still across block group boundary. -* -* Also account for superblock, inode, quota and xattr blocks +* Calculate number of credits needed in a transaction to: +* * Allocate data blocks from @alloc_extents different groups - note that
fs/ext4/inode.c static int ext4_index_trans_blocks(struct inode *inode, int lblocks, · line 6322
Quoted 10 lines
fs/ext4/inode.c +8
-1+* with flexbg a single physical extent can span multiple groups but +* single mballoc request only returns extent within one group. +* * Allocate metatadata (extent tree blocks, indirect blocks) to store +* pointers to @pextents data extents having @lblocks in total. +* * Modify extent tree / indirect block tree, inode, superblock, quota +* tracking, xattr blocks */ -int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents) +int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents, +int alloc_extents)
From what I can see, alloc_extents currently can only be 0 or equal to pextents — 0 means a pure conversion, and equal to pextents means a new allocation. pextents is supposed to cover the total count of alloc_extents. Should we add some sanity checks to guard against invalid inputs, like alloc_extents > pextents?
Thanks,
Yi.
Quoted 30 lines
fs/ext4/inode.c +3
-3{ext4_group_t groups, ngroups =ext4_get_groups_count (inode->i_sb);int gdpblocks;@@ -6334,7 +6336,7 @@ int ext4_meta_trans_blocks(struct inode *inode, int lblocks, int pextents)* Now let's see how many group bitmaps and group descriptors need * to account */ - groups = idxblocks + pextents; + groups = idxblocks + alloc_extents; gdpblocks = groups;if (groups > ngroups) groups = ngroups;@@ -6360,7 +6362,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks) {int ret; - ret =ext4_meta_trans_blocks (inode, nrblocks,1 ); + ret =ext4_meta_trans_blocks (inode, nrblocks,1 ,1 );/* Account for data blocks for journalled mode */ if (ext4_should_journal_data (inode)) ret += nrblocks;@@ -6378,7 +6380,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks)*/ int ext4_chunk_trans_blocks (struct inode *inode,int nrblocks) { -return ext4_meta_trans_blocks (inode, nrblocks,1 ); +return ext4_meta_trans_blocks (inode, nrblocks,1 ,1 ); }/*
Ojaswin Mujoo
reviewerRe: [PATCH 1/3] ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
Quoted 60 lines · Zhang Yi
Quoted 11 lines · Jan Kara
So far
ext4_meta_trans_blocks()expects that each extent counted in @pextents will be allocated in the transaction we estimate credits for. This is correct for the use inext4_chunk_trans_blocks()andext4_chunk_trans_extent()however the use in atomic write path (ext4_convert_unwritten_extents_atomic()andext4_iomap_alloc()forIOMAP_ATOMIC) unnecessarily overestimates the number of necessary credits as neither of them allocates any data. Add argument toext4_meta_trans_blocks()for number of extents that are going to be allocated in the transaction.Signed-off-by Jan Kara <[email protected]>Hi Jan,
I have some questions below.
Quoted 43 lines
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents.c | 2 +- fs/ext4/inode.c | 32 +++++++++++++++++--------------- 3 files changed, 19 insertions(+), 17 deletions(-)fs/ext4/ext4.h +1
-1@@ -3184,7 +3184,7 @@ extern int ext4_normal_submit_inode_data_buffers(struct jbd2_inode *jinode);extern int ext4_chunk_trans_blocks (struct inode *,int nrblocks);extern int ext4_chunk_trans_extent (struct inode *inode,int nrblocks);extern int ext4_meta_trans_blocks (struct inode *inode,int lblocks, -int pextents); +int pextents,int alloc_extents);extern int ext4_block_zero_eof (struct inode *inode,loff_t from,loff_t end);extern int ext4_zero_partial_blocks (struct inode *inode,loff_t lstart,loff_t length,bool *did_zero);fs/ext4/extents.c +1
-1@@ -4976,7 +4976,7 @@ int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode,* it can tell if the extent in the cache is a split extent. * But for now let's assume pextents as 2 always. */ - credits =ext4_meta_trans_blocks (inode, max_blocks,2 ); + credits =ext4_meta_trans_blocks (inode, max_blocks,2 ,0 ); }if (credits) {fs/ext4/inode.c +2
-0@@ -3693,9 +3693,11 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,if (ret <0 )return ret;if (map->m_len < orig_mlen) { +int bpg =EXT4_BLOCKS_PER_GROUP (inode->i_sb); + map->m_len = orig_mlen;
Quoted 8 lines
fs/ext4/inode.c static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map, · line 3700
Quoted 4 lines
fs/ext4/inode.c +2
-2- dio_credits =ext4_meta_trans_blocks (inode, orig_mlen, - map->m_len); + dio_credits =ext4_meta_trans_blocks (inode, map->m_len, + map->m_len,0 );This pertains to the mixed map case, where holes may exist in between. My understanding is that we cannot assume all blocks are already allocated. Is this right?
Hi Yi,
With atomic writes, we can only allow multi block writes when bigalloc is set. So the common cases are that the whole bigalloc cluster is present or whole is missing.
If we do see map->m_len < orig_mlen that means the bigalloc cluster range we queried has some allocation. This can only happen if someone punched through the cluster because punch-hole can still happen at block granularity.
Regardless, if we do see a mixed map here, it means the cluster is allocated already, just a few blocks are punched in it, hence we won't need to do any allocation.
Regards,
ojaswin
Quoted 70 lines
Quoted 19 lines
fs/ext4/inode.c +2
-9}else { dio_credits =ext4_chunk_trans_blocks (inode, map->m_len);@@ -6307,17 +6309,17 @@ static int ext4_index_trans_blocks(struct inode *inode, int lblocks, }/* -* Account for index blocks, block groups bitmaps and block group -* descriptor blocks if modify datablocks and index blocks -* worse case, the indexs blocks spread over different block groups -* -* If datablocks are discontiguous, they are possible to spread over -* different block groups too. If they are contiguous, with flexbg, -* they could still across block group boundary. -* -* Also account for superblock, inode, quota and xattr blocks +* Calculate number of credits needed in a transaction to: +* * Allocate data blocks from @alloc_extents different groups - note that fs/ext4/inode.c static int ext4_index_trans_blocks(struct inode *inode, int lblocks, · line 6322
Quoted 10 lines
fs/ext4/inode.c +8
-1+* with flexbg a single physical extent can span multiple groups but +* single mballoc request only returns extent within one group. +* * Allocate metatadata (extent tree blocks, indirect blocks) to store +* pointers to @pextents data extents having @lblocks in total. +* * Modify extent tree / indirect block tree, inode, superblock, quota +* tracking, xattr blocks */ -int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents) +int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents, +int alloc_extents)From what I can see,
alloc_extentscurrently can only be 0 or equal to pextents — 0 means a pure conversion, and equal to pextents means a new allocation. pextents is supposed to cover the total count ofalloc_extents. Should we add some sanity checks to guard against invalid inputs, likealloc_extents> pextents?Thanks,
Yi.Quoted 30 lines
fs/ext4/inode.c +3
-3{ext4_group_t groups, ngroups =ext4_get_groups_count (inode->i_sb);int gdpblocks;@@ -6334,7 +6336,7 @@ int ext4_meta_trans_blocks(struct inode *inode, int lblocks, int pextents)* Now let's see how many group bitmaps and group descriptors need * to account */ - groups = idxblocks + pextents; + groups = idxblocks + alloc_extents; gdpblocks = groups;if (groups > ngroups) groups = ngroups;@@ -6360,7 +6362,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks) {int ret; - ret =ext4_meta_trans_blocks (inode, nrblocks,1 ); + ret =ext4_meta_trans_blocks (inode, nrblocks,1 ,1 );/* Account for data blocks for journalled mode */ if (ext4_should_journal_data (inode)) ret += nrblocks;@@ -6378,7 +6380,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks)*/ int ext4_chunk_trans_blocks (struct inode *inode,int nrblocks) { -return ext4_meta_trans_blocks (inode, nrblocks,1 ); +return ext4_meta_trans_blocks (inode, nrblocks,1 ,1 ); }/*
Re: [PATCH 1/3] ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
Quoted 81 lines · Ojaswin Mujoo
Quoted 60 lines · Zhang Yi
Quoted 11 lines · Jan Kara
So far
ext4_meta_trans_blocks()expects that each extent counted in @pextents will be allocated in the transaction we estimate credits for. This is correct for the use inext4_chunk_trans_blocks()andext4_chunk_trans_extent()however the use in atomic write path (ext4_convert_unwritten_extents_atomic()andext4_iomap_alloc()forIOMAP_ATOMIC) unnecessarily overestimates the number of necessary credits as neither of them allocates any data. Add argument toext4_meta_trans_blocks()for number of extents that are going to be allocated in the transaction.Signed-off-by Jan Kara <[email protected]>Hi Jan,
I have some questions below.
Quoted 43 lines
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents.c | 2 +- fs/ext4/inode.c | 32 +++++++++++++++++--------------- 3 files changed, 19 insertions(+), 17 deletions(-)fs/ext4/ext4.h +1
-1@@ -3184,7 +3184,7 @@ extern int ext4_normal_submit_inode_data_buffers(struct jbd2_inode *jinode);extern int ext4_chunk_trans_blocks (struct inode *,int nrblocks);extern int ext4_chunk_trans_extent (struct inode *inode,int nrblocks);extern int ext4_meta_trans_blocks (struct inode *inode,int lblocks, -int pextents); +int pextents,int alloc_extents);extern int ext4_block_zero_eof (struct inode *inode,loff_t from,loff_t end);extern int ext4_zero_partial_blocks (struct inode *inode,loff_t lstart,loff_t length,bool *did_zero);fs/ext4/extents.c +1
-1@@ -4976,7 +4976,7 @@ int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode,* it can tell if the extent in the cache is a split extent. * But for now let's assume pextents as 2 always. */ - credits =ext4_meta_trans_blocks (inode, max_blocks,2 ); + credits =ext4_meta_trans_blocks (inode, max_blocks,2 ,0 ); }if (credits) {fs/ext4/inode.c +2
-0@@ -3693,9 +3693,11 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,if (ret <0 )return ret;if (map->m_len < orig_mlen) { +int bpg =EXT4_BLOCKS_PER_GROUP (inode->i_sb); + map->m_len = orig_mlen;Quoted 8 lines
fs/ext4/inode.c static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map, · line 3700
Quoted 4 lines
fs/ext4/inode.c +2
-2- dio_credits =ext4_meta_trans_blocks (inode, orig_mlen, - map->m_len); + dio_credits =ext4_meta_trans_blocks (inode, map->m_len, + map->m_len,0 );This pertains to the mixed map case, where holes may exist in between. My understanding is that we cannot assume all blocks are already allocated. Is this right?
Hi Yi,
With atomic writes, we can only allow multi block writes when bigalloc is set. So the common cases are that the whole bigalloc cluster is present or whole is missing.
If we do see
map->m_len<orig_mlenthat means the bigalloc cluster range we queried has some allocation. This can only happen if someone punched through the cluster because punch-hole can still happen at block granularity.
Quoted 6 lines
Regardless, if we do see a mixed map here, it means the cluster is allocated already, just a few blocks are punched in it, hence we won't need to do any allocation.
Regards,
ojaswin
Hi, ojaswin,
Thank you for the clarification!
My initial concern wasn't about the new cluster allocation, but rather the possibility of a new extent being allocated in this cluster. Which means the hole-covered portion might require a new extent.
But after looking at ext4_map_blocks_atomic_write_slow(), I see that it guarantees both the old and new blocks in this cluster are written type, so the new range ought to merge into the existing extent. Furthermore, ext4_ext_insert_extent() does not perform merging across leaf extent blocks, so the extent count remains unchanged — no new extent allocation should happen. But this point is indeed subtle, as you suggested, adding a comment would be helpful.
Thanks,
Yi.
Quoted 71 lines
Quoted 71 lines
Quoted 20 lines
fs/ext4/inode.c +2
-9}else { dio_credits =ext4_chunk_trans_blocks (inode, map->m_len);@@ -6307,17 +6309,17 @@ static int ext4_index_trans_blocks(struct inode *inode, int lblocks, }/* -* Account for index blocks, block groups bitmaps and block group -* descriptor blocks if modify datablocks and index blocks -* worse case, the indexs blocks spread over different block groups -* -* If datablocks are discontiguous, they are possible to spread over -* different block groups too. If they are contiguous, with flexbg, -* they could still across block group boundary. -* -* Also account for superblock, inode, quota and xattr blocks +* Calculate number of credits needed in a transaction to: +* * Allocate data blocks from @alloc_extents different groups - note that fs/ext4/inode.c static int ext4_index_trans_blocks(struct inode *inode, int lblocks, · line 6322
Quoted 10 lines
fs/ext4/inode.c +8
-1+* with flexbg a single physical extent can span multiple groups but +* single mballoc request only returns extent within one group. +* * Allocate metatadata (extent tree blocks, indirect blocks) to store +* pointers to @pextents data extents having @lblocks in total. +* * Modify extent tree / indirect block tree, inode, superblock, quota +* tracking, xattr blocks */ -int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents) +int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents, +int alloc_extents)From what I can see,
alloc_extentscurrently can only be 0 or equal to pextents — 0 means a pure conversion, and equal to pextents means a new allocation. pextents is supposed to cover the total count ofalloc_extents. Should we add some sanity checks to guard against invalid inputs, likealloc_extents> pextents?Thanks,
Yi.Quoted 30 lines
fs/ext4/inode.c +3
-3{ext4_group_t groups, ngroups =ext4_get_groups_count (inode->i_sb);int gdpblocks;@@ -6334,7 +6336,7 @@ int ext4_meta_trans_blocks(struct inode *inode, int lblocks, int pextents)* Now let's see how many group bitmaps and group descriptors need * to account */ - groups = idxblocks + pextents; + groups = idxblocks + alloc_extents; gdpblocks = groups;if (groups > ngroups) groups = ngroups;@@ -6360,7 +6362,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks) {int ret; - ret =ext4_meta_trans_blocks (inode, nrblocks,1 ); + ret =ext4_meta_trans_blocks (inode, nrblocks,1 ,1 );/* Account for data blocks for journalled mode */ if (ext4_should_journal_data (inode)) ret += nrblocks;@@ -6378,7 +6380,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks)*/ int ext4_chunk_trans_blocks (struct inode *inode,int nrblocks) { -return ext4_meta_trans_blocks (inode, nrblocks,1 ); +return ext4_meta_trans_blocks (inode, nrblocks,1 ,1 ); }/*
Ojaswin Mujoo
reviewerRe: [PATCH 1/3] ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
Quoted 97 lines · Zhang Yi
Quoted 81 lines · Ojaswin Mujoo
Quoted 60 lines · Zhang Yi
Quoted 11 lines · Jan Kara
So far
ext4_meta_trans_blocks()expects that each extent counted in @pextents will be allocated in the transaction we estimate credits for. This is correct for the use inext4_chunk_trans_blocks()andext4_chunk_trans_extent()however the use in atomic write path (ext4_convert_unwritten_extents_atomic()andext4_iomap_alloc()forIOMAP_ATOMIC) unnecessarily overestimates the number of necessary credits as neither of them allocates any data. Add argument toext4_meta_trans_blocks()for number of extents that are going to be allocated in the transaction.Signed-off-by Jan Kara <[email protected]>Hi Jan,
I have some questions below.
Quoted 43 lines
--- fs/ext4/ext4.h | 2 +- fs/ext4/extents.c | 2 +- fs/ext4/inode.c | 32 +++++++++++++++++--------------- 3 files changed, 19 insertions(+), 17 deletions(-)fs/ext4/ext4.h +1
-1@@ -3184,7 +3184,7 @@ extern int ext4_normal_submit_inode_data_buffers(struct jbd2_inode *jinode);extern int ext4_chunk_trans_blocks (struct inode *,int nrblocks);extern int ext4_chunk_trans_extent (struct inode *inode,int nrblocks);extern int ext4_meta_trans_blocks (struct inode *inode,int lblocks, -int pextents); +int pextents,int alloc_extents);extern int ext4_block_zero_eof (struct inode *inode,loff_t from,loff_t end);extern int ext4_zero_partial_blocks (struct inode *inode,loff_t lstart,loff_t length,bool *did_zero);fs/ext4/extents.c +1
-1@@ -4976,7 +4976,7 @@ int ext4_convert_unwritten_extents_atomic(handle_t *handle, struct inode *inode,* it can tell if the extent in the cache is a split extent. * But for now let's assume pextents as 2 always. */ - credits =ext4_meta_trans_blocks (inode, max_blocks,2 ); + credits =ext4_meta_trans_blocks (inode, max_blocks,2 ,0 ); }if (credits) {fs/ext4/inode.c +2
-0@@ -3693,9 +3693,11 @@ static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map,if (ret <0 )return ret;if (map->m_len < orig_mlen) { +int bpg =EXT4_BLOCKS_PER_GROUP (inode->i_sb); + map->m_len = orig_mlen;Quoted 8 lines
fs/ext4/inode.c static int ext4_iomap_alloc(struct inode *inode, struct ext4_map_blocks *map, · line 3700
Quoted 4 lines
fs/ext4/inode.c +2
-2- dio_credits =ext4_meta_trans_blocks (inode, orig_mlen, - map->m_len); + dio_credits =ext4_meta_trans_blocks (inode, map->m_len, + map->m_len,0 );This pertains to the mixed map case, where holes may exist in between. My understanding is that we cannot assume all blocks are already allocated. Is this right?
Hi Yi,
With atomic writes, we can only allow multi block writes when bigalloc is set. So the common cases are that the whole bigalloc cluster is present or whole is missing.
If we do see
map->m_len<orig_mlenthat means the bigalloc cluster range we queried has some allocation. This can only happen if someone punched through the cluster because punch-hole can still happen at block granularity.Quoted 6 lines
Regardless, if we do see a mixed map here, it means the cluster is allocated already, just a few blocks are punched in it, hence we won't need to do any allocation.
Regards,
ojaswinHi, ojaswin,
Thank you for the clarification!
My initial concern wasn't about the new cluster allocation, but rather the possibility of a new extent being allocated in this cluster. Which means the hole-covered portion might require a new extent.
Quoted 7 lines
But after looking at
ext4_map_blocks_atomic_write_slow(), I see that it guarantees both the old and new blocks in this cluster are written type, so the new range ought to merge into the existing extent. Furthermore,ext4_ext_insert_extent()does not perform merging across leaf extent blocks, so the extent count remains unchanged — no new extent allocation should happen. But this point is indeed subtle, as you suggested, adding a comment would be helpful.
Yes that's correct, we will keep merging into the existing extents so there will be no new allocation (data or metadata) needed.
Regards,
ojaswin
Quoted 76 lines
Thanks,
Yi.Quoted 71 lines
Quoted 71 lines
Quoted 20 lines
fs/ext4/inode.c +2
-9}else { dio_credits =ext4_chunk_trans_blocks (inode, map->m_len);@@ -6307,17 +6309,17 @@ static int ext4_index_trans_blocks(struct inode *inode, int lblocks, }/* -* Account for index blocks, block groups bitmaps and block group -* descriptor blocks if modify datablocks and index blocks -* worse case, the indexs blocks spread over different block groups -* -* If datablocks are discontiguous, they are possible to spread over -* different block groups too. If they are contiguous, with flexbg, -* they could still across block group boundary. -* -* Also account for superblock, inode, quota and xattr blocks +* Calculate number of credits needed in a transaction to: +* * Allocate data blocks from @alloc_extents different groups - note that fs/ext4/inode.c static int ext4_index_trans_blocks(struct inode *inode, int lblocks, · line 6322
Quoted 10 lines
fs/ext4/inode.c +8
-1+* with flexbg a single physical extent can span multiple groups but +* single mballoc request only returns extent within one group. +* * Allocate metatadata (extent tree blocks, indirect blocks) to store +* pointers to @pextents data extents having @lblocks in total. +* * Modify extent tree / indirect block tree, inode, superblock, quota +* tracking, xattr blocks */ -int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents) +int ext4_meta_trans_blocks (struct inode *inode,int lblocks,int pextents, +int alloc_extents)From what I can see,
alloc_extentscurrently can only be 0 or equal to pextents — 0 means a pure conversion, and equal to pextents means a new allocation. pextents is supposed to cover the total count ofalloc_extents. Should we add some sanity checks to guard against invalid inputs, likealloc_extents> pextents?Thanks,
Yi.Quoted 30 lines
fs/ext4/inode.c +3
-3{ext4_group_t groups, ngroups =ext4_get_groups_count (inode->i_sb);int gdpblocks;@@ -6334,7 +6336,7 @@ int ext4_meta_trans_blocks(struct inode *inode, int lblocks, int pextents)* Now let's see how many group bitmaps and group descriptors need * to account */ - groups = idxblocks + pextents; + groups = idxblocks + alloc_extents; gdpblocks = groups;if (groups > ngroups) groups = ngroups;@@ -6360,7 +6362,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks) {int ret; - ret =ext4_meta_trans_blocks (inode, nrblocks,1 ); + ret =ext4_meta_trans_blocks (inode, nrblocks,1 ,1 );/* Account for data blocks for journalled mode */ if (ext4_should_journal_data (inode)) ret += nrblocks;@@ -6378,7 +6380,7 @@ int ext4_chunk_trans_extent(struct inode *inode, int nrblocks)*/ int ext4_chunk_trans_blocks (struct inode *inode,int nrblocks) { -return ext4_meta_trans_blocks (inode, nrblocks,1 ); +return ext4_meta_trans_blocks (inode, nrblocks,1 ,1 ); }/*
Jan Kara
reviewerPATCH 2/3 ext4: Fix transaction overflow during writeback
In reply to Jan KaraCommit 95ad8ee45cdb ("ext4: correct the reserved credits for extent conversion") was correct to note that we need to reserve enough credits for all extents possibly underlying a large folio. However it was too eager to reduce the number of reserved credits. Extent conversion may not only need to touch several leaf extent blocks, it may also need to split extents - for example a single large unwritten extent may need to be split into many small written ones in case of sparse folio dirtying. This can thus result not only in extent leaf modifications but also in a need to allocate new extent tree nodes. As a result the reserved transaction credits were not sufficient in some corner cases. Use ext4_meta_trans_blocks() for correct upper bound credit estimate.
--- fs/ext4/inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/ext4/inode.c +3 -3
@@ -2852,10 +2852,10 @@ static int ext4_do_writepages(struct mpage_da_data *mpd)if (ext4_should_dioread_nolock (inode)) {int bpf =ext4_journal_blocks_per_folio (inode);/* -* We may need to convert up to one extent per block in -* the folio and we may dirty the inode. +* We may need to convert up to one extent per block in the +* folio. */ - rsv_blocks =1 +ext4_ext_index_trans_blocks (inode, bpf); + rsv_blocks =ext4_meta_trans_blocks (inode, bpf, bpf,0 ); }if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX )
Signature · 1 line
2.51.0
Re: [PATCH 2/3] ext4: Fix transaction overflow during writeback
Quoted 12 lines · Jan Kara
Commit
95ad8ee45cdb("ext4: correct the reserved credits for extent conversion") was correct to note that we need to reserve enough credits for all extents possibly underlying a large folio. However it was too eager to reduce the number of reserved credits. Extent conversion may not only need to touch several leaf extent blocks, it may also need to split extents - for example a single large unwritten extent may need to be split into many small written ones in case of sparse folio dirtying. This can thus result not only in extent leaf modifications but also in a need to allocate new extent tree nodes. As a result the reserved transaction credits were not sufficient in some corner cases. Useext4_meta_trans_blocks()for correct upper bound credit estimate.
Quoted 2 lines
Fixes 95ad8ee45cdb ("ext4: correct the reserved credits for extent conversion")Signed-off-by Jan Kara <[email protected]>
Indeed, I overlooked that the newly allocated index blocks will modify bitmaps and group descriptor blocks. Given that the folio length is capped at 2048 filesystem blocks, calling ext4_meta_trans_blocks(bpf, 0) shouldn't result in an overly large credits calculation. So,
Quoted 22 lines
--- fs/ext4/inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/ext4/inode.c +3
-3@@ -2852,10 +2852,10 @@ static int ext4_do_writepages(struct mpage_da_data *mpd)if (ext4_should_dioread_nolock (inode)) {int bpf =ext4_journal_blocks_per_folio (inode);/* -* We may need to convert up to one extent per block in -* the folio and we may dirty the inode. +* We may need to convert up to one extent per block in the +* folio. */ - rsv_blocks =1 +ext4_ext_index_trans_blocks (inode, bpf); + rsv_blocks =ext4_meta_trans_blocks (inode, bpf, bpf,0 ); }if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX )
Re: [PATCH 2/3] ext4: Fix transaction overflow during writeback
In reply to Jan KaraQuoted 23 lines · Jan Kara
Commit
95ad8ee45cdb("ext4: correct the reserved credits for extent conversion") was correct to note that we need to reserve enough credits for all extents possibly underlying a large folio. However it was too eager to reduce the number of reserved credits. Extent conversion may not only need to touch several leaf extent blocks, it may also need to split extents - for example a single large unwritten extent may need to be split into many small written ones in case of sparse folio dirtying. This can thus result not only in extent leaf modifications but also in a need to allocate new extent tree nodes. As a result the reserved transaction credits were not sufficient in some corner cases. Useext4_meta_trans_blocks()for correct upper bound credit estimate.Fixes 95ad8ee45cdb ("ext4: correct the reserved credits for extent conversion")Signed-off-by Jan Kara <[email protected]>--- fs/ext4/inode.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/ext4/inode.c +0
-0
fs/ext4/inode.c static int ext4_do_writepages(struct mpage_da_data *mpd) · line 2858
Quoted 10 lines
fs/ext4/inode.c +3
-3@@ -2852,10 +2852,10 @@ static int ext4_do_writepages(struct mpage_da_data *mpd)if (ext4_should_dioread_nolock (inode)) {int bpf =ext4_journal_blocks_per_folio (inode);/* -* We may need to convert up to one extent per block in -* the folio and we may dirty the inode. +* We may need to convert up to one extent per block in the +* folio. */ - rsv_blocks =1 +ext4_ext_index_trans_blocks (inode, bpf); + rsv_blocks =ext4_meta_trans_blocks (inode, bpf, bpf,0 );
Looks good Jan, feel free to add:
Regards,
ojaswin
Quoted 6 lines
fs/ext4/inode.c +0
-0}if (wbc->range_start ==0 && wbc->range_end ==LLONG_MAX )Signature · 1 line
2.51.0
Jan Kara
reviewerPATCH 3/3 ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks()
In reply to Jan KaraThe estimate of the number of impacted extent tree index blocks could be one-too-low. If we modify say 2 extents, already two leaf index blocks could be impacted, not just one the current estimate counts with. Fix the estimate.
--- fs/ext4/extents.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)
fs/ext4/extents.c +11 -3
@@ -2427,9 +2427,17 @@ int ext4_ext_index_trans_blocks(struct inode *inode, int extents)*/ if (extents <=1 ) index = (EXT4_MAX_EXTENT_DEPTH *2 ) + extents; -else - index = (EXT4_MAX_EXTENT_DEPTH *3 ) + -DIV_ROUND_UP (extents,ext4_ext_space_block (inode,0 )); +else { +int ext_max =ext4_ext_space_block (inode,0 ); + + index =EXT4_MAX_EXTENT_DEPTH *3 ; +/* +* Modified extents need not start at the beginning of the +* leaf. Already two extents may need two leaf block +* modifications... +*/ + index +=DIV_ROUND_UP (extents + ext_max -1 , ext_max); + }return index; }
Signature · 1 line
2.51.0
Re: [PATCH 3/3] ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks()
Quoted 6 lines · Jan Kara
The estimate of the number of impacted extent tree index blocks could be one-too-low. If we modify say 2 extents, already two leaf index blocks could be impacted, not just one the current estimate counts with. Fix the estimate.
Signed-off-by Jan Kara <[email protected]>
This is a good catch!
Quoted 29 lines
--- fs/ext4/extents.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)fs/ext4/extents.c +11
-3@@ -2427,9 +2427,17 @@ int ext4_ext_index_trans_blocks(struct inode *inode, int extents)*/ if (extents <=1 ) index = (EXT4_MAX_EXTENT_DEPTH *2 ) + extents; -else - index = (EXT4_MAX_EXTENT_DEPTH *3 ) + -DIV_ROUND_UP (extents,ext4_ext_space_block (inode,0 )); +else { +int ext_max =ext4_ext_space_block (inode,0 ); + + index =EXT4_MAX_EXTENT_DEPTH *3 ; +/* +* Modified extents need not start at the beginning of the +* leaf. Already two extents may need two leaf block +* modifications... +*/ + index +=DIV_ROUND_UP (extents + ext_max -1 , ext_max); + }return index; }
Ojaswin Mujoo
reviewerRe: [PATCH 3/3] ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks()
In reply to Jan KaraQuoted 2 lines · Jan Kara
The estimate of the number of impacted extent tree index blocks could be one-too-low. If we modify say 2 extents, already two leaf index blocks
Hi Jan,
Its minor but just to be sure, by two "leaf index blocks" do you mean the two leaf blocks that hold extents or the last index blocks which hold pointers to leaf blocks?
Quoted 36 lines
could be impacted, not just one the current estimate counts with. Fix the estimate.
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/extents.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)fs/ext4/extents.c +11
-3@@ -2427,9 +2427,17 @@ int ext4_ext_index_trans_blocks(struct inode *inode, int extents)*/ if (extents <=1 ) index = (EXT4_MAX_EXTENT_DEPTH *2 ) + extents; -else - index = (EXT4_MAX_EXTENT_DEPTH *3 ) + -DIV_ROUND_UP (extents,ext4_ext_space_block (inode,0 )); +else { +int ext_max =ext4_ext_space_block (inode,0 ); + + index =EXT4_MAX_EXTENT_DEPTH *3 ; +/* +* Modified extents need not start at the beginning of the +* leaf. Already two extents may need two leaf block +* modifications... +*/ + index +=DIV_ROUND_UP (extents + ext_max -1 , ext_max); + }return index; }Signature · 1 line
2.51.0
Jan Kara
reviewerRe: [PATCH 3/3] ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks()
Hi!
Quoted 7 lines · Ojaswin Mujoo
Quoted 2 lines · Jan Kara
The estimate of the number of impacted extent tree index blocks could be one-too-low. If we modify say 2 extents, already two leaf index blocks
Its minor but just to be sure, by two "leaf index blocks" do you mean the two leaf blocks that hold extents or the last index blocks which hold pointers to leaf blocks?
I meant two leaf blocks that hold extents.
Honza
Quoted 37 lines
Quoted 37 lines
could be impacted, not just one the current estimate counts with. Fix the estimate.
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/extents.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)fs/ext4/extents.c +11
-3@@ -2427,9 +2427,17 @@ int ext4_ext_index_trans_blocks(struct inode *inode, int extents)*/ if (extents <=1 ) index = (EXT4_MAX_EXTENT_DEPTH *2 ) + extents; -else - index = (EXT4_MAX_EXTENT_DEPTH *3 ) + -DIV_ROUND_UP (extents,ext4_ext_space_block (inode,0 )); +else { +int ext_max =ext4_ext_space_block (inode,0 ); + + index =EXT4_MAX_EXTENT_DEPTH *3 ; +/* +* Modified extents need not start at the beginning of the +* leaf. Already two extents may need two leaf block +* modifications... +*/ + index +=DIV_ROUND_UP (extents + ext_max -1 , ext_max); + }return index; }Signature · 1 line
2.51.0
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Re: [PATCH 3/3] ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks()
Quoted 7 lines · Jan Kara
Hi!
Quoted 4 lines · Ojaswin Mujoo
Quoted 2 lines · Jan Kara
The estimate of the number of impacted extent tree index blocks could be one-too-low. If we modify say 2 extents, already two leaf index blocks
Quoted 7 lines
Quoted 3 lines
Its minor but just to be sure, by two "leaf index blocks" do you mean the two leaf blocks that hold extents or the last index blocks which hold pointers to leaf blocks?
I meant two leaf blocks that hold extents.
Honza
Yep, thanks for confirming. I think Ted has picked these up but FWIW:
Regards,
ojaswin
Quoted 41 lines
Quoted 38 lines
Quoted 38 lines
could be impacted, not just one the current estimate counts with. Fix the estimate.
Signed-off-by Jan Kara <[email protected]>--- fs/ext4/extents.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-)fs/ext4/extents.c +11
-3@@ -2427,9 +2427,17 @@ int ext4_ext_index_trans_blocks(struct inode *inode, int extents)*/ if (extents <=1 ) index = (EXT4_MAX_EXTENT_DEPTH *2 ) + extents; -else - index = (EXT4_MAX_EXTENT_DEPTH *3 ) + -DIV_ROUND_UP (extents,ext4_ext_space_block (inode,0 )); +else { +int ext_max =ext4_ext_space_block (inode,0 ); + + index =EXT4_MAX_EXTENT_DEPTH *3 ; +/* +* Modified extents need not start at the beginning of the +* leaf. Already two extents may need two leaf block +* modifications... +*/ + index +=DIV_ROUND_UP (extents + ext_max -1 , ext_max); + }return index; }Signature · 1 line
2.51.0Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 8 lines · Jan Kara
This patch set fixes credit estimates for extent tree modification during convertion of unwritten extents on writeback completion. Otherwise we are occasionally seeing assertion failures during extent conversion in end io completion callback due to insufficient transation credits.
Honza
[...]
Applied, thanks!
[1/3] ext4: Teach ext4_meta_trans_blocks() about number of allocated extents
commit: 25b2a7e8d420c49fc46f25b6f953603e4dae1dbf
[2/3] ext4: Fix transaction overflow during writeback
commit: 46e8e31771f4f1c5e1cdec37a889a6730e42e9f1
[3/3] ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks()
commit: 9091c97be34083587a75db174aab51551d8e8543Best regards,
Signature · 1 line
Theodore Ts'o <[email protected]>
Ojaswin Mujoo
reviewerQuoted 13 lines · Theodore Ts'o
Quoted 8 lines · Jan Kara
This patch set fixes credit estimates for extent tree modification during convertion of unwritten extents on writeback completion. Otherwise we are occasionally seeing assertion failures during extent conversion in end io completion callback due to insufficient transation credits.
Honza
[...]
Applied, thanks!
Quoted 10 lines
[1/3] ext4: Teach
ext4_meta_trans_blocks()about number of allocated extentscommit: 25b2a7e8d420c49fc46f25b6f953603e4dae1dbf [2/3] ext4: Fix transaction overflow during writeback commit: 46e8e31771f4f1c5e1cdec37a889a6730e42e9f1 [3/3] ext4: Fix estimate extent index blocks in ext4_ext_index_trans_blocks() commit: 9091c97be34083587a75db174aab51551d8e8543Best regards,
Signature · 1 line
Theodore Ts'o <[email protected]>
Hey Ted,
I just noticed that the RVBs were not picked in your ext4/dev branch. I think they got missed somehow.
Regards,
ojaswin
Theodore Tso
maintainerQuoted 3 lines · Ojaswin Mujoo
I just noticed that the RVBs were not picked in your ext4/dev branch. I think they got missed somehow.
Hmm, I see them on the dev branch:
https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/log/?h=dev
and they are on today's linux-next branch.
- Ted
Ojaswin Mujoo
reviewerQuoted 8 lines · Theodore Tso
Quoted 3 lines · Ojaswin Mujoo
I just noticed that the RVBs were not picked in your ext4/dev branch. I think they got missed somehow.
Hmm, I see them on the dev branch:
https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/log/?h=dev
Hi Ted,
I don't see the Reviewed-by tags here as well.
Regards,
ojaswin
Quoted 4 lines
and they are on today's linux-next branch.
- Ted
Jan Kara
reviewerQuoted 8 lines · Theodore Tso
Quoted 3 lines · Ojaswin Mujoo
I just noticed that the RVBs were not picked in your ext4/dev branch. I think they got missed somehow.
Hmm, I see them on the dev branch:
https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/log/?h=dev
Hum, if I look at e.g. https://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git/commit/?h=dev&id=25b2a7e8d420c49fc46f25b6f953603e4dae1dbf
I can see:
ext4: teach ext4_meta_trans_blocks() about number of allocated extents
...
allocated in the transaction.
So indeed as Ojaswin writes there are no Reviewed-by tags from Zhang and Ojaswin. And I think that's because you have picked the patches into your tree before Ojaswin and Zhang actually sent their RB tags. Anyway I think commits are on their way to Linus now so not much to do here...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR