btrfs: trim: Fix a bug certain range may not be trimmed properly
v4 cover btrfs: trim: Fix a bug certain range may not be trimmed properly
There is a bug report about discard mount option not trimming some range properly, and causing unexpected space usage for thin device.
It turns out to be that, if there are pinned extents across block group boundary, we will only trim to the end of current block group, skipping the remaining.
This patchset will fix it by ensuring btrfs_discard_extent() will iterate the full range before exiting.
The first patch is just a tiny readability improvement found during the fix.
The second patch is the main body of the fix.
Meanwhile I'm still looking into how to craft such test case for btrfs, so the test case may be late for several days.
Changelog:
v2:
- Fold the __btrfs_map_block_for_discard() @length change into the main patch
Since the @length parameter change itself doesn't make much sense, folding it into the fix looks more reasonable.
- Split the tiny readability improvement patch into its own patch
v3:
- Reset @num_bytes for btrfs_map_block() to limit the trim range inside the original range.
Or we can trim into existing extent to corrupt data.
v4:
- Use better ASCII graph to show the bug
No code change at all.
Qu Wenruo (2):
btrfs: volumes: Use more straightforward way to calculate map length
btrfs: extent-tree: Ensure we trim ranges across block group boundary
fs/btrfs/extent-tree.c | 41 +++++++++++++++++++++++++++++++---------- fs/btrfs/volumes.c | 8 +++++--- 2 files changed, 36 insertions(+), 13 deletions(-)
Signature · 1 line
2.23.0
PATCH v4 1/2 btrfs: volumes: Use more straightforward way to calculate map length
The old code goes:
offset = logical - em->start; length = min_t(u64, em->len - offset, length);
Where @length calculate is dependent on offset, it can take reader several more seconds to find it's just the same code as:
offset = logical - em->start; length = min_t(u64, em->start + em->len - logical, length);
Use above code to make the length calculate independent from other variable, thus slightly increase the readability.
--- fs/btrfs/volumes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/volumes.c +1 -1
@@ -5616,7 +5616,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, } offset = logical - em->start; - length =min_t (u64, em->len - offset, length); + length =min_t (u64, em->start + em->len - logical, length); stripe_len = map->stripe_len;/*
Signature · 1 line
2.23.0
Re: [PATCH v4 1/2] btrfs: volumes: Use more straightforward way to calculate map length
Quoted 8 lines · Qu Wenruo
The old code goes:
offset = logical - em->start; length = min_t(u64, em->len - offset, length);Where @length calculate is dependent on offset, it can take reader several more seconds to find it's just the same code as:
Quoted 7 lines
offset = logical - em->start; length = min_t(u64, em->start + em->len - logical, length);Use above code to make the length calculate independent from other variable, thus slightly increase the readability.
Signed-off-by Qu Wenruo <[email protected]>
Thanks,
Josef
PATCH v4 2/2 btrfs: extent-tree: Ensure we trim ranges across block group boundary
In reply to Qu Wenruo[BUG]
When deleting large files (which cross block group boundary) with discard mount option, we find some btrfs_discard_extent() calls only trimmed part of its space, not the whole range:
btrfs_discard_extent: type=0x1 start=19626196992 len=2144530432 trimmed=1073741824 ratio=50%
type: bbio->map_type, in above case, it's SINGLE DATA.
start: Logical address of this trim
len: Logical length of this trim
trimmed: Physically trimmed bytes
ratio: trimmed / len
Thus leading some unused space not discarded.
[CAUSE]
When discard mount option is specified, after a transaction is fully committed (super block written to disk), we begin to cleanup pinned extents in the following call chain:
btrfs_commit_transaction()
|- btrfs_finish_extent_commit()
|- find_first_extent_bit(unpin, 0, &start, &end, EXTENT_DIRTY); |- btrfs_discard_extent()
However pinned extents are recorded in an extent_io_tree, which can merge adjacent extent states.
When a large file get deleted and it has adjacent file extents across block group boundary, we will get a large merged range, like this:
|<--- BG1 --->|<--- BG2 --->|
|//////|<-- Range to discard --->|/////|To discard that range, we have the following calls:btrfs_discard_extent()
|- btrfs_map_block() | Returned bbio will end at BG1's end. As btrfs_map_block() | never returns result across block group boundary. |- btrfs_issuse_discard()
Issue discard for each stripe.
So we will only discard the range in BG1, not the remaining part in BG2.
Furthermore, this bug is not that reliably observed, for above case, if there is no other extent in BG2, BG2 will be empty and btrfs will trim all space of BG2, covering up the bug.
[FIX]
- Allow __btrfs_map_block_for_discard() to modify @length parameter btrfs_map_block() uses its @length paramter to notify the caller how many bytes are mapped in current call.
With __btrfs_map_block_for_discard() also modifing the @length, btrfs_discard_extent() now understands when to do extra trim.
- Call btrfs_map_block() in a loop until we hit the range end Since we now know how many bytes are mapped each time, we can iterate through each block group boundary and issue correct trim for each range.
--- fs/btrfs/extent-tree.c | 41 +++++++++++++++++++++++++++++++---------- fs/btrfs/volumes.c | 6 ++++-- 2 files changed, 35 insertions(+), 12 deletions(-)
fs/btrfs/extent-tree.c +31 -10
@@ -1306,8 +1306,10 @@ static int btrfs_issue_discard(struct block_device *bdev, u64 start, u64 len,int btrfs_discard_extent (struct btrfs_fs_info *fs_info,u64 bytenr,u64 num_bytes,u64 *actual_bytes) { -int ret; +int ret =0 ;u64 discarded_bytes =0 ; +u64 end = bytenr + num_bytes; +u64 cur = bytenr;struct btrfs_bio *bbio =NULL ;
@@ -1316,15 +1318,23 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,* associated to its stripes that don't go away while we are discarding. */ btrfs_bio_counter_inc_blocked (fs_info); -/* Tell the block device(s) that the sectors can be discarded */ - ret =btrfs_map_block (fs_info,BTRFS_MAP_DISCARD , bytenr, &num_bytes, - &bbio,0 ); -/* Error condition is -ENOMEM */ -if (!ret) { -struct btrfs_bio_stripe *stripe = bbio->stripes; +while (cur < end) { +struct btrfs_bio_stripe *stripe;int i; + num_bytes = end - cur; +/* Tell the block device(s) that the sectors can be discarded */ + ret =btrfs_map_block (fs_info,BTRFS_MAP_DISCARD , cur, + &num_bytes, &bbio,0 ); +/* +* Error can be -ENOMEM, -ENOENT (no such chunk mapping) or +* -EOPNOTSUPP. For any such error, @num_bytes is not updated, +* thus we can't continue anyway. +*/ +if (ret <0 ) + goto out; + stripe = bbio->stripes;for (i =0 ; i < bbio->num_stripes; i++, stripe++) {u64 bytes;struct request_queue *req_q;
@@ -1341,10 +1351,19 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, stripe->physical, stripe->length, &bytes); -if (!ret) +if (!ret) { discarded_bytes += bytes; -else if (ret != -EOPNOTSUPP ) -break ;/* Logic errors or -ENOMEM, or -EIO but I don't know how that could happen JDM */ + }else if (ret != -EOPNOTSUPP ) { +/* +* Logic errors or -ENOMEM, or -EIO but I don't +* know how that could happen JDM +* +* Ans since there are two loops, explicitly +* goto out to avoid confusion. +*/ +btrfs_put_bbio (bbio); + goto out; + }/* * Just in case we get back EOPNOTSUPP for some reason,
@@ -1354,7 +1373,9 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, ret =0 ; }btrfs_put_bbio (bbio); + cur += num_bytes; } +out:btrfs_bio_counter_dec (fs_info);if (actual_bytes)
fs/btrfs/volumes.c +4 -2
@@ -5578,12 +5578,13 @@ void btrfs_put_bbio(struct btrfs_bio *bbio)* replace. */ static int __btrfs_map_block_for_discard (struct btrfs_fs_info *fs_info, -u64 logical,u64 length, +u64 logical,u64 *length_ret,struct btrfs_bio **bbio_ret) {struct extent_map *em;struct map_lookup *map;struct btrfs_bio *bbio; +u64 length = *length_ret;u64 offset;u64 stripe_nr;u64 stripe_nr_end;
@@ -5617,6 +5618,7 @@ static int __btrfs_map_block_for_discard(struct btrfs_fs_info *fs_info, offset = logical - em->start; length =min_t (u64, em->start + em->len - logical, length); + *length_ret = length; stripe_len = map->stripe_len;/*
@@ -6031,7 +6033,7 @@ static int __btrfs_map_block(struct btrfs_fs_info *fs_info,if (op ==BTRFS_MAP_DISCARD )return __btrfs_map_block_for_discard (fs_info, logical, - *length, bbio_ret); + length, bbio_ret); ret =btrfs_get_io_geometry (fs_info, op, logical, *length, &geom);if (ret <0 )
Signature · 1 line
2.23.0
Re: [PATCH v4 2/2] btrfs: extent-tree: Ensure we trim ranges across block group boundary
Quoted 117 lines · Qu Wenruo
[BUG]
When deleting large files (which cross block group boundary) with discard mount option, we find somebtrfs_discard_extent()calls only trimmed part of its space, not the whole range:
btrfs_discard_extent: type=0x1 start=19626196992 len=2144530432 trimmed=1073741824 ratio=50%type:
bbio->map_type, in above case, it's SINGLE DATA.
start: Logical address of this trim
len: Logical length of this trim
trimmed: Physically trimmed bytes
ratio: trimmed / lenThus leading some unused space not discarded.
[CAUSE]
When discard mount option is specified, after a transaction is fully committed (super block written to disk), we begin to cleanup pinned extents in the following call chain:
btrfs_commit_transaction()
|-btrfs_finish_extent_commit()
|-find_first_extent_bit(unpin, 0, &start, &end, EXTENT_DIRTY); |-btrfs_discard_extent()However pinned extents are recorded in an
extent_io_tree, which can merge adjacent extent states.When a large file get deleted and it has adjacent file extents across block group boundary, we will get a large merged range, like this:
|<--- BG1 --->|<--- BG2 --->| |//////|<-- Range to discard --->|/////|To discard that range, we have the following calls:
btrfs_discard_extent()|- btrfs_map_block() | Returned bbio will end at BG1's end. As btrfs_map_block() | never returns result across block group boundary. |- btrfs_issuse_discard()Issue discard for each stripe.
So we will only discard the range in BG1, not the remaining part in BG2.
Furthermore, this bug is not that reliably observed, for above case, if there is no other extent in BG2, BG2 will be empty and btrfs will trim all space of BG2, covering up the bug.
[FIX]
- Allow__btrfs_map_block_for_discard()to modify @length parameterbtrfs_map_block()uses its @length paramter to notify the caller how many bytes are mapped in current call.
With__btrfs_map_block_for_discard()also modifing the @length,btrfs_discard_extent()now understands when to do extra trim.- Call
btrfs_map_block()in a loop until we hit the range end Since we now know how many bytes are mapped each time, we can iterate through each block group boundary and issue correct trim for each range.Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/extent-tree.c | 41 +++++++++++++++++++++++++++++++---------- fs/btrfs/volumes.c | 6 ++++-- 2 files changed, 35 insertions(+), 12 deletions(-)fs/btrfs/extent-tree.c +17
-8@@ -1306,8 +1306,10 @@ static int btrfs_issue_discard(struct block_device *bdev, u64 start, u64 len,int btrfs_discard_extent (struct btrfs_fs_info *fs_info,u64 bytenr,u64 num_bytes,u64 *actual_bytes) { -int ret; +int ret =0 ;u64 discarded_bytes =0 ; +u64 end = bytenr + num_bytes; +u64 cur = bytenr;struct btrfs_bio *bbio =NULL ;@@ -1316,15 +1318,23 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr,* associated to its stripes that don't go away while we are discarding. */ btrfs_bio_counter_inc_blocked (fs_info); -/* Tell the block device(s) that the sectors can be discarded */ - ret =btrfs_map_block (fs_info,BTRFS_MAP_DISCARD , bytenr, &num_bytes, - &bbio,0 ); -/* Error condition is -ENOMEM */ -if (!ret) { -struct btrfs_bio_stripe *stripe = bbio->stripes; +while (cur < end) { +struct btrfs_bio_stripe *stripe;int i; + num_bytes = end - cur; +/* Tell the block device(s) that the sectors can be discarded */ + ret =btrfs_map_block (fs_info,BTRFS_MAP_DISCARD , cur, + &num_bytes, &bbio,0 ); +/* +* Error can be -ENOMEM, -ENOENT (no such chunk mapping) or +* -EOPNOTSUPP. For any such error, @num_bytes is not updated, +* thus we can't continue anyway. +*/ +if (ret <0 ) + goto out; + stripe = bbio->stripes;for (i =0 ; i < bbio->num_stripes; i++, stripe++) {u64 bytes;struct request_queue *req_q;@@ -1341,10 +1351,19 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, stripe->physical, stripe->length, &bytes); -if (!ret)
fs/btrfs/extent-tree.c int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, · line 1361
Quoted 10 lines
fs/btrfs/extent-tree.c +7
-2+if (!ret) { discarded_bytes += bytes; - else if (ret != -EOPNOTSUPP) - break; /* Logic errors or -ENOMEM, or -EIO but I don't know how that could happen JDM */ + }else if (ret != -EOPNOTSUPP ) { +/* +* Logic errors or -ENOMEM, or -EIO but I don't +* know how that could happen JDM +* +* Ans since there are two loops, explicitly
Hate for there to be a v5 at this point, but it should be "and". Anyway
Thanks,
Josef
Re: [PATCH v4 2/2] btrfs: extent-tree: Ensure we trim ranges across block group boundary
Quoted 9 lines · Josef Bacik
Quoted 9 lines
fs/btrfs/extent-tree.c +1
-3@@ -1341,10 +1351,19 @@ int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, stripe->physical, stripe->length, &bytes); -if (!ret) +if (!ret) { discarded_bytes += bytes; -else if (ret != -EOPNOTSUPP ) -break ;/* Logic errors or -ENOMEM, or -EIO but I don't know how that could happen JDM */
Quoted 8 lines
fs/btrfs/extent-tree.c int btrfs_discard_extent(struct btrfs_fs_info *fs_info, u64 bytenr, · line 1361
Quoted 6 lines
fs/btrfs/extent-tree.c +6
-0+ }else if (ret != -EOPNOTSUPP ) { +/* +* Logic errors or -ENOMEM, or -EIO but I don't +* know how that could happen JDM +* +* Ans since there are two loops, explicitly Hate for there to be a v5 at this point, but it should be "and".
Fixed and the first sentence of the comment needed to be updated anyway.