btrfs: zoned: limitt active zones to max_open_zones
v2 cover btrfs: zoned: limitt active zones to max_open_zones
When there is no active zone limit, we can technically write into any number of zones at the same time. However, exceeding the max open zones can degrade performance. To prevent this, this series set the max_active_zones to bdev_max_open_zones() if there is no active zone limit.
Using max_open_zones as an active zone limit enables the active zone tracking and enforcement on SMR too. The revealed some bugs on the active zone tracking. This series also includes the fixes for them.
Changes:
- v2:
- Refine commit message and add code comment to descrive the WARN condition.
- Introduce and use BTRFS_DEFAULT_MAX_ACTIVE_ZONES (= 128).
Naohiro Aota (3):
btrfs: zoned: do not select metadata BG as finish target
btrfs: zoned: fix write time activation failure for metadata block
group
btrfs: zoned: limit active zones to max_open_zones
fs/btrfs/zoned.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
Signature · 1 line
2.50.0
PATCH v2 1/3 btrfs: zoned: do not select metadata BG as finish target
We call btrfs_zone_finish_one_bg() to zone finish one block group and make room to activate another block group. Currently, we can choose a metadata block group as a target. But, as we reserve an active metadata block group, we no longer want to select a metadata block group. So, skip it in the loop.
--- fs/btrfs/zoned.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/btrfs/zoned.c +1 -1
@@ -2650,7 +2650,7 @@ int btrfs_zone_finish_one_bg(struct btrfs_fs_info *fs_info)spin_lock (&block_group->lock);if (block_group->reserved || block_group->alloc_offset ==0 || - (block_group->flags &BTRFS_BLOCK_GROUP_SYSTEM ) || + !(block_group->flags &BTRFS_BLOCK_GROUP_DATA ) ||test_bit (BLOCK_GROUP_FLAG_ZONED_DATA_RELOC , &block_group->runtime_flags)) {spin_unlock (&block_group->lock);continue ;
Signature · 1 line
2.50.0
PATCH v2 2/3 btrfs: zoned: fix write time activation failure for metadata block group
In reply to Naohiro AotaSince commit 13bb483d32ab ("btrfs: zoned: activate metadata block group on write time"), we activate a metadata block group at the write time. If the zone capacity is small enough, we can allocate the entire region before the first write. Then, we hit the btrfs_zoned_bg_is_full() in btrfs_zone_activate() and the activation fails.
For a data block group, we activate it at the allocation time and we should check the fullness condition in the caller side. Add, a WARN to check the fullness condition.
For a metadata block group, we don't need the fullness check because we activate it at the write time. Instead, activating it once it is written should be invalid. Catch that with a WARN too.
--- fs/btrfs/zoned.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
fs/btrfs/zoned.c +9 -4
@@ -2168,10 +2168,15 @@ bool btrfs_zone_activate(struct btrfs_block_group *block_group) goto out_unlock; } -/* No space left */ -if (btrfs_zoned_bg_is_full (block_group)) { - ret = false; - goto out_unlock; +if (block_group->flags &BTRFS_BLOCK_GROUP_DATA ) { +/* The caller should check if the block group is full. */ +if (WARN_ON_ONCE (btrfs_zoned_bg_is_full (block_group))) { + ret = false; + goto out_unlock; + } + }else { +/* Since it is already written, it should have been active. */ +WARN_ON_ONCE (block_group->meta_write_pointer != block_group->start); }for (i =0 ; i < map->num_stripes; i++) {
Signature · 1 line
2.50.0
Re: [PATCH v2 2/3] btrfs: zoned: fix write time activation failure for metadata block group
Quoted 2 lines · Naohiro Aota
Since commit
13bb483d32ab("btrfs: zoned: activate metadata block group on write time"), we activate a metadata block group at the write time. If the
s/the// ~^
Quoted 2 lines
zone capacity is small enough, we can allocate the entire region before the first write. Then, we hit the
btrfs_zoned_bg_is_full()in
+ condition? ~^
Quoted 3 lines
btrfs_zone_activate()and the activation fails.For a data block group, we activate it at the allocation time and we should
s/the// ~^
Quoted 1 line
check the fullness condition in the caller side. Add, a WARN to check the
s/,// ~^
Quoted 4 lines
fullness condition.
For a metadata block group, we don't need the fullness check because we activate it at the write time. Instead, activating it once it is written
s/the// ~^
Quoted 5 lines
should be invalid. Catch that with a WARN too.
Fixes 13bb483d32ab ("btrfs: zoned: activate metadata block group on write time")CC [email protected] # 6.6+Signed-off-by Naohiro Aota <[email protected]>
Otherwise looks good to me,
Re: [PATCH v2 2/3] btrfs: zoned: fix write time activation failure for metadata block group
In reply to Naohiro Aotafs/btrfs/zoned.c bool btrfs_zone_activate(struct btrfs_block_group *block_group) · line 2173
Quoted 3 lines · Naohiro Aota
fs/btrfs/zoned.c +3
-0+if (block_group->flags &BTRFS_BLOCK_GROUP_DATA ) { +/* The caller should check if the block group is full. */ +if (WARN_ON_ONCE (btrfs_zoned_bg_is_full (block_group))) {
I get that WARN_ON() triggering when running fio benchmarks:
[ 2054.634831] RIP: 0010:btrfs_zone_activate+0x1e0/0x220 [btrfs] [ 2054.660477] RSP: 0018:ffffd20f0bd039b8 EFLAGS: 00010246 [ 2054.666069] RAX: 0000000010000000 RBX: ffff8b7d98747c00 RCX: ffff8b7d4729f508 [ 2054.673576] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff8b7d98747c10 [ 2054.681069] RBP: 0000000000000000 R08: ffff8b7d5c13b000 R09: 0000000000000000 [ 2054.688563] R10: ffff8b9c88239480 R11: 0000000000000001 R12: ffff8b7d5c13b000 [ 2054.696050] R13: ffff8b7d5315b000 R14: ffff8b7d47389400 R15: 0000000000001000 [ 2054.703546] FS: 00007ff45a4b2840(0000) GS:ffff8b9d02a7e000(0000) knlGS:0000000000000000 [ 2054.711989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2054.718090] CR2: 000056051a88df6c CR3: 000000034dbc0000 CR4: 0000000000350ef0 [ 2054.725581] Call Trace: [ 2054.728392] <TASK> [ 2054.730854] ? srso_return_thunk+0x5/0x5f [ 2054.735233] ? btrfs_zoned_reserve_data_reloc_bg+0x77/0x230 [btrfs] [ 2054.741915] open_ctree+0x7e9/0xb40 [btrfs]
But before that I see write time errors:
[ 203.230377] BTRFS info (device sdh): host-managed zoned block device /dev/sdh, 111760 zones of 268435456 bytes [ 203.240465] BTRFS info (device sdh): zoned mode enabled with zone size 268435456 [ 203.251064] BTRFS info (device sdh): checking UUID tree [ 1958.689947] sdh: zone 1324: Aborting plugged BIOs [ 1958.694697] BTRFS error (device sdh): bdev /dev/sdh errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 [ 1958.703428] BTRFS error (device sdh): bdev /dev/sdh errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 [ 1958.712157] BTRFS error (device sdh): bdev /dev/sdh errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 [ 1958.720881] BTRFS error (device sdh): bdev /dev/sdh errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 [ 1958.729614] BTRFS error (device sdh): bdev /dev/sdh errs: wr 5, rd 0, flush 0, corrupt 0, gen 0 [ 1958.738345] BTRFS error (device sdh): bdev /dev/sdh errs: wr 6, rd 0, flush 0, corrupt 0, gen 0 [ 1958.747072] BTRFS error (device sdh): bdev /dev/sdh errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 [ 1958.755798] BTRFS error (device sdh): bdev /dev/sdh errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 [ 1958.764525] BTRFS error (device sdh): bdev /dev/sdh errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 [ 1958.773257] BTRFS error (device sdh): bdev /dev/sdh errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 [ 1959.306908] BTRFS error (device sdh state A): Transaction aborted (error -5) [ 1959.314030] BTRFS: error (device sdh state A) in __btrfs_update_delayed_inode:1015: errno=-5 IO failure [ 1959.323481] BTRFS info (device sdh state EA): forced readonly [ 1959.536934] sdh: zone 1325: Aborting plugged BIOs
This is with today's for-next and the two patch sets of you applied. Will dig deeper.
Re: [PATCH v2 2/3] btrfs: zoned: fix write time activation failure for metadata block group
Quoted 46 lines · Johannes Thumshirn
fs/btrfs/zoned.c bool btrfs_zone_activate(struct btrfs_block_group *block_group) · line 2173
Quoted 3 lines · Naohiro Aota
fs/btrfs/zoned.c +3
-0+if (block_group->flags &BTRFS_BLOCK_GROUP_DATA ) { +/* The caller should check if the block group is full. */ +if (WARN_ON_ONCE (btrfs_zoned_bg_is_full (block_group))) {I get that
WARN_ON()triggering when running fio benchmarks:[ 2054.634831] RIP: 0010:btrfs_zone_activate+0x1e0/0x220 [btrfs] [ 2054.660477] RSP: 0018:ffffd20f0bd039b8 EFLAGS: 00010246 [ 2054.666069] RAX: 0000000010000000 RBX: ffff8b7d98747c00 RCX: ffff8b7d4729f508 [ 2054.673576] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff8b7d98747c10 [ 2054.681069] RBP: 0000000000000000 R08: ffff8b7d5c13b000 R09: 0000000000000000 [ 2054.688563] R10: ffff8b9c88239480 R11: 0000000000000001 R12: ffff8b7d5c13b000 [ 2054.696050] R13: ffff8b7d5315b000 R14: ffff8b7d47389400 R15: 0000000000001000 [ 2054.703546] FS: 00007ff45a4b2840(0000) GS:ffff8b9d02a7e000(0000) knlGS:0000000000000000 [ 2054.711989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2054.718090] CR2: 000056051a88df6c CR3: 000000034dbc0000 CR4: 0000000000350ef0 [ 2054.725581] Call Trace: [ 2054.728392] <TASK> [ 2054.730854] ? srso_return_thunk+0x5/0x5f [ 2054.735233] ? btrfs_zoned_reserve_data_reloc_bg+0x77/0x230 [btrfs] [ 2054.741915] open_ctree+0x7e9/0xb40 [btrfs]But before that I see write time errors:
[ 203.230377] BTRFS info (device sdh): host-managed zoned block device /dev/sdh, 111760 zones of 268435456 bytes [ 203.240465] BTRFS info (device sdh): zoned mode enabled with zone size 268435456 [ 203.251064] BTRFS info (device sdh): checking UUID tree [ 1958.689947] sdh: zone 1324: Aborting plugged BIOs [ 1958.694697] BTRFS error (device sdh): bdev /dev/sdh errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 [ 1958.703428] BTRFS error (device sdh): bdev /dev/sdh errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 [ 1958.712157] BTRFS error (device sdh): bdev /dev/sdh errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 [ 1958.720881] BTRFS error (device sdh): bdev /dev/sdh errs: wr 4, rd 0, flush 0, corrupt 0, gen 0 [ 1958.729614] BTRFS error (device sdh): bdev /dev/sdh errs: wr 5, rd 0, flush 0, corrupt 0, gen 0 [ 1958.738345] BTRFS error (device sdh): bdev /dev/sdh errs: wr 6, rd 0, flush 0, corrupt 0, gen 0 [ 1958.747072] BTRFS error (device sdh): bdev /dev/sdh errs: wr 7, rd 0, flush 0, corrupt 0, gen 0 [ 1958.755798] BTRFS error (device sdh): bdev /dev/sdh errs: wr 8, rd 0, flush 0, corrupt 0, gen 0 [ 1958.764525] BTRFS error (device sdh): bdev /dev/sdh errs: wr 9, rd 0, flush 0, corrupt 0, gen 0 [ 1958.773257] BTRFS error (device sdh): bdev /dev/sdh errs: wr 10, rd 0, flush 0, corrupt 0, gen 0 [ 1959.306908] BTRFS error (device sdh state A): Transaction aborted (error -5) [ 1959.314030] BTRFS: error (device sdh state A) in __btrfs_update_delayed_inode:1015: errno=-5 IO failure [ 1959.323481] BTRFS info (device sdh state EA): forced readonly [ 1959.536934] sdh: zone 1325: Aborting plugged BIOs
Quoted 1 line
This is with today's for-next and the two patch sets of you applied. Will dig deeper.
Ah, yes. I know this issue. This happens because btrfs_zoned_reserve_data_reloc_bg() uses "if (bg->used > 0)" to choose (skip) a block group to be reserved. If a block group is fully allocated and get unused, bg->used == 0 && bg->alloc_offset != 0. Since, reserving this block group itself is an issue. I should have included the fix for this. I'll update this series to include the fix.
PATCH v2 3/3 btrfs: zoned: limit active zones to max_open_zones
In reply to Naohiro AotaWhen there is no active zone limit, we can technically write into any number of zones at the same time. However, exceeding the max open zones can degrade performance. To prevent this, set the max_active_zones to bdev_max_open_zones() if there is no active zone limit.
If the device has no limit, use a reasonable default value BTRFS_DEFAULT_MAX_ACTIVE_ZONES to limit the open zones.
--- fs/btrfs/zoned.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
fs/btrfs/zoned.c +7 -1
@@ -42,6 +42,9 @@/* Number of superblock log zones */ #define BTRFS_NR_SB_LOG_ZONES 2 +/* Default number of max active zones when the device has no limits. */ +#define BTRFS_DEFAULT_MAX_ACTIVE_ZONES 128 +/* * Minimum of active zones we need: *
@@ -416,7 +419,10 @@ int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache)if (!IS_ALIGNED (nr_sectors, zone_sectors)) zone_info->nr_zones++; - max_active_zones =bdev_max_active_zones (bdev); + max_active_zones =min_not_zero (bdev_max_active_zones (bdev), +bdev_max_open_zones (bdev)); +if (!max_active_zones && zone_info->nr_zones >BTRFS_DEFAULT_MAX_ACTIVE_ZONES ) + max_active_zones =BTRFS_DEFAULT_MAX_ACTIVE_ZONES ;if (max_active_zones && max_active_zones <BTRFS_MIN_ACTIVE_ZONES ) { btrfs_err(fs_info, "zoned: %s: max active zones %u is too small, need at least %u active zones",
Signature · 1 line
2.50.0
Re: [PATCH v2 3/3] btrfs: zoned: limit active zones to max_open_zones
fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache) · line 426
Quoted 6 lines · Naohiro Aota
fs/btrfs/zoned.c +4
-1- max_active_zones =bdev_max_active_zones (bdev); + max_active_zones =min_not_zero (bdev_max_active_zones (bdev), +bdev_max_open_zones (bdev)); +if (!max_active_zones && zone_info->nr_zones >BTRFS_DEFAULT_MAX_ACTIVE_ZONES ) + max_active_zones =BTRFS_DEFAULT_MAX_ACTIVE_ZONES ;if (max_active_zones && max_active_zones <BTRFS_MIN_ACTIVE_ZONES ) {
Can max_active_zones be 0 here?
Isn't it either 'bdev_max_active_zones(bdev)', 'bdev_max_open_zones(bdev)' or BTRFS_DEFAULT_MAX_ACTIVE_ZONES?
If min_not_zero() returns 0 AND zone_info->nr_zones <= BTRFS_DEFAULT_MAX_ACTIVE_ZONES we can end up in the case where max_active_zones is still 0.
Say we have a hypothetical device with MAZ = 0 and MOZ = 0 but nr_zones = 120, then we get max_active_zones = 0 and then do:
zone_info->max_active_zones = max_active_zones;
Or am I talking BS now?
Johannes Thumshirn
Reviewed-byRe: [PATCH v2 3/3] btrfs: zoned: limit active zones to max_open_zones
Quoted 18 lines · Johannes Thumshirn
fs/btrfs/zoned.c int btrfs_get_dev_zone_info(struct btrfs_device *device, bool populate_cache) · line 426
Quoted 6 lines · Naohiro Aota
fs/btrfs/zoned.c +4
-1- max_active_zones =bdev_max_active_zones (bdev); + max_active_zones =min_not_zero (bdev_max_active_zones (bdev), +bdev_max_open_zones (bdev)); +if (!max_active_zones && zone_info->nr_zones >BTRFS_DEFAULT_MAX_ACTIVE_ZONES ) + max_active_zones =BTRFS_DEFAULT_MAX_ACTIVE_ZONES ;if (max_active_zones && max_active_zones <BTRFS_MIN_ACTIVE_ZONES ) {Can
max_active_zonesbe 0 here?Isn't it either '
bdev_max_active_zones(bdev)', 'bdev_max_open_zones(bdev)' orBTRFS_DEFAULT_MAX_ACTIVE_ZONES?If
min_not_zero()returns 0 ANDzone_info->nr_zones<=BTRFS_DEFAULT_MAX_ACTIVE_ZONESwe can end up in the case wheremax_active_zonesis still 0.
Quoted 7 lines
Say we have a hypothetical device with MAZ = 0 and MOZ = 0 but
nr_zones= 120, then we getmax_active_zones= 0 and then do:
zone_info->max_active_zones=max_active_zones;Or am I talking BS now?
Forget about it, we don't care if we have a MAZ of 0.