btrfs: zoned: fix active-zone accounting and transitions
cover btrfs: zoned: fix active-zone accounting and transitions
After systems using HC620 host-managed zoned devices were upgraded to Linux 6.18, multiple users reported transaction writeback failures with -EAGAIN that forced filesystems read-only, as well as hung tasks.
Enforcing the device's max_active_zones=128 limit exposed existing active-zone handling problems. We reproduced representative cases locally, including an -EAGAIN transaction abort that forced the filesystem read-only and a balance self-deadlock that stalled transaction commit and data writers.
Further investigation traced these failures to several related gaps in Btrfs active-zone handling around mount recovery, non-data block group activation and zone finishing:
- Conventional zones participate in runtime active-zone accounting, but that state cannot be reconstructed from device zone conditions at mount.
- Active metadata and system block groups are reconstructed at mount, but their runtime write roles are not.
- An obsolete allocation-time activation helper bypasses the write-time role pivot and can recursively enter zone finishing.
- do_zone_finish() clears the active state before the device operation has released the active-zone slot.
Together, these bugs can leave no active-zone slot available to metadata or system writeback, or recursively enter zone finishing while holding ro_block_group_mutex, resulting in the two failure modes described above.
Fix the active-zone model and its state transitions as follows:
1. Track only sequential zones as active.
2. Recover the normal metadata and system write roles on mount, and finish extra active block groups left by the old behavior.
3. Remove the obsolete allocation-time non-data activation helper so metadata and system activation goes through the write-time role pivot.
4. Serialize concurrent finish operations on the same block group and keep it active until the device operation and software cleanup complete.
The individual commits contain the corresponding reproduction conditions, kernel stacks and state-transition traces.
Dongjiang Zhu (4):
btrfs: zoned: track only sequential zones as active
btrfs: zoned: recover active non-data block group roles on mount
btrfs: zoned: remove obsolete non-data block group activation helper
btrfs: zoned: serialize zone finishing per block group
fs/btrfs/block-group.c | 38 +++----- fs/btrfs/block-group.h | 7 +- fs/btrfs/disk-io.c | 7 +- fs/btrfs/fs.h | 4 + fs/btrfs/zoned.c | 193 ++++++++++++++++++++++++++++------------- fs/btrfs/zoned.h | 9 +- 6 files changed, 160 insertions(+), 98 deletions(-)
Signature · 1 line
2.39.5
PATCH 1/4 btrfs: zoned: track only sequential zones as active
[CAUSE]
The runtime activation and finish paths apply active-zone accounting to both conventional and sequential stripes.
At mount, however, active_zones_left is rebuilt from device zone conditions. Conventional zones report NOT_WP and are therefore not included. Meanwhile, all-conventional block groups are marked active and added to zone_active_bgs, so mount recovery still decreases reserved_active_zones for their metadata and system stripes.
The two counters can therefore diverge across a remount.
[BUG]
Each unmatched reservation decrement lets data consume one more sequential active-zone slot. Once the reservation becomes negative, data can exhaust all such slots.
A later transition from a conventional metadata or system target to a sequential one can then fail, because finishing the conventional target does not release a sequential active-zone slot.
If this happens during synchronous transaction writeback, btrfs_check_meta_write_pointer() returns -EAGAIN. The transaction is then aborted and the filesystem is forced read-only.
The reservation mismatch was reproduced by creating small files with large xattrs in batches, syncing them and remounting an HC620 filesystem using SINGLE profiles:
active_zones_left: 126
reserved_active_zones: 3 -> -8
The active list contained one partially used conventional system block group, nine full conventional metadata block groups and one partially used conventional metadata block group.
[FIX]
Use sequential stripes as the unit of active-zone tracking.
Do not mark all-conventional block groups active. Only sequential stripes update the active-zone bitmap and non-data reservation during activation, finish and mount recovery.
--- fs/btrfs/block-group.h | 6 +++++- fs/btrfs/zoned.c | 48 +++++++++++++++++++++++++++++------------- 2 files changed, 38 insertions(+), 16 deletions(-)
fs/btrfs/block-group.h +5 -1
@@ -80,13 +80,17 @@ enum btrfs_block_group_flags {BLOCK_GROUP_FLAG_TO_COPY ,BLOCK_GROUP_FLAG_RELOCATING_REPAIR ,BLOCK_GROUP_FLAG_CHUNK_ITEM_INSERTED , +/* +* Only block groups containing sequential zones can have this bit set; +* conventional-only block groups never do. +*/ BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE ,BLOCK_GROUP_FLAG_ZONED_DATA_RELOC ,/* Does the block group need to be added to the free space tree? */ BLOCK_GROUP_FLAG_NEEDS_FREE_SPACE ,/* Set after we add a new block group to the free space tree. */ BLOCK_GROUP_FLAG_FREE_SPACE_ADDED , -/* Indicate that the block group is placed on a sequential zone */ +/* Indicate that the block group contains at least one sequential zone. */ BLOCK_GROUP_FLAG_SEQUENTIAL_ZONE ,/* * Indicate that block group is in the list of new block groups of a
fs/btrfs/zoned.c +33 -15
98 lines not on this page · raw message
Signature · 1 line
2.39.5
Re: [PATCH 1/4] btrfs: zoned: track only sequential zones as active
Quoted 39 lines · Dongjiang Zhu
[CAUSE]
The runtime activation and finish paths apply active-zone accounting to both conventional and sequential stripes.At mount, however,
active_zones_leftis rebuilt from device zone conditions. Conventional zones reportNOT_WPand are therefore not included. Meanwhile, all-conventional block groups are marked active and added tozone_active_bgs, so mount recovery still decreasesreserved_active_zonesfor their metadata and system stripes.The two counters can therefore diverge across a remount.
[BUG]
Each unmatched reservation decrement lets data consume one more sequential active-zone slot. Once the reservation becomes negative, data can exhaust all such slots.A later transition from a conventional metadata or system target to a sequential one can then fail, because finishing the conventional target does not release a sequential active-zone slot.
If this happens during synchronous transaction writeback,
btrfs_check_meta_write_pointer()returns-EAGAIN. The transaction is then aborted and the filesystem is forced read-only.The reservation mismatch was reproduced by creating small files with large xattrs in batches, syncing them and remounting an HC620 filesystem using SINGLE profiles:
active_zones_left: 126
reserved_active_zones: 3 -> -8The active list contained one partially used conventional system block group, nine full conventional metadata block groups and one partially used conventional metadata block group.
[FIX]
Use sequential stripes as the unit of active-zone tracking.
Quoted 6 lines
Do not mark all-conventional block groups active. Only sequential stripes update the active-zone bitmap and non-data reservation during activation, finish and mount recovery.
Assisted-by LLMSigned-off-by Dongjiang Zhu <[email protected]>
Looks good, two small nits though in case you need to resend the series:
1) Please don't do the [CAUSE], [BUG], [FIX] thingy, I know Qu does it but noone else does that.
2) We now have a repeated pattern of:
if (!btrfs_dev_is_sequential())
return
if (max_active_zones)
return
Maybe have a helper for that? Something like
static inline bool btrfs_needs_active_zone_tracking(struct btrfs_device *dev,
u64 physical)
{
if (!btrfs_dev_is_sequential(dev, physical))
return false;
if (dev->zone_info->max_active_zones == 0)
return false;
return true;
}Otherwise looks good,
Re: [PATCH 1/4] btrfs: zoned: track only sequential zones as active
Quoted 44 lines · Johannes Thumshirn
Quoted 43 lines · Dongjiang Zhu
[CAUSE]
The runtime activation and finish paths apply active-zone accounting to both conventional and sequential stripes.At mount, however,
active_zones_leftis rebuilt from device zone conditions. Conventional zones reportNOT_WPand are therefore not included. Meanwhile, all-conventional block groups are marked active and added tozone_active_bgs, so mount recovery still decreasesreserved_active_zonesfor their metadata and system stripes.The two counters can therefore diverge across a remount.
[BUG]
Each unmatched reservation decrement lets data consume one more sequential active-zone slot. Once the reservation becomes negative, data can exhaust all such slots.A later transition from a conventional metadata or system target to a sequential one can then fail, because finishing the conventional target does not release a sequential active-zone slot.
If this happens during synchronous transaction writeback,
btrfs_check_meta_write_pointer()returns-EAGAIN. The transaction is then aborted and the filesystem is forced read-only.The reservation mismatch was reproduced by creating small files with large xattrs in batches, syncing them and remounting an HC620 filesystem using SINGLE profiles:
active_zones_left: 126 reserved_active_zones: 3 -> -8The active list contained one partially used conventional system block group, nine full conventional metadata block groups and one partially used conventional metadata block group.
[FIX]
Use sequential stripes as the unit of active-zone tracking.Do not mark all-conventional block groups active. Only sequential stripes update the active-zone bitmap and non-data reservation during activation, finish and mount recovery.
Quoted 6 lines
Quoted 2 lines
Assisted-by LLMSigned-off-by Dongjiang Zhu <[email protected]>Looks good, two small nits though in case you need to resend the series:
1) Please don't do the [CAUSE], [BUG], [FIX] thingy, I know Qu does it but noone else does that.
Sorry, I thought that was a good way to explain a bug, but this doesn't look like so for everyone else.
I'll also change the future style.
Thanks,
Qu
Quoted 24 lines
2) We now have a repeated pattern of:
if (!btrfs_dev_is_sequential()) return if (max_active_zones) returnMaybe have a helper for that? Something like
static inline bool btrfs_needs_active_zone_tracking(struct
btrfs_device*dev,
u64 physical){ if (!btrfs_dev_is_sequential(dev, physical)) return false; if (dev->zone_info->max_active_zones == 0) return false; return true; }Otherwise looks good,
Reviewed-by Johannes Thumshirn <[email protected]>
Re: [PATCH 1/4] btrfs: zoned: track only sequential zones as active
Quoted 7 lines · Qu Wenruo
Quoted 3 lines
Looks good, two small nits though in case you need to resend the series:
1) Please don't do the [CAUSE], [BUG], [FIX] thingy, I know Qu does it but noone else does that.Sorry, I thought that was a good way to explain a bug, but this doesn't look like so for everyone else.
I'll also change the future style.
Nah it's kind of your signature, like me writing Byte instead of Bye.
Re: [PATCH 1/4] btrfs: zoned: track only sequential zones as active
In reply to Johannes ThumshirnQuoted 44 lines · Johannes Thumshirn
Quoted 43 lines · Dongjiang Zhu
[CAUSE]
The runtime activation and finish paths apply active-zone accounting to both conventional and sequential stripes.At mount, however,
active_zones_leftis rebuilt from device zone conditions. Conventional zones reportNOT_WPand are therefore not included. Meanwhile, all-conventional block groups are marked active and added tozone_active_bgs, so mount recovery still decreasesreserved_active_zonesfor their metadata and system stripes.The two counters can therefore diverge across a remount.
[BUG]
Each unmatched reservation decrement lets data consume one more sequential active-zone slot. Once the reservation becomes negative, data can exhaust all such slots.A later transition from a conventional metadata or system target to a sequential one can then fail, because finishing the conventional target does not release a sequential active-zone slot.
If this happens during synchronous transaction writeback,
btrfs_check_meta_write_pointer()returns-EAGAIN. The transaction is then aborted and the filesystem is forced read-only.The reservation mismatch was reproduced by creating small files with large xattrs in batches, syncing them and remounting an HC620 filesystem using SINGLE profiles:
active_zones_left: 126 reserved_active_zones: 3 -> -8The active list contained one partially used conventional system block group, nine full conventional metadata block groups and one partially used conventional metadata block group.
[FIX]
Use sequential stripes as the unit of active-zone tracking.Do not mark all-conventional block groups active. Only sequential stripes update the active-zone bitmap and non-data reservation during activation, finish and mount recovery.
Quoted 5 lines
Quoted 2 lines
Assisted-by LLMSigned-off-by Dongjiang Zhu<[email protected]>Looks good, two small nits though in case you need to resend the series:
1) Please don't do the [CAUSE], [BUG], [FIX] thingy, I know Qu does it but noone else does that.
Sure, I will remove these headings from the commit messages in v2.
Quoted 11 lines
2) We now have a repeated pattern of:
if (!btrfs_dev_is_sequential()) return if (max_active_zones) returnMaybe have a helper for that? Something like
static inline bool btrfs_needs_active_zone_tracking(struct
btrfs_device*dev,
u64 physical)
Quoted 10 lines
{
if (!btrfs_dev_is_sequential(dev, physical))
return false;
if (dev->zone_info->max_active_zones == 0)
return false;
return true;
}Good idea. I will add the helper and use it for the repeated checks in v2.
Thanks for the review.
Dongjiang Zhu
Quoted 3 lines
Otherwise looks good,
Reviewed-by Johannes Thumshirn<[email protected]>
PATCH 2/4 btrfs: zoned: recover active non-data block group roles on mount
In reply to Dongjiang Zhu[BUG]
After a remount, a later write can activate a new metadata or system block group without finishing the recovered one. Repeating this across remounts consumes additional active-zone slots and reservations, and can eventually exhaust all active-zone slots.
Once no slot is available, activation of a metadata or system block group fails. During synchronous transaction writeback this propagates as -EAGAIN, aborting the transaction and forcing the filesystem read-only.
A remount/fsync/remount reproducer on an HC620 left two partially used sequential metadata block groups active while active_meta_bg remained NULL:
bg=343865819136 alloc=176095232 cap=268435456
bg=344402690048 alloc=3260416 cap=268435456
[CAUSE]active_meta_bg and active_system_bg assign active block groups to the normal metadata and system write roles. These runtime-only pointers are reset to NULL on every mount.
Mount reconstructs partially used sequential block groups in zone_active_bgs and adjusts their reservations, but does not recover the role assignments.
The write-time pivot can finish the previous block group only when the corresponding role pointer is set. If it is NULL, activation proceeds without releasing an already active block group of the same role.
[FIX]
Recover one active block group for the normal metadata role and one for the system role from zone_active_bgs. For each role, keep the block group with the largest writable tail and finish the others.
The tree-log block group role is runtime-only and cannot be recovered after a remount. Use the selected metadata block group for the normal metadata role and leave one metadata reservation for a new tree-log block group.
Run the recovery on every mount. ZONE_FINISH preserves existing extents but abandons the unused tail, so account that tail as zone unusable immediately.
--- fs/btrfs/disk-io.c | 7 +++- fs/btrfs/fs.h | 4 +++ fs/btrfs/zoned.c | 90 ++++++++++++++++++++++++++++++++++++++++++---- fs/btrfs/zoned.h | 7 ++-- 4 files changed, 98 insertions(+), 10 deletions(-)
fs/btrfs/disk-io.c +6 -1
@@ -3702,7 +3702,12 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_devicebtrfs_free_zone_cache (fs_info); -btrfs_check_active_zone_reservation (fs_info); + ret =btrfs_restore_active_nondata_bgs (fs_info); +if (ret) { +btrfs_err (fs_info,"failed to restore active non-data block groups: %pe" , +ERR_PTR (ret)); + goto fail_sysfs; + }if (!sb_rdonly (sb) && fs_info->fs_devices->missing_devices && !btrfs_check_rw_degradable (fs_info,NULL )) {
fs/btrfs/fs.h +4 -0
@@ -931,6 +931,10 @@ struct btrfs_fs_info {u64 data_reloc_bg;struct mutex zoned_data_reloc_io_lock; +/* +* Active block groups for normal metadata and system writes. Mount +* recovery restores one block group for each role and finishes any extras. +*/ struct btrfs_block_group *active_meta_bg;struct btrfs_block_group *active_system_bg;
fs/btrfs/zoned.c +83 -7
126 lines not on this page · raw message
fs/btrfs/zoned.h +5 -2
@@ -95,7 +95,7 @@ void btrfs_zoned_release_data_reloc_bg(struct btrfs_fs_info *fs_info, u64 logica u64 length);int btrfs_zone_finish_one_bg (struct btrfs_fs_info *fs_info);int btrfs_zoned_activate_one_bg (struct btrfs_space_info *space_info,bool do_finish); -void btrfs_check_active_zone_reservation (struct btrfs_fs_info *fs_info); +int btrfs_restore_active_nondata_bgs (struct btrfs_fs_info *fs_info);int btrfs_reset_unused_block_groups (struct btrfs_space_info *space_info,u64 num_bytes);void btrfs_show_zoned_stats (struct btrfs_fs_info *fs_info,struct seq_file *seq);
@@ -279,7 +279,10 @@ static inline int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_infreturn 0 ; } -static inline void btrfs_check_active_zone_reservation (struct btrfs_fs_info *fs_info) { } +static inline int btrfs_restore_active_nondata_bgs (struct btrfs_fs_info *fs_info) +{ +return 0 ; +}static inline int btrfs_reset_unused_block_groups (struct btrfs_space_info *space_info,u64 num_bytes)
Signature · 1 line
2.39.5
Re: [PATCH 2/4] btrfs: zoned: recover active non-data block group roles on mount
Quoted 10 lines · Dongjiang Zhu
[BUG]
After a remount, a later write can activate a new metadata or system block group without finishing the recovered one. Repeating this across remounts consumes additional active-zone slots and reservations, and can eventually exhaust all active-zone slots.Once no slot is available, activation of a metadata or system block group fails. During synchronous transaction writeback this propagates as
-EAGAIN, aborting the transaction and forcing the filesystem read-only.
Quoted 7 lines
A remount/fsync/remount reproducer on an HC620 left two partially used sequential metadata block groups active while
active_meta_bgremained NULL:bg=343865819136 alloc=176095232 cap=268435456
bg=344402690048 alloc=3260416 cap=268435456
Can you submit a fstest for that please? Maybe even in generic/ so XFS and F2FS also get coverage?
fs/btrfs/fs.h struct btrfs_fs_info { · line 937
Quoted 12 lines
fs/btrfs/fs.h +4
-0@@ -931,6 +931,10 @@ struct btrfs_fs_info {u64 data_reloc_bg;struct mutex zoned_data_reloc_io_lock; +/* +* Active block groups for normal metadata and system writes. Mount +* recovery restores one block group for each role and finishes any extras. +*/
I don't think this comments adds any value.
Quoted 20 lines
fs/btrfs/fs.h +0
-0struct btrfs_block_group *active_meta_bg;struct btrfs_block_group *active_system_bg;fs/btrfs/zoned.c +9
-0@@ -3050,22 +3050,73 @@ int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_info, bool do_finreturn 0 ; } +static int finish_extra_active_nondata_bgs (struct btrfs_fs_info *fs_info) +{ +struct btrfs_block_group *block_group; +struct btrfs_block_group *next; +u64 tail_unusable; +int ret; + +list_for_each_entry_safe (block_group, next, &fs_info->zone_active_bgs, + active_bg_list) {
fs/btrfs/zoned.c int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_info, bool do_fin · line 3071
Quoted 10 lines
fs/btrfs/zoned.c +10
-0+if (!(block_group->flags & + (BTRFS_BLOCK_GROUP_METADATA |BTRFS_BLOCK_GROUP_SYSTEM ))) +continue ; + +if (block_group == fs_info->active_meta_bg || + block_group == fs_info->active_system_bg) +continue ; + +btrfs_get_block_group (block_group); + tail_unusable = block_group->zone_capacity - block_group->alloc_offset;
Shouldn't these be accessed under the block_group->lock?
fs/btrfs/zoned.c void btrfs_check_active_zone_reservation(struct btrfs_fs_info *fs_info) · line 3162
Quoted 13 lines
fs/btrfs/zoned.c +10
-2-if (!(block_group->flags & - (BTRFS_BLOCK_GROUP_METADATA |BTRFS_BLOCK_GROUP_SYSTEM ))) +if (block_group->flags &BTRFS_BLOCK_GROUP_METADATA ) { + active_bg = &active_meta_bg; + active_avail = &active_meta_avail; + }else if (block_group->flags &BTRFS_BLOCK_GROUP_SYSTEM ) { + active_bg = &active_system_bg; + active_avail = &active_system_avail; + }else {continue ; + } + + avail = block_group->zone_capacity - block_group->alloc_offset;
Same here, don't you need the block_group->lock?
Re: [PATCH 2/4] btrfs: zoned: recover active non-data block group roles on mount
Quoted 15 lines · Johannes Thumshirn
Quoted 14 lines · Dongjiang Zhu
[BUG]
After a remount, a later write can activate a new metadata or system block group without finishing the recovered one. Repeating this across remounts consumes additional active-zone slots and reservations, and can eventually exhaust all active-zone slots.Once no slot is available, activation of a metadata or system block group fails. During synchronous transaction writeback this propagates as
-EAGAIN, aborting the transaction and forcing the filesystem read-only.A remount/fsync/remount reproducer on an HC620 left two partially used sequential metadata block groups active while
active_meta_bgremained NULL:
Quoted 7 lines
Quoted 3 lines
bg=343865819136 alloc=176095232 cap=268435456 bg=344402690048 alloc=3260416 cap=268435456Can you submit a fstest for that please? Maybe even in generic/ so XFS and F2FS also get coverage?
Yes, I can work on a separate fstest. To make sure I understand the suggestion correctly, do you have in mind a zoned-device test that creates a metadata-heavy workload, syncs, fully unmounts and mounts the filesystem repeatedly, and then verifies that further writes and syncs still succeed?
Quoted 8 lines
Quoted 8 lines
fs/btrfs/fs.h +0
-0@@ -931,6 +931,10 @@ struct btrfs_fs_info {u64 data_reloc_bg;struct mutex zoned_data_reloc_io_lock;
Quoted 7 lines
fs/btrfs/fs.h struct btrfs_fs_info { · line 937
Quoted 4 lines
fs/btrfs/fs.h +4
-0+/* +* Active block groups for normal metadata and system writes. Mount +* recovery restores one block group for each role and finishes any extras. +*/ I don't think this comments adds any value.
Agreed, I will remove it.
Quoted 33 lines
Quoted 20 lines
fs/btrfs/fs.h +0
-0struct btrfs_block_group *active_meta_bg;struct btrfs_block_group *active_system_bg;fs/btrfs/zoned.c +9
-0@@ -3050,22 +3050,73 @@ int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_info, bool do_finreturn 0 ; } +static int finish_extra_active_nondata_bgs (struct btrfs_fs_info *fs_info) +{ +struct btrfs_block_group *block_group; +struct btrfs_block_group *next; +u64 tail_unusable; +int ret; + +list_for_each_entry_safe (block_group, next, &fs_info->zone_active_bgs, + active_bg_list) {fs/btrfs/zoned.c int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_info, bool do_fin · line 3071
Quoted 10 lines
fs/btrfs/zoned.c +10
-0+if (!(block_group->flags & + (BTRFS_BLOCK_GROUP_METADATA |BTRFS_BLOCK_GROUP_SYSTEM ))) +continue ; + +if (block_group == fs_info->active_meta_bg || + block_group == fs_info->active_system_bg) +continue ; + +btrfs_get_block_group (block_group); + tail_unusable = block_group->zone_capacity - block_group->alloc_offset;Shouldn't these be accessed under the
block_group->lock?
Quoted 3 lines
[...]
Same here, don't you need the
block_group->lock?
Both accesses only happen during open_ctree(), after all block groups have been loaded and before the background threads are started. Following other mount-time initialization code, I assumed that block_group->lock was not necessary because the allocation state cannot change concurrently.
On the other hand, taking the lock would make the usual protection of alloc_offset explicit and avoid relying on the mount-time context for future maintenance. Would you prefer that I add the locking around both reads in v2?
Thanks,
Dongjiang Zhu
Re: [PATCH 2/4] btrfs: zoned: recover active non-data block group roles on mount
Quoted 9 lines · Dongjiang Zhu
Quoted 3 lines · Johannes Thumshirn
Can you submit a fstest for that please? Maybe even in generic/ so XFS and F2FS also get coverage?
Yes, I can work on a separate fstest. To make sure I understand the suggestion correctly, do you have in mind a zoned-device test that creates a metadata-heavy workload, syncs, fully unmounts and mounts the filesystem repeatedly, and then verifies that further writes and syncs still succeed?
Yep.
Quoted 38 lines
Quoted 30 lines
Quoted 18 lines
fs/btrfs/zoned.c +9
-0@@ -3050,22 +3050,73 @@ int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_info, bool do_finreturn 0 ; } +static int finish_extra_active_nondata_bgs (struct btrfs_fs_info *fs_info) +{ +struct btrfs_block_group *block_group; +struct btrfs_block_group *next; +u64 tail_unusable; +int ret; + +list_for_each_entry_safe (block_group, next, &fs_info->zone_active_bgs, + active_bg_list) {fs/btrfs/zoned.c int btrfs_zoned_activate_one_bg(struct btrfs_space_info *space_info, bool do_fin · line 3071
Quoted 10 lines
fs/btrfs/zoned.c +10
-0+if (!(block_group->flags & + (BTRFS_BLOCK_GROUP_METADATA |BTRFS_BLOCK_GROUP_SYSTEM ))) +continue ; + +if (block_group == fs_info->active_meta_bg || + block_group == fs_info->active_system_bg) +continue ; + +btrfs_get_block_group (block_group); + tail_unusable = block_group->zone_capacity - block_group->alloc_offset;Shouldn't these be accessed under the
block_group->lock?Quoted 3 lines
[...]
Same here, don't you need the
block_group->lock?Both accesses only happen during
open_ctree(), after all block groups have been loaded and before the background threads are started. Following other mount-time initialization code, I assumed thatblock_group->lockwas not necessary because the allocation state cannot change concurrently.
Quoted 4 lines
On the other hand, taking the lock would make the usual protection of
alloc_offsetexplicit and avoid relying on the mount-time context for future maintenance. Would you prefer that I add the locking around both reads in v2?
Yes please.
PATCH 3/4 btrfs: zoned: remove obsolete non-data block group activation helper
In reply to Dongjiang Zhu[BUG]
On an HC620 with max_active_zones=128, file writes occupied all 128 active-zone slots. With four direct-I/O writers still running, starting a balance with
btrfs balance start -f -dusage=0 -musage=0 -susage=100 <mountpoint>
left the balance task blocked in D state. SysRq-w reported:
task:btrfs state:D stack:0 pid:8145
Call Trace: <TASK> __schedule+0x50e/0xfe0 schedule+0x22/0xa0 schedule_preempt_disabled+0x10/0x20 __mutex_lock.constprop.0+0x380/0x920 btrfs_inc_block_group_ro+0x6a/0x270 [btrfs] do_zone_finish+0x2b6/0x4c0 [btrfs] btrfs_zone_finish_one_bg+0x184/0x1f0 [btrfs] btrfs_zoned_activate_one_bg+0x189/0x1d0 [btrfs] reserve_chunk_space+0xdb/0x180 [btrfs] btrfs_inc_block_group_ro+0x252/0x270 [btrfs] btrfs_relocate_block_group+0xc2/0x2260 [btrfs] btrfs_relocate_chunk+0x39/0x1c0 [btrfs] btrfs_balance+0x911/0x1810 [btrfs] </TASK>
The blocked balance task also stalls transaction commit and data writers, so the filesystem can no longer make write progress or unmount cleanly.
[CAUSE]
The outer btrfs_inc_block_group_ro() holds ro_block_group_mutex while reserve_chunk_space() tries to activate a system block group. When no active-zone slot is available, btrfs_zoned_activate_one_bg() starts finishing a partially written data block group.
do_zone_finish() then enters btrfs_inc_block_group_ro() again. The same task waits for ro_block_group_mutex that it already holds and self-deadlocks.
[FIX]
Remove the two allocation-time activation calls and their helper for two reasons:
1. They were added for active_total_bytes accounting, which no longer exists. Metadata and system block groups are now activated at write time, and both callers already handle no block group being activated.
2. Allocation-time activation bypasses the role pivot. If A is the current system block group, activating B consumes another active-zone slot instead of finishing A. If no slot is available, the helper tries to finish an unrelated data block group, creating the recursive path above.
With these activation paths removed, the first write to a sequential metadata or system block group enters check_bg_is_active(), which finishes the previous block group for that role before activating the new one.
--- fs/btrfs/block-group.c | 38 +++++++++----------------------- fs/btrfs/zoned.c | 50 ------------------------------------------ fs/btrfs/zoned.h | 8 ------- 3 files changed, 10 insertions(+), 86 deletions(-)
fs/btrfs/block-group.c +10 -28
@@ -3219,13 +3219,6 @@ int btrfs_inc_block_group_ro(struct btrfs_block_group *cache, alloc_flags =btrfs_get_alloc_profile (fs_info, space_info->flags); ret =btrfs_chunk_alloc (trans, space_info, alloc_flags,CHUNK_ALLOC_FORCE ); -if (ret <0 ) - goto out; -/* -* We have allocated a new chunk. We also need to activate that chunk to -* grant metadata tickets for zoned filesystem. -*/ - ret =btrfs_zoned_activate_one_bg (space_info, true);if (ret <0 ) goto out;
@@ -4537,29 +4530,18 @@ static void reserve_chunk_space(struct btrfs_trans_handle *trans,if (IS_ERR (bg)) { ret =PTR_ERR (bg); }else { -int activate_ret; -/* -* We have a new chunk. We also need to activate it for -* zoned filesystem. +* If we fail to add the chunk item here, we end +* up trying again at phase 2 of chunk allocation, +* at btrfs_create_pending_block_groups(). So +* ignore any error here. An ENOSPC here could +* happen, due to the cases described at +* do_chunk_alloc() - the system block group we +* just created was just turned into RO mode by a +* scrub for example, or a running discard +* temporarily removed its free space entries, etc. */ - activate_ret =btrfs_zoned_activate_one_bg (info, true); -if (activate_ret <0 ) { - ret = activate_ret; - }else { -/* -* If we fail to add the chunk item here, we end -* up trying again at phase 2 of chunk allocation, -* at btrfs_create_pending_block_groups(). So -* ignore any error here. An ENOSPC here could -* happen, due to the cases described at -* do_chunk_alloc() - the system block group we -* just created was just turned into RO mode by a -* scrub for example, or a running discard -* temporarily removed its free space entries, etc. -*/ -btrfs_chunk_alloc_add_chunk_item (trans, bg); - } +btrfs_chunk_alloc_add_chunk_item (trans, bg); } }
fs/btrfs/zoned.c +0 -50
56 lines not on this page · raw message
fs/btrfs/zoned.h +0 -8
@@ -94,7 +94,6 @@ bool btrfs_zoned_should_reclaim(const struct btrfs_fs_info *fs_info);void btrfs_zoned_release_data_reloc_bg (struct btrfs_fs_info *fs_info,u64 logical,u64 length);int btrfs_zone_finish_one_bg (struct btrfs_fs_info *fs_info); -int btrfs_zoned_activate_one_bg (struct btrfs_space_info *space_info,bool do_finish);int btrfs_restore_active_nondata_bgs (struct btrfs_fs_info *fs_info);int btrfs_reset_unused_block_groups (struct btrfs_space_info *space_info,u64 num_bytes);void btrfs_show_zoned_stats (struct btrfs_fs_info *fs_info,struct seq_file *seq);
@@ -272,13 +271,6 @@ static inline int btrfs_zone_finish_one_bg(struct btrfs_fs_info *fs_info)return 1 ; } -static inline int btrfs_zoned_activate_one_bg (struct btrfs_space_info *space_info, -bool do_finish) -{ -/* Consider all the block groups are active */ -return 0 ; -} -static inline int btrfs_restore_active_nondata_bgs (struct btrfs_fs_info *fs_info) {return 0 ;
Signature · 1 line
2.39.5
Re: [PATCH 3/4] btrfs: zoned: remove obsolete non-data block group activation helper
Apart from the commit message style nit
PATCH 4/4 btrfs: zoned: serialize zone finishing per block group
In reply to Dongjiang Zhu[BUG]
On an HC620 with max_active_zones=128, buffered writes occupied all 128 active-zone slots. A metadata-heavy create/delete workload followed by sync then forced a metadata block group switch.
A kernel with additional zone finish tracing reported:
zoned-debug: finish clear-runtime bg=140928614400 caller=btrfs_zone_finish_endio_workfn comm=kworker/u8:7
zoned-debug: finish early-inactive bg=140928614400 caller=btrfs_check_meta_write_pointer comm=sync
zoned-debug: pivot finish ret=0 old_bg=140928614400 new_bg=339839287296
zoned-debug: activate fail slot bg=339839287296 type=metadata left=0 max_active=128 comm=sync
zoned-debug: finish zone-mgmt bg=140928614400 ret=0 left=0 caller=btrfs_zone_finish_endio_workfn comm=kworker/u8:7 BTRFS error (device sdb): error while writing out transaction: -EAGAIN
BTRFS error (device sdb state A): Transaction 67 aborted (-EAGAIN)
BTRFS info (device sdb state EA): forced readonly
zoned-debug: clear-active dev=/dev/sdb pos=140928614400 left=1 max_active=128
The transaction was aborted before the old block group returned its active-zone slot, forcing the filesystem read-only.
[CAUSE]do_zone_finish() clears BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE before issuing the zone finish operation. This allows the following interleaving:
Thread A, endio worker Thread B, transaction writeback
clear ZONE_IS_ACTIVE
call_zone_finish(old)
observe old as inactive return from do_zone_finish(old) activate(new)
fail with left == 0
clear the device active bit
increment active_zones_left
Thread B treats the cleared active bit as a completed finish while thread A is still performing the device operation.
[FIX]
Add BLOCK_GROUP_FLAG_ZONE_FINISHING and keep ZONE_IS_ACTIVE set until the zone finish operation and active-list cleanup are complete.
Concurrent finishers wait for ZONE_FINISHING to be cleared instead of returning early. Wake them after clearing ZONE_IS_ACTIVE and removing the block group from the active list. If a waiter acquired a temporary read-only reference, release it after the wait.
--- fs/btrfs/block-group.h | 1 + fs/btrfs/zoned.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-)
fs/btrfs/block-group.h +1 -0
@@ -99,6 +99,7 @@ enum btrfs_block_group_flags {BLOCK_GROUP_FLAG_NEW ,BLOCK_GROUP_FLAG_FULLY_REMAPPED ,BLOCK_GROUP_FLAG_STRIPE_REMOVAL_PENDING , + BLOCK_GROUP_FLAG_ZONE_FINISHING, }; enum btrfs_caching_type {
fs/btrfs/zoned.c +28 -1
@@ -2558,6 +2558,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writconst bool is_metadata = (block_group->flags & (BTRFS_BLOCK_GROUP_METADATA |BTRFS_BLOCK_GROUP_SYSTEM ));struct btrfs_dev_replace *dev_replace = &fs_info->dev_replace; +bool bg_ro = false;int ret =0 ;int i;
@@ -2567,6 +2568,11 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writreturn 0 ; } +if (test_bit (BLOCK_GROUP_FLAG_ZONE_FINISHING , &block_group->runtime_flags)) { +spin_unlock (&block_group->lock); + goto wait_finish; + } +/* Check if we have unwritten allocated space */ if (is_metadata && block_group->start + block_group->alloc_offset > block_group->meta_write_pointer) {
@@ -2591,6 +2597,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ ret =btrfs_inc_block_group_ro (block_group, false);if (ret)return ret; + bg_ro = true;/* Ensure all writes in this block group finish */ btrfs_wait_block_group_reservations (block_group);
@@ -2613,6 +2620,12 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writreturn 0 ; } +if (test_bit (BLOCK_GROUP_FLAG_ZONE_FINISHING , + &block_group->runtime_flags)) { +spin_unlock (&block_group->lock); + goto wait_finish; + } +if (block_group->reserved ||test_bit (BLOCK_GROUP_FLAG_ZONED_DATA_RELOC , &block_group->runtime_flags)) {
@@ -2622,7 +2635,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ } } -clear_bit (BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE , &block_group->runtime_flags); +set_bit (BLOCK_GROUP_FLAG_ZONE_FINISHING , &block_group->runtime_flags); block_group->alloc_offset = block_group->zone_capacity;if (block_group->flags & (BTRFS_BLOCK_GROUP_METADATA |BTRFS_BLOCK_GROUP_SYSTEM )) block_group->meta_write_pointer = block_group->start +
@@ -2646,8 +2659,13 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writbtrfs_dec_block_group_ro (block_group);spin_lock (&fs_info->zone_active_bgs_lock); +spin_lock (&block_group->lock); +clear_bit (BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE , &block_group->runtime_flags);ASSERT (!list_empty (&block_group->active_bg_list));list_del_init (&block_group->active_bg_list); +clear_and_wake_up_bit (BLOCK_GROUP_FLAG_ZONE_FINISHING , + &block_group->runtime_flags); +spin_unlock (&block_group->lock);spin_unlock (&fs_info->zone_active_bgs_lock);/* For active_bg_list */
@@ -2656,6 +2674,15 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writclear_and_wake_up_bit (BTRFS_FS_NEED_ZONE_FINISH , &fs_info->flags);return ret; + +wait_finish: +wait_on_bit_io (&block_group->runtime_flags, +BLOCK_GROUP_FLAG_ZONE_FINISHING , +TASK_UNINTERRUPTIBLE ); +if (bg_ro) +btrfs_dec_block_group_ro (block_group); + +return 0 ; }int btrfs_zone_finish (struct btrfs_block_group *block_group)
Signature · 1 line
2.39.5
Re: [PATCH 4/4] btrfs: zoned: serialize zone finishing per block group
Quoted 4 lines · Dongjiang Zhu
[BUG]
On an HC620 with max_active_zones=128, buffered writes occupied all 128 active-zone slots. A metadata-heavy create/delete workload followed by sync then forced a metadata block group switch.
Can you explain that to me. In my mental model, if all 128 active-zones are used and you call sync, we need to add a treelog-bg. But there is no more active zone resource available so we cannot create the treelog bg. This is why we fail, so we need to finish a zone first to then create a treelog bg.
Quoted 13 lines
A kernel with additional zone finish tracing reported:
zoned-debug: finish clear-runtime bg=140928614400 caller=btrfs_zone_finish_endio_workfn comm=kworker/u8:7
zoned-debug: finish early-inactive bg=140928614400 caller=btrfs_check_meta_write_pointer comm=sync
zoned-debug: pivot finish ret=0 old_bg=140928614400 new_bg=339839287296
zoned-debug: activate fail slot bg=339839287296 type=metadata left=0 max_active=128 comm=sync
zoned-debug: finish zone-mgmt bg=140928614400 ret=0 left=0 caller=btrfs_zone_finish_endio_workfn comm=kworker/u8:7 BTRFS error (device sdb): error while writing out transaction:-EAGAIN
BTRFS error (device sdb state A): Transaction 67 aborted (-EAGAIN)
BTRFS info (device sdb state EA): forced readonly
zoned-debug: clear-active dev=/dev/sdb pos=140928614400 left=1 max_active=128
Quoted 2 lines
The transaction was aborted before the old block group returned its active-zone slot, forcing the filesystem read-only.
Ah now I see we're on the same page.
Quoted 19 lines
[CAUSE]
do_zone_finish()clearsBLOCK_GROUP_FLAG_ZONE_IS_ACTIVEbefore issuing the zone finish operation. This allows the following interleaving:Thread A, endio worker Thread B, transaction writeback
clear
ZONE_IS_ACTIVE
call_zone_finish(old)
observe old as inactive return fromdo_zone_finish(old)activate(new)
fail with left == 0
clear the device active bit
incrementactive_zones_leftThread B treats the cleared active bit as a completed finish while thread A is still performing the device operation.
Quoted 9 lines
[FIX]
AddBLOCK_GROUP_FLAG_ZONE_FINISHINGand keepZONE_IS_ACTIVEset until the zone finish operation and active-list cleanup are complete.Concurrent finishers wait for
ZONE_FINISHINGto be cleared instead of returning early. Wake them after clearingZONE_IS_ACTIVEand removing the block group from the active list. If a waiter acquired a temporary read-only reference, release it after the wait.
Stupid question, wouldn't it be enough to re-order the call to do_zone_finish() vs the clearing of BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE?
Re: [PATCH 4/4] btrfs: zoned: serialize zone finishing per block group
Quoted 51 lines · Johannes Thumshirn
Quoted 4 lines · Dongjiang Zhu
[BUG]
On an HC620 with max_active_zones=128, buffered writes occupied all 128 active-zone slots. A metadata-heavy create/delete workload followed by sync then forced a metadata block group switch.Can you explain that to me. In my mental model, if all 128 active-zones are used and you call sync, we need to add a treelog-bg. But there is no more active zone resource available so we cannot create the treelog bg. This is why we fail, so we need to finish a zone first to then create a treelog bg.
Quoted 13 lines
A kernel with additional zone finish tracing reported:
zoned-debug: finish clear-runtime bg=140928614400 caller=btrfs_zone_finish_endio_workfn comm=kworker/u8:7
zoned-debug: finish early-inactive bg=140928614400 caller=btrfs_check_meta_write_pointer comm=sync
zoned-debug: pivot finish ret=0 old_bg=140928614400 new_bg=339839287296
zoned-debug: activate fail slot bg=339839287296 type=metadata left=0 max_active=128 comm=sync
zoned-debug: finish zone-mgmt bg=140928614400 ret=0 left=0 caller=btrfs_zone_finish_endio_workfn comm=kworker/u8:7 BTRFS error (device sdb): error while writing out transaction:-EAGAIN
BTRFS error (device sdb state A): Transaction 67 aborted (-EAGAIN)
BTRFS info (device sdb state EA): forced readonly
zoned-debug: clear-active dev=/dev/sdb pos=140928614400 left=1 max_active=128Quoted 2 lines
The transaction was aborted before the old block group returned its active-zone slot, forcing the filesystem read-only.
Ah now I see we're on the same page.
Quoted 23 lines
[CAUSE]
do_zone_finish()clearsBLOCK_GROUP_FLAG_ZONE_IS_ACTIVEbefore issuing the zone finish operation. This allows the following interleaving:Thread A, endio worker Thread B, transaction writeback
clear
ZONE_IS_ACTIVE
call_zone_finish(old)
observe old as inactive return fromdo_zone_finish(old)activate(new)
fail with left == 0
clear the device active bit
incrementactive_zones_leftThread B treats the cleared active bit as a completed finish while thread A is still performing the device operation.
[FIX]
AddBLOCK_GROUP_FLAG_ZONE_FINISHINGand keepZONE_IS_ACTIVEset until the zone finish operation and active-list cleanup are complete.
Quoted 8 lines
Quoted 5 lines
Concurrent finishers wait for
ZONE_FINISHINGto be cleared instead of returning early. Wake them after clearingZONE_IS_ACTIVEand removing the block group from the active list. If a waiter acquired a temporary read-only reference, release it after the wait.Stupid question, wouldn't it be enough to re-order the call to
do_zone_finish()vs the clearing ofBLOCK_GROUP_FLAG_ZONE_IS_ACTIVE?
Do you mean keeping BLOCK_GROUP_FLAG_ZONE_IS_ACTIVE set while call_zone_finish() is running, and clearing it together with list_del_init() after the device operation completes?
If so, I think that closes the early-inactive window, but it still allows two callers to finish the same block group concurrently:
Thread A Thread B
call_zone_finish()
observe ZONE_IS_ACTIVE call_zone_finish()
While thread A is blocked in blkdev_zone_mgmt(), thread B still sees the block group as active and can enter call_zone_finish() as well.
Even if issuing ZONE_FINISH twice is harmless at the device level, the software cleanup is not idempotent. If both calls succeed for a non-data block group, both increment reserved_active_zones, while active_zones_left is restored only once because the device active bitmap uses test_and_clear_bit(). Both callers would also remove the same active-list entry and drop its reference.
Am I missing some existing serialization that prevents two do_zone_finish() callers for the same block group from reaching call_zone_finish() concurrently?
Thanks,
Dongjiang Zhu