zbd/009 (btrfs on zone gaps) fail on current Linus tree
At least in my usual x86_64 test VM:
zbd/009 (test gap zone support with BTRFS)
[ 21.102324] run blktests zbd/009 at 2025-10-07 06:10:44 [ 21.251500] sd 3:0:0:0: [sda] Synchronizing SCSI cache [ 21.413634] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw ) [ 21.414056] scsi host3: scsi_debug: version 0191 [20210520] [ 21.414056] dev_size_mb=1024, opts=0x0, submit_queues=1, statistics=0 [ 21.415307] scsi 3:0:0:0: Direct-Access-ZBC Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 21.416384] scsi 3:0:0:0: Power-on or device reset occurred [ 21.416981] sd 3:0:0:0: Attached scsi generic sg1 type 20 [ 21.417533] sd 3:0:0:0: [sda] Host-managed zoned block device [ 21.418153] sd 3:0:0:0: [sda] 262144 4096-byte logical blocks: (1.07 GB/1.00 GiB) [ 21.418676] sd 3:0:0:0: [sda] Write Protect is off [ 21.419017] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.419685] sd 3:0:0:0: [sda] permanent stream count = 5 [ 21.420158] sd 3:0:0:0: [sda] Preferred minimum I/O size 4096 bytes [ 21.420593] sd 3:0:0:0: [sda] Optimal transfer size 4194304 bytes [ 21.421261] sd 3:0:0:0: [sda] 256 zones of 1024 logical blocks [ 21.456700] sd 3:0:0:0: [sda] Attached SCSI disk [ 21.523845] BTRFS: device fsid 9bcd6f4c-db2e-44d7-8597-4eb5774c1460 devid 1 transid 6 /dev/s) [ 21.528211] BTRFS info (device sda): first mount of filesystem 9bcd6f4c-db2e-44d7-8597-4eb570 [ 21.528623] BTRFS info (device sda): using crc32c (crc32c-lib) checksum algorithm [ 21.530206] BTRFS info (device sda): host-managed zoned block device /dev/sda, 256 zones of s [ 21.530663] BTRFS info (device sda): zoned mode enabled with zone size 4194304 [ 21.532601] BTRFS info (device sda): checking UUID tree [ 21.532909] BTRFS info (device sda): enabling ssd optimizations [ 21.533145] BTRFS info (device sda): enabling free space tree
[ 242.795457] INFO: task kworker/u8:4:859 blocked for more than 120 seconds. [ 242.796028] Tainted: G N 6.17.0+ #4047 [ 242.796426] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.796941] task:kworker/u8:4 state:D stack:0 pid:859 tgid:859 ppid:2 task_f0 [ 242.797667] Workqueue: writeback wb_workfn (flush-btrfs-2) [ 242.798065] Call Trace: [ 242.798227] <TASK> [ 242.798369] __schedule+0x524/0xb60 [ 242.798601] schedule+0x29/0xe0 [ 242.798804] io_schedule+0x4b/0x70 [ 242.799024] folio_wait_bit_common+0x126/0x390 [ 242.799300] ? filemap_get_folios_tag+0x247/0x2a0 [ 242.800054] ? __pfx_wake_page_function+0x10/0x10 [ 242.800354] extent_write_cache_pages+0x5c6/0x9c0 [ 242.800631] ? stack_depot_save_flags+0x29/0x870 [ 242.800904] ? set_track_prepare+0x45/0x70 [ 242.801145] ? __kmalloc_noprof+0x3a7/0x4e0 [ 242.801391] ? virtqueue_add_sgs+0x308/0x720 [ 242.801644] ? virtblk_add_req+0x81/0xe0 [ 242.801875] ? virtblk_add_req_batch+0x4b/0x100 [ 242.802140] ? virtio_queue_rqs+0x133/0x180 [ 242.802385] ? blk_mq_dispatch_queue_requests+0x155/0x180 [ 242.802697] ? blk_mq_flush_plug_list+0x73/0x160 [ 242.802967] ? preempt_count_add+0x4d/0xb0 [ 242.803210] btrfs_writepages+0x70/0x120 [ 242.803636] do_writepages+0xc5/0x160 [ 242.803870] __writeback_single_inode+0x3c/0x330 [ 242.804154] writeback_sb_inodes+0x21a/0x4d0 [ 242.804436] __writeback_inodes_wb+0x47/0xe0 [ 242.804758] wb_writeback+0x19a/0x310 [ 242.805029] wb_workfn+0x348/0x440 [ 242.805248] process_one_work+0x169/0x320 [ 242.805487] worker_thread+0x246/0x390 [ 242.805711] ? _raw_spin_unlock_irqrestore+0x1d/0x40 [ 242.806003] ? __pfx_worker_thread+0x10/0x10 [ 242.806253] kthread+0x106/0x220 [ 242.806454] ? __pfx_kthread+0x10/0x10 [ 242.806683] ? __pfx_kthread+0x10/0x10 [ 242.806915] ret_from_fork+0x11d/0x160 [ 242.807145] ? __pfx_kthread+0x10/0x10 [ 242.807646] ret_from_fork_asm+0x1a/0x30 [ 242.807904] </TASK>
Quoted 63 lines · Christoph Hellwig
At least in my usual
x86_64test VM:zbd/009 (test gap zone support with BTRFS)
[ 21.102324] run blktests zbd/009 at 2025-10-07 06:10:44 [ 21.251500] sd 3:0:0:0: [sda] Synchronizing SCSI cache [ 21.413634] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw ) [ 21.414056] scsi host3: scsi_debug: version 0191 [20210520] [ 21.414056] dev_size_mb=1024, opts=0x0, submit_queues=1, statistics=0 [ 21.415307] scsi 3:0:0:0: Direct-Access-ZBC Linux scsi_debug 0191 PQ: 0 ANSI: 7 [ 21.416384] scsi 3:0:0:0: Power-on or device reset occurred [ 21.416981] sd 3:0:0:0: Attached scsi generic sg1 type 20 [ 21.417533] sd 3:0:0:0: [sda] Host-managed zoned block device [ 21.418153] sd 3:0:0:0: [sda] 262144 4096-byte logical blocks: (1.07 GB/1.00 GiB) [ 21.418676] sd 3:0:0:0: [sda] Write Protect is off [ 21.419017] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.419685] sd 3:0:0:0: [sda] permanent stream count = 5 [ 21.420158] sd 3:0:0:0: [sda] Preferred minimum I/O size 4096 bytes [ 21.420593] sd 3:0:0:0: [sda] Optimal transfer size 4194304 bytes [ 21.421261] sd 3:0:0:0: [sda] 256 zones of 1024 logical blocks [ 21.456700] sd 3:0:0:0: [sda] Attached SCSI disk [ 21.523845] BTRFS: device fsid 9bcd6f4c-db2e-44d7-8597-4eb5774c1460 devid 1 transid 6 /dev/s) [ 21.528211] BTRFS info (device sda): first mount of filesystem 9bcd6f4c-db2e-44d7-8597-4eb570 [ 21.528623] BTRFS info (device sda): using crc32c (crc32c-lib) checksum algorithm [ 21.530206] BTRFS info (device sda): host-managed zoned block device /dev/sda, 256 zones of s [ 21.530663] BTRFS info (device sda): zoned mode enabled with zone size 4194304 [ 21.532601] BTRFS info (device sda): checking UUID tree [ 21.532909] BTRFS info (device sda): enabling ssd optimizations [ 21.533145] BTRFS info (device sda): enabling free space tree[ 242.795457] INFO: task kworker/u8:4:859 blocked for more than 120 seconds. [ 242.796028] Tainted: G N 6.17.0+ #4047 [ 242.796426] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.796941] task:kworker/u8:4 state:D stack:0 pid:859 tgid:859 ppid:2 task_f0 [ 242.797667] Workqueue: writeback wb_workfn (flush-btrfs-2) [ 242.798065] Call Trace: [ 242.798227] <TASK> [ 242.798369] __schedule+0x524/0xb60 [ 242.798601] schedule+0x29/0xe0 [ 242.798804] io_schedule+0x4b/0x70 [ 242.799024] folio_wait_bit_common+0x126/0x390 [ 242.799300] ? filemap_get_folios_tag+0x247/0x2a0 [ 242.800054] ? __pfx_wake_page_function+0x10/0x10 [ 242.800354] extent_write_cache_pages+0x5c6/0x9c0 [ 242.800631] ? stack_depot_save_flags+0x29/0x870 [ 242.800904] ? set_track_prepare+0x45/0x70 [ 242.801145] ? __kmalloc_noprof+0x3a7/0x4e0 [ 242.801391] ? virtqueue_add_sgs+0x308/0x720 [ 242.801644] ? virtblk_add_req+0x81/0xe0 [ 242.801875] ? virtblk_add_req_batch+0x4b/0x100 [ 242.802140] ? virtio_queue_rqs+0x133/0x180 [ 242.802385] ? blk_mq_dispatch_queue_requests+0x155/0x180 [ 242.802697] ? blk_mq_flush_plug_list+0x73/0x160 [ 242.802967] ? preempt_count_add+0x4d/0xb0 [ 242.803210] btrfs_writepages+0x70/0x120 [ 242.803636] do_writepages+0xc5/0x160 [ 242.803870] __writeback_single_inode+0x3c/0x330 [ 242.804154] writeback_sb_inodes+0x21a/0x4d0 [ 242.804436] __writeback_inodes_wb+0x47/0xe0 [ 242.804758] wb_writeback+0x19a/0x310 [ 242.805029] wb_workfn+0x348/0x440 [ 242.805248] process_one_work+0x169/0x320 [ 242.805487] worker_thread+0x246/0x390 [ 242.805711] ? _raw_spin_unlock_irqrestore+0x1d/0x40
Quoted 11 lines
[ 242.806003] ? __pfx_worker_thread+0x10/0x10 [ 242.806253] kthread+0x106/0x220 [ 242.806454] ? __pfx_kthread+0x10/0x10 [ 242.806683] ? __pfx_kthread+0x10/0x10 [ 242.806915] ret_from_fork+0x11d/0x160 [ 242.807145] ? __pfx_kthread+0x10/0x10 [ 242.807646] ret_from_fork_asm+0x1a/0x30 [ 242.807904] </TASK>
hmm how reproducible is it on your side? I cannot reproduce it (yet)
zbd/009 (test gap zone support with BTRFS) [passed]
runtime ... 12.516s
Will run the test in a loop (note I've tested both today's tree from Linus as well as btrfs/for-next from today)
Quoted 1 line · Johannes Thumshirn
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
My kernel .config and qemu command line are attached.
Quoted 7 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
My kernel .config and qemu command line are attached.
OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(
Quoted 12 lines · Johannes Thumshirn
Quoted 7 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
My kernel .config and qemu command line are attached.
OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(
Nope, even with your kconfig no success on recreating the bug.
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
My kernel .config and qemu command line are attached.
OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(
Nope, even with your kconfig no success on recreating the bug.
Weird. I'll see if I can find some time to bisect it.
I've bisected the hang to:
commit 04147d8394e80acaaebf0365f112339e8b606c05 (HEAD)
Author: Naohiro Aota <[email protected]> Date: Wed Jul 16 16:59:55 2025 +0900
btrfs: zoned: limit active zones to max_open_zones
with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
My kernel .config and qemu command line are attached.
OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(
Nope, even with your kconfig no success on recreating the bug.
---end quoted text---
Quoted 10 lines · Christoph Hellwig
I've bisected the hang to:
commit
04147d8394e80acaaebf0365f112339e8b606c05(HEAD)Author: Naohiro Aota <[email protected]> Date: Wed Jul 16 16:59:55 2025 +0900btrfs: zoned: limit active zones to
max_open_zoneswith that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.
I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have 53de7ee4e28f ("btrfs: zoned: don't fail mount needlessly due to too many active zones")?
@Naohiro can you have a look if you can reproduce it?
Thanks,
Johannes
Quoted 18 lines · Johannes Thumshirn
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 5 lines · Johannes Thumshirn
Quoted 1 line
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
My kernel .config and qemu command line are attached.
OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(
Nope, even with your kconfig no success on recreating the bug.
---end quoted text---
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig
I've bisected the hang to:
commit
04147d8394e80acaaebf0365f112339e8b606c05(HEAD)Author: Naohiro Aota <[email protected]> Date: Wed Jul 16 16:59:55 2025 +0900
Quoted 8 lines
Quoted 4 lines
btrfs: zoned: limit active zones to
max_open_zoneswith that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.
I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have
53de7ee4e28f("btrfs: zoned: don't fail mount needlessly due to too many active zones")?
Still hanging on -rc2 that has it.
Quoted 8 lines · On 10/20/25 8:47 AM, [email protected] wrote:
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig
I've bisected the hang to:
commit
04147d8394e80acaaebf0365f112339e8b606c05(HEAD)Author: Naohiro Aota <[email protected]> Date: Wed Jul 16 16:59:55 2025 +0900
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
btrfs: zoned: limit active zones to
max_open_zoneswith that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.
I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have
53de7ee4e28f("btrfs: zoned: don't fail mount needlessly due to too many active zones")?Still hanging on -rc2 that has it.
OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?
Quoted 2 lines · Johannes Thumshirn
OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?
Just running zbd/009 is fine.
Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.
Quoted 8 lines · On 10/20/25 8:53 AM, [email protected] wrote:
Quoted 2 lines · Johannes Thumshirn
OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?
Just running zbd/009 is fine.
Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.
Yeah the issue on my side could be that I don't have virtio-blk devices attached to the VM as I don't have debian images lying aroudn for the rootfs (I'm using my hosts' root as a RO virtiofs mount).
I try to find something.
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/20/25 8:53 AM, [email protected] wrote:
Quoted 2 lines · Johannes Thumshirn
OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?
Just running zbd/009 is fine.
Quoted 7 lines
Quoted 3 lines
Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.
Yeah the issue on my side could be that I don't have virtio-blk devices attached to the VM as I don't have debian images lying aroudn for the rootfs (I'm using my hosts' root as a RO virtiofs mount).
Actually, the zbd tests emulate the zoned devices on top of the regular ones, so it should not matter. Here is my blktests config, btw:
TEST_DEVS=(/dev/vdb)
nvme_trtype=loop
so nothing special.
Quoted 7 lines · On 10/20/25 9:22 AM, [email protected] wrote:
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/20/25 8:53 AM, [email protected] wrote:
Quoted 2 lines · Johannes Thumshirn
OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?
Just running zbd/009 is fine.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.
Yeah the issue on my side could be that I don't have virtio-blk devices attached to the VM as I don't have debian images lying aroudn for the rootfs (I'm using my hosts' root as a RO virtiofs mount).
Actually, the zbd tests emulate the zoned devices on top of the regular ones, so it should not matter. Here is my blktests config, btw:
yes
Quoted 5 lines
TEST_DEVS=(/dev/vdb)
nvme_trtype=loopso nothing special.
and yes.
Can you generate me a vmcore and send me the vmlinux (with debug info) and core? Maybe I can find out what's going on that way.
Quoted 9 lines · Johannes Thumshirn
Quoted 8 lines · Christoph Hellwig
I've bisected the hang to:
commit
04147d8394e80acaaebf0365f112339e8b606c05(HEAD)Author: Naohiro Aota <[email protected]> Date: Wed Jul 16 16:59:55 2025 +0900btrfs: zoned: limit active zones to
max_open_zones
Quoted 8 lines
Quoted 2 lines
with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.
I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have
53de7ee4e28f("btrfs: zoned: don't fail mount needlessly due to too many active zones")?@Naohiro can you have a look if you can reproduce it?
I'm running the test case 100 times on my virtme setup, and it all passed so far.
Quoted 25 lines
Thanks,
Johannes
Quoted 18 lines · Johannes Thumshirn
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 5 lines · Johannes Thumshirn
Quoted 1 line
hmm how reproducible is it on your side? I cannot reproduce it (yet)
100% over about a dozen runs, a few of those including unrelated patches.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
My kernel .config and qemu command line are attached.
OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(
Nope, even with your kconfig no success on recreating the bug.
---end quoted text---