Skip to contentSkip to conversation

zbd/009 (btrfs on zone gaps) fail on current Linus tree

DISCUSSION Christoph Hellwig 3 participants 15 messages Oct 7, 2025 – Oct 20, 2025 Btrfs file system

lore
Share & tools
lore
#0 lore raw

At least in my usual x86_64 test VM:

zbd/009 (test gap zone support with BTRFS)

[   21.102324] run blktests zbd/009 at 2025-10-07 06:10:44
[   21.251500] sd 3:0:0:0: [sda] Synchronizing SCSI cache
[   21.413634] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw )
[   21.414056] scsi host3: scsi_debug: version 0191 [20210520]
[   21.414056]   dev_size_mb=1024, opts=0x0, submit_queues=1, statistics=0
[   21.415307] scsi 3:0:0:0: Direct-Access-ZBC Linux    scsi_debug 0191 PQ: 0 ANSI: 7
[   21.416384] scsi 3:0:0:0: Power-on or device reset occurred
[   21.416981] sd 3:0:0:0: Attached scsi generic sg1 type 20
[   21.417533] sd 3:0:0:0: [sda] Host-managed zoned block device
[   21.418153] sd 3:0:0:0: [sda] 262144 4096-byte logical blocks: (1.07 GB/1.00 GiB)
[   21.418676] sd 3:0:0:0: [sda] Write Protect is off
[   21.419017] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   21.419685] sd 3:0:0:0: [sda] permanent stream count = 5
[   21.420158] sd 3:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[   21.420593] sd 3:0:0:0: [sda] Optimal transfer size 4194304 bytes
[   21.421261] sd 3:0:0:0: [sda] 256 zones of 1024 logical blocks
[   21.456700] sd 3:0:0:0: [sda] Attached SCSI disk
[   21.523845] BTRFS: device fsid 9bcd6f4c-db2e-44d7-8597-4eb5774c1460 devid 1 transid 6 /dev/s)
[   21.528211] BTRFS info (device sda): first mount of filesystem 9bcd6f4c-db2e-44d7-8597-4eb570
[   21.528623] BTRFS info (device sda): using crc32c (crc32c-lib) checksum algorithm
[   21.530206] BTRFS info (device sda): host-managed zoned block device /dev/sda, 256 zones of s
[   21.530663] BTRFS info (device sda): zoned mode enabled with zone size 4194304
[   21.532601] BTRFS info (device sda): checking UUID tree
[   21.532909] BTRFS info (device sda): enabling ssd optimizations
[   21.533145] BTRFS info (device sda): enabling free space tree
[  242.795457] INFO: task kworker/u8:4:859 blocked for more than 120 seconds.
[  242.796028]       Tainted: G                 N  6.17.0+ #4047
[  242.796426] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.796941] task:kworker/u8:4    state:D stack:0     pid:859 tgid:859   ppid:2      task_f0
[  242.797667] Workqueue: writeback wb_workfn (flush-btrfs-2)
[  242.798065] Call Trace:
[  242.798227]  <TASK>
[  242.798369]  __schedule+0x524/0xb60
[  242.798601]  schedule+0x29/0xe0
[  242.798804]  io_schedule+0x4b/0x70
[  242.799024]  folio_wait_bit_common+0x126/0x390
[  242.799300]  ? filemap_get_folios_tag+0x247/0x2a0
[  242.800054]  ? __pfx_wake_page_function+0x10/0x10
[  242.800354]  extent_write_cache_pages+0x5c6/0x9c0
[  242.800631]  ? stack_depot_save_flags+0x29/0x870
[  242.800904]  ? set_track_prepare+0x45/0x70
[  242.801145]  ? __kmalloc_noprof+0x3a7/0x4e0
[  242.801391]  ? virtqueue_add_sgs+0x308/0x720
[  242.801644]  ? virtblk_add_req+0x81/0xe0
[  242.801875]  ? virtblk_add_req_batch+0x4b/0x100
[  242.802140]  ? virtio_queue_rqs+0x133/0x180
[  242.802385]  ? blk_mq_dispatch_queue_requests+0x155/0x180
[  242.802697]  ? blk_mq_flush_plug_list+0x73/0x160
[  242.802967]  ? preempt_count_add+0x4d/0xb0
[  242.803210]  btrfs_writepages+0x70/0x120
[  242.803636]  do_writepages+0xc5/0x160
[  242.803870]  __writeback_single_inode+0x3c/0x330
[  242.804154]  writeback_sb_inodes+0x21a/0x4d0
[  242.804436]  __writeback_inodes_wb+0x47/0xe0
[  242.804758]  wb_writeback+0x19a/0x310
[  242.805029]  wb_workfn+0x348/0x440
[  242.805248]  process_one_work+0x169/0x320
[  242.805487]  worker_thread+0x246/0x390
[  242.805711]  ? _raw_spin_unlock_irqrestore+0x1d/0x40
[  242.806003]  ? __pfx_worker_thread+0x10/0x10
[  242.806253]  kthread+0x106/0x220
[  242.806454]  ? __pfx_kthread+0x10/0x10
[  242.806683]  ? __pfx_kthread+0x10/0x10
[  242.806915]  ret_from_fork+0x11d/0x160
[  242.807145]  ? __pfx_kthread+0x10/0x10
[  242.807646]  ret_from_fork_asm+0x1a/0x30
[  242.807904]  </TASK>
#1 lore raw
Quoted 63 lines · Christoph Hellwig

At least in my usual x86_64 test VM:

zbd/009 (test gap zone support with BTRFS)

[   21.102324] run blktests zbd/009 at 2025-10-07 06:10:44
[   21.251500] sd 3:0:0:0: [sda] Synchronizing SCSI cache
[   21.413634] scsi_debug:sdebug_driver_probe: scsi_debug: trim poll_queues to 0. poll_q/nr_hw )
[   21.414056] scsi host3: scsi_debug: version 0191 [20210520]
[   21.414056]   dev_size_mb=1024, opts=0x0, submit_queues=1, statistics=0
[   21.415307] scsi 3:0:0:0: Direct-Access-ZBC Linux    scsi_debug 0191 PQ: 0 ANSI: 7
[   21.416384] scsi 3:0:0:0: Power-on or device reset occurred
[   21.416981] sd 3:0:0:0: Attached scsi generic sg1 type 20
[   21.417533] sd 3:0:0:0: [sda] Host-managed zoned block device
[   21.418153] sd 3:0:0:0: [sda] 262144 4096-byte logical blocks: (1.07 GB/1.00 GiB)
[   21.418676] sd 3:0:0:0: [sda] Write Protect is off
[   21.419017] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   21.419685] sd 3:0:0:0: [sda] permanent stream count = 5
[   21.420158] sd 3:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
[   21.420593] sd 3:0:0:0: [sda] Optimal transfer size 4194304 bytes
[   21.421261] sd 3:0:0:0: [sda] 256 zones of 1024 logical blocks
[   21.456700] sd 3:0:0:0: [sda] Attached SCSI disk
[   21.523845] BTRFS: device fsid 9bcd6f4c-db2e-44d7-8597-4eb5774c1460 devid 1 transid 6 /dev/s)
[   21.528211] BTRFS info (device sda): first mount of filesystem 9bcd6f4c-db2e-44d7-8597-4eb570
[   21.528623] BTRFS info (device sda): using crc32c (crc32c-lib) checksum algorithm
[   21.530206] BTRFS info (device sda): host-managed zoned block device /dev/sda, 256 zones of s
[   21.530663] BTRFS info (device sda): zoned mode enabled with zone size 4194304
[   21.532601] BTRFS info (device sda): checking UUID tree
[   21.532909] BTRFS info (device sda): enabling ssd optimizations
[   21.533145] BTRFS info (device sda): enabling free space tree
[  242.795457] INFO: task kworker/u8:4:859 blocked for more than 120 seconds.
[  242.796028]       Tainted: G                 N  6.17.0+ #4047
[  242.796426] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.796941] task:kworker/u8:4    state:D stack:0     pid:859 tgid:859   ppid:2      task_f0
[  242.797667] Workqueue: writeback wb_workfn (flush-btrfs-2)
[  242.798065] Call Trace:
[  242.798227]  <TASK>
[  242.798369]  __schedule+0x524/0xb60
[  242.798601]  schedule+0x29/0xe0
[  242.798804]  io_schedule+0x4b/0x70
[  242.799024]  folio_wait_bit_common+0x126/0x390
[  242.799300]  ? filemap_get_folios_tag+0x247/0x2a0
[  242.800054]  ? __pfx_wake_page_function+0x10/0x10
[  242.800354]  extent_write_cache_pages+0x5c6/0x9c0
[  242.800631]  ? stack_depot_save_flags+0x29/0x870
[  242.800904]  ? set_track_prepare+0x45/0x70
[  242.801145]  ? __kmalloc_noprof+0x3a7/0x4e0
[  242.801391]  ? virtqueue_add_sgs+0x308/0x720
[  242.801644]  ? virtblk_add_req+0x81/0xe0
[  242.801875]  ? virtblk_add_req_batch+0x4b/0x100
[  242.802140]  ? virtio_queue_rqs+0x133/0x180
[  242.802385]  ? blk_mq_dispatch_queue_requests+0x155/0x180
[  242.802697]  ? blk_mq_flush_plug_list+0x73/0x160
[  242.802967]  ? preempt_count_add+0x4d/0xb0
[  242.803210]  btrfs_writepages+0x70/0x120
[  242.803636]  do_writepages+0xc5/0x160
[  242.803870]  __writeback_single_inode+0x3c/0x330
[  242.804154]  writeback_sb_inodes+0x21a/0x4d0
[  242.804436]  __writeback_inodes_wb+0x47/0xe0
[  242.804758]  wb_writeback+0x19a/0x310
[  242.805029]  wb_workfn+0x348/0x440
[  242.805248]  process_one_work+0x169/0x320
[  242.805487]  worker_thread+0x246/0x390
[  242.805711]  ? _raw_spin_unlock_irqrestore+0x1d/0x40
Quoted 11 lines
[  242.806003]  ? __pfx_worker_thread+0x10/0x10
[  242.806253]  kthread+0x106/0x220
[  242.806454]  ? __pfx_kthread+0x10/0x10
[  242.806683]  ? __pfx_kthread+0x10/0x10
[  242.806915]  ret_from_fork+0x11d/0x160
[  242.807145]  ? __pfx_kthread+0x10/0x10
[  242.807646]  ret_from_fork_asm+0x1a/0x30
[  242.807904]  </TASK>

hmm how reproducible is it on your side? I cannot reproduce it (yet)

zbd/009 (test gap zone support with BTRFS)  [passed]
    runtime    ...  12.516s

Will run the test in a loop (note I've tested both today's tree from Linus as well as btrfs/for-next from today)

#2 lore raw
Quoted 1 line · Johannes Thumshirn

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

My kernel .config and qemu command line are attached.

#3 lore raw
Quoted 7 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

My kernel .config and qemu command line are attached.

OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(

#4 lore raw
Quoted 12 lines · Johannes Thumshirn
Quoted 7 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

My kernel .config and qemu command line are attached.

OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(

Nope, even with your kconfig no success on recreating the bug.

#5 lore raw
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

My kernel .config and qemu command line are attached.

OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(

Nope, even with your kconfig no success on recreating the bug.

Weird. I'll see if I can find some time to bisect it.

#6 lore raw
In reply to Johannes Thumshirn

I've bisected the hang to:

commit 04147d8394e80acaaebf0365f112339e8b606c05 (HEAD)

Author: Naohiro Aota <[email protected]>
Date:   Wed Jul 16 16:59:55 2025 +0900

btrfs: zoned: limit active zones to max_open_zones

with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.

Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 1 line · Johannes Thumshirn

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

My kernel .config and qemu command line are attached.

OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(

Nope, even with your kconfig no success on recreating the bug.

---end quoted text---

#7 lore raw
Quoted 10 lines · Christoph Hellwig

I've bisected the hang to:

commit 04147d8394e80acaaebf0365f112339e8b606c05 (HEAD)

Author: Naohiro Aota <[email protected]>
Date:   Wed Jul 16 16:59:55 2025 +0900

btrfs: zoned: limit active zones to max_open_zones

with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.

I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have 53de7ee4e28f ("btrfs: zoned: don't fail mount needlessly due to too many active zones")?

@Naohiro can you have a look if you can reproduce it?

Thanks,

    Johannes

Quoted 18 lines · Johannes Thumshirn
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 5 lines · Johannes Thumshirn
Quoted 1 line

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

My kernel .config and qemu command line are attached.

OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(

Nope, even with your kconfig no success on recreating the bug.

---end quoted text---

#8 lore raw
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig

I've bisected the hang to:

commit 04147d8394e80acaaebf0365f112339e8b606c05 (HEAD)

Author: Naohiro Aota <[email protected]>
Date:   Wed Jul 16 16:59:55 2025 +0900
Quoted 8 lines
Quoted 4 lines

btrfs: zoned: limit active zones to max_open_zones

with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.

I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have 53de7ee4e28f ("btrfs: zoned: don't fail mount needlessly due to too many active zones")?

Still hanging on -rc2 that has it.

#9 lore raw
Quoted 8 lines · On 10/20/25 8:47 AM, [email protected] wrote:
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · Christoph Hellwig

I've bisected the hang to:

commit 04147d8394e80acaaebf0365f112339e8b606c05 (HEAD)

Author: Naohiro Aota <[email protected]>
Date:   Wed Jul 16 16:59:55 2025 +0900
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

btrfs: zoned: limit active zones to max_open_zones

with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.

I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have 53de7ee4e28f ("btrfs: zoned: don't fail mount needlessly due to too many active zones")?

Still hanging on -rc2 that has it.

OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?

#10 lore raw
Quoted 2 lines · Johannes Thumshirn

OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?

Just running zbd/009 is fine.

Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.

#11 lore raw
Quoted 8 lines · On 10/20/25 8:53 AM, [email protected] wrote:
Quoted 2 lines · Johannes Thumshirn

OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?

Just running zbd/009 is fine.

Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.

Yeah the issue on my side could be that I don't have virtio-blk devices attached to the VM as I don't have debian images lying aroudn for the rootfs (I'm using my hosts' root as a RO virtiofs mount).

I try to find something.

#12 lore raw
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/20/25 8:53 AM, [email protected] wrote:
Quoted 2 lines · Johannes Thumshirn

OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?

Just running zbd/009 is fine.

Quoted 7 lines
Quoted 3 lines

Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.

Yeah the issue on my side could be that I don't have virtio-blk devices attached to the VM as I don't have debian images lying aroudn for the rootfs (I'm using my hosts' root as a RO virtiofs mount).

Actually, the zbd tests emulate the zoned devices on top of the regular ones, so it should not matter. Here is my blktests config, btw:

TEST_DEVS=(/dev/vdb)
nvme_trtype=loop

so nothing special.

#13 lore raw
Quoted 7 lines · On 10/20/25 9:22 AM, [email protected] wrote:
Quoted 6 lines · Johannes Thumshirn
Quoted 5 lines · On 10/20/25 8:53 AM, [email protected] wrote:
Quoted 2 lines · Johannes Thumshirn

OK, maybe I tested wrong. Does it also hang if you only run zbd/009 or do you need to run the other zbd tests before?

Just running zbd/009 is fine.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Also make sure to try to use my kvm script as-is as it gets, given the bisected commit I might be sensitive to the open zone limits or zone append size or something like that.

Yeah the issue on my side could be that I don't have virtio-blk devices attached to the VM as I don't have debian images lying aroudn for the rootfs (I'm using my hosts' root as a RO virtiofs mount).

Actually, the zbd tests emulate the zoned devices on top of the regular ones, so it should not matter. Here is my blktests config, btw:

yes

Quoted 5 lines

TEST_DEVS=(/dev/vdb)
nvme_trtype=loop

so nothing special.

and yes.

Can you generate me a vmcore and send me the vmlinux (with debug info) and core? Maybe I can find out what's going on that way.

#14 lore raw
In reply to Johannes Thumshirn
Quoted 9 lines · Johannes Thumshirn
Quoted 8 lines · Christoph Hellwig

I've bisected the hang to:

commit 04147d8394e80acaaebf0365f112339e8b606c05 (HEAD)

Author: Naohiro Aota <[email protected]>
Date:   Wed Jul 16 16:59:55 2025 +0900

btrfs: zoned: limit active zones to max_open_zones

Quoted 8 lines
Quoted 2 lines

with that patch zbd/009 hangs 100% for my config, and without it, it works fine 100%.

I still can't reproduce it. We seen a mount error as fallout of it though, can you check if you have 53de7ee4e28f ("btrfs: zoned: don't fail mount needlessly due to too many active zones")?

@Naohiro can you have a look if you can reproduce it?

I'm running the test case 100 times on my virtme setup, and it all passed so far.

Quoted 25 lines

Thanks,

    Johannes

Quoted 18 lines · Johannes Thumshirn
Quoted 7 lines · Johannes Thumshirn
Quoted 6 lines · On 10/8/25 8:02 AM, [email protected] wrote:
Quoted 5 lines · Johannes Thumshirn
Quoted 1 line

hmm how reproducible is it on your side? I cannot reproduce it (yet)

100% over about a dozen runs, a few of those including unrelated patches.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

My kernel .config and qemu command line are attached.

OK I'll give it a shot. For my config + qemu it survived 250 runs of zbd/009 yesterday without a hang :(

Nope, even with your kconfig no success on recreating the bug.

---end quoted text---