Skip to contentSkip to conversation

nvme: fio workload hangs on 16m random read

bug report Andreas Hindborg 2 participants 3 messages Mar 6, 2025 – Mar 7, 2025 NVMe

lore
Share & tools
lore
#0 lore raw

Hi All,

My CI system has found an issue for the following fio workload:

root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme-core.ko
root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme.ko poll_queues=1

"fio" "--group_reporting" "--name=default" "--filename=/dev/nvme0n1" "--readwrite=randread" "--blocksize=16777216" "--direct=1" "--numjobs=1" "--ioengine=io_uring" "--iodepth=1" "--hipri=1" "--time_based=1" "--runtime=3000"

fio will hang, some times immediately, some times after a few minutes.

I observed this on v6.14-rc1 and v6.14-rc5. It is not present in v6.13.

Can any of you reproduce this? I was able to reproduce this on real hardware and in qemu.

Best regards,
Andreas Hindborg

maintainer
#1 lore raw
Quoted 12 lines · Andreas Hindborg

Hi All,

My CI system has found an issue for the following fio workload:

root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme-core.ko
root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme.ko poll_queues=1

"fio" "--group_reporting" "--name=default" "--filename=/dev/nvme0n1" "--readwrite=randread" "--blocksize=16777216" "--direct=1" "--numjobs=1" "--ioengine=io_uring" "--iodepth=1" "--hipri=1" "--time_based=1" "--runtime=3000"

Quoted 6 lines

fio will hang, some times immediately, some times after a few minutes.

I observed this on v6.14-rc1 and v6.14-rc5. It is not present in v6.13.

Can any of you reproduce this? I was able to reproduce this on real hardware and in qemu.

Could you try adding this patch to your kernel?

https://lore.kernel.org/io-uring/[email protected]/

#2 lore raw
Quoted 15 lines · Keith Busch
Quoted 14 lines · Andreas Hindborg

Hi All,

My CI system has found an issue for the following fio workload:

root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme-core.ko
root@debian:~# insmod /mnt/v6.14-rc5-build/drivers/nvme/host/nvme.ko poll_queues=1

"fio" "--group_reporting" "--name=default" "--filename=/dev/nvme0n1" "--readwrite=randread" "--blocksize=16777216" "--direct=1" "--numjobs=1" "--ioengine=io_uring" "--iodepth=1" "--hipri=1" "--time_based=1" "--runtime=3000"

fio will hang, some times immediately, some times after a few minutes.

Quoted 8 lines
Quoted 4 lines

I observed this on v6.14-rc1 and v6.14-rc5. It is not present in v6.13.

Can any of you reproduce this? I was able to reproduce this on real hardware and in qemu.

Could you try adding this patch to your kernel?

https://lore.kernel.org/io-uring/[email protected]/

Thanks Keith, at first glance that seems to solve the issue. Will restart my CI with this applied.

Best regards,
Andreas Hindborg