Skip to contentSkip to conversation

nvme: blktests bug fix for 6.19

PATCH Superseded Chaitanya Kulkarni 3 participants 7 messages Nov 23, 2025 – Nov 24, 2025 NVMe

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover nvme: blktests bug fix for 6.19

Hi,

While testing my discard return value series I discovered two blktests failures, here are two fixes, below blktest testing after these patches.

-ck

Chaitanya Kulkarni (2):
nvme-tcp: use __fput_sync() to avoid use-after-free on reset
nvme-multipath: clear BIO_QOS flags on requeue

 drivers/nvme/host/multipath.c | 10 ++++++++++
 drivers/nvme/host/tcp.c       |  9 +++++----
 2 files changed, 15 insertions(+), 4 deletions(-)

nvme (nvme-6.19) # gitlog -3
0e53aeec7e61 (HEAD -> nvme-6.19) nvme-multipath: clear BIO_QOS flags on requeue d0147be50d71 nvme-tcp: use __fput_sync() to avoid use-after-free on reset 1c2ad96dbb33 (origin/nvme-6.19) nvme: Fix typo error in nvme target nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) # ./compile_nvme.sh
+ unload
+ sh ./unload-vfio-nvme.sh
rmmod: ERROR: Module drivers/vfio/pci/nvme/nvme_vfio_pci is not currently loaded
rmmod: ERROR: Module vfio_pci is not currently loaded
rmmod: ERROR: Module vfio_pci_core is not currently loaded
rmmod: ERROR: Module vfio_iommu_type1 is not currently loaded
rmmod: ERROR: Module vfio is not currently loaded ############################## UNLOAD #############################

nvme_loop              20480  0
nvmet                 221184  1 nvme_loop
nvme_tcp               90112  0
nvme_fabrics           40960  2 nvme_tcp,nvme_loop
nvme_keyring           20480  3 nvmet,nvme_tcp,nvme_fabrics
nvme                   69632  0
nvme_core             233472  5 nvmet,nvme_tcp,nvme,nvme_loop,nvme_fabrics

umount: /mnt/nvme0n1: no mount point specified.
NQN:testnqn disconnected 0 controller(s)

real 0m0.011s
user 0m0.001s
sys 0m0.010s
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory ./delete.sh: line 14: /sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file or directory
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or directory
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such file or directory
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
umount: /mnt/nvme0n1: no mount point specified.
umount: /mnt/backend: not mounted.
############################## DELETE #############################

nvme_tcp               90112  0
nvme_fabrics           40960  1 nvme_tcp
nvme_keyring           20480  2 nvme_tcp,nvme_fabrics
nvme                   69632  0
nvme_core             233472  3 nvme_tcp,nvme,nvme_fabrics
nvme_tcp               90112  0
nvme_fabrics           40960  1 nvme_tcp
nvme_keyring           20480  2 nvme_tcp,nvme_fabrics
nvme                   69632  0
nvme_core             233472  3 nvme_tcp,nvme,nvme_fabrics
nvme                   69632  0
nvme_core             233472  1 nvme
nvme                   69632  0
nvme_core             233472  1 nvme

modprobe: FATAL: Module nvme_keryring not found.
modprobe: FATAL: Module nvme_auth not found.

3 directories, 0 files
############################## UNLOAD ############################# ### nvme_loop unload
### nvmet unload
### nvme_tcp unload
### nvme_fabrics unload
### nvme unload
### nvme_core unload
### nvme_keryring unload
### nvme_auth unload
modprobe: FATAL: Module nvme_auth not found.

make[1]: Entering directory '/mnt/data100G/nvme/drivers/nvme'
make[1]: Leaving directory '/mnt/data100G/nvme/drivers/nvme'
+ ls -lrth /lib/modules/6.17.0-rc3nvme+/kernel/drivers/nvme/host/ /lib/modules/6.17.0-rc3nvme+/kernel/drivers/nvme/target// /lib/modules/6.17.0-rc3nvme+/kernel/drivers/nvme/host/:
total 7.9M
-rw-r--r--. 1 root root 4.2M Nov 23 10:21 nvme-core.ko
-rw-r--r--. 1 root root 592K Nov 23 10:21 nvme-fabrics.ko
-rw-r--r--. 1 root root 1.2M Nov 23 10:21 nvme-fc.ko
-rw-r--r--. 1 root root 929K Nov 23 10:21 nvme.ko
-rw-r--r--. 1 root root 1.2M Nov 23 10:21 nvme-tcp.ko

/lib/modules/6.17.0-rc3nvme+/kernel/drivers/nvme/target//:
total 9.5M
-rw-r--r--. 1 root root 660K Nov 23 10:21 nvme-fcloop.ko -rw-r--r--. 1 root root 562K Nov 23 10:21 nvme-loop.ko -rw-r--r--. 1 root root 1009K Nov 23 10:21 nvmet-fc.ko

-rw-r--r--. 1 root root  4.6M Nov 23 10:21 nvmet.ko
-rw-r--r--. 1 root root  765K Nov 23 10:21 nvmet-pci-epf.ko
-rw-r--r--. 1 root root  1.1M Nov 23 10:21 nvmet-rdma.ko
-rw-r--r--. 1 root root  992K Nov 23 10:21 nvmet-tcp.ko

+ sync
+ modprobe nvme-core
+ modprobe nvme
+ modprobe nvme-fabrics
+ modprobe nvme-tcp
+ modprobe nvme_loop
+ modprobe nvmet
+ lsmod
+ grep nvme

nvme_loop              20480  0
nvmet                 221184  1 nvme_loop
nvme_tcp               90112  0
nvme_fabrics           40960  2 nvme_tcp,nvme_loop
nvme_keyring           20480  3 nvmet,nvme_tcp,nvme_fabrics
nvme                   69632  0
nvme_core             233472  5 nvmet,nvme_tcp,nvme,nvme_loop,nvme_fabrics

nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) #
nvme (nvme-6.19) # cdblktests
blktests (master) #
blktests (master) #
blktests (master) #
blktests (master) #
blktests (master) #

blktests (master) # ./test-nvme.sh
++ for t in loop tcp
++ echo '################nvme_trtype=loop############' ################nvme_trtype=loop############
++ nvme_img_size=900M
++ nvme_num_iter=1
++ nvme_trtype=loop
++ ./check nvme
nvme/002 (tr=loop) (create many subsystems and test discovery) [passed]

    runtime  40.593s  ...  41.344s
nvme/003 (tr=loop) (test if we're sending keep-alives to a discovery controller) [passed]
    runtime  10.256s  ...  10.266s
nvme/004 (tr=loop) (test nvme and nvmet UUID NS descriptors) [passed]
    runtime  0.893s  ...  0.911s
nvme/005 (tr=loop) (reset local loopback target)             [passed]
    runtime  1.449s  ...  1.482s
nvme/006 (tr=loop bd=device) (create an NVMeOF target)       [passed]
    runtime  0.119s  ...  0.112s
nvme/006 (tr=loop bd=file) (create an NVMeOF target)         [passed]
    runtime  0.082s  ...  0.078s
nvme/008 (tr=loop bd=device) (create an NVMeOF host)         [passed]
    runtime  0.907s  ...  0.909s
nvme/008 (tr=loop bd=file) (create an NVMeOF host)           [passed]
    runtime  0.879s  ...  0.867s
nvme/010 (tr=loop bd=device) (run data verification fio job) [passed]
    runtime  38.757s  ...  71.268s
nvme/010 (tr=loop bd=file) (run data verification fio job)   [passed]
    runtime  101.443s  ...  175.439s
nvme/012 (tr=loop bd=device) (run mkfs and data verification fio) [passed]
    runtime  37.683s  ...  82.689s
nvme/012 (tr=loop bd=file) (run mkfs and data verification fio) [passed]
    runtime  81.227s  ...  156.973s
nvme/014 (tr=loop bd=device) (flush a command from host)     [passed]
    runtime  6.871s  ...  15.076s
nvme/014 (tr=loop bd=file) (flush a command from host)       [passed]
    runtime  6.281s  ...  10.713s
nvme/016 (tr=loop) (create/delete many NVMeOF block device-backed ns and test discovery) [passed]
    runtime  0.165s  ...  0.162s
nvme/017 (tr=loop) (create/delete many file-ns and test discovery) [passed]
    runtime  0.164s  ...  0.162s
nvme/018 (tr=loop) (unit test NVMe-oF out of range access on a file backend) [passed]
    runtime  0.849s  ...  0.852s
nvme/019 (tr=loop bd=device) (test NVMe DSM Discard command) [passed]
    runtime  0.892s  ...  0.871s
nvme/019 (tr=loop bd=file) (test NVMe DSM Discard command)   [passed]
    runtime  0.865s  ...  0.849s
nvme/021 (tr=loop bd=device) (test NVMe list command)        [passed]
    runtime  0.971s  ...  0.887s
nvme/021 (tr=loop bd=file) (test NVMe list command)          [passed]
    runtime  0.903s  ...  0.857s
nvme/022 (tr=loop bd=device) (test NVMe reset command)       [passed]
    runtime  1.506s  ...  1.507s
nvme/022 (tr=loop bd=file) (test NVMe reset command)         [passed]
    runtime  1.441s  ...  1.450s
nvme/023 (tr=loop bd=device) (test NVMe smart-log command)   [passed]
    runtime  0.887s  ...  0.871s
nvme/023 (tr=loop bd=file) (test NVMe smart-log command)     [passed]
    runtime  0.849s  ...  0.872s
nvme/025 (tr=loop bd=device) (test NVMe effects-log)         [passed]
    runtime  0.883s  ...  0.916s
nvme/025 (tr=loop bd=file) (test NVMe effects-log)           [passed]
    runtime  0.855s  ...  0.862s
nvme/026 (tr=loop bd=device) (test NVMe ns-descs)            [passed]
    runtime  0.874s  ...  0.892s
nvme/026 (tr=loop bd=file) (test NVMe ns-descs)              [passed]
    runtime  0.857s  ...  0.895s
nvme/027 (tr=loop bd=device) (test NVMe ns-rescan command)   [passed]
    runtime  0.910s  ...  0.930s
nvme/027 (tr=loop bd=file) (test NVMe ns-rescan command)     [passed]
    runtime  0.887s  ...  0.891s
nvme/028 (tr=loop bd=device) (test NVMe list-subsys)         [passed]
    runtime  0.884s  ...  0.899s
nvme/028 (tr=loop bd=file) (test NVMe list-subsys)           [passed]
    runtime  0.858s  ...  0.851s
nvme/029 (tr=loop) (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  1.111s  ...  1.108s
nvme/030 (tr=loop) (ensure the discovery generation counter is updated appropriately) [passed]
    runtime  0.524s  ...  0.551s
nvme/031 (tr=loop) (test deletion of NVMeOF controllers immediately after setup) [passed]
    runtime  8.303s  ...  8.372s
nvme/038 (tr=loop) (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.029s  ...  0.029s
nvme/040 (tr=loop) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
    runtime  7.817s  ...  7.811s

nvme/041 (tr=loop) (Create authenticated connections) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/042 (tr=loop) (Test dhchap key types for authenticated connections) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/043 (tr=loop) (Test hash and DH group variations for authenticated connections) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/044 (tr=loop) (Test bi-directional authentication) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/045 (tr=loop) (Test re-authentication) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/047 (tr=loop) (test different queue types for fabric transports) [not run]
nvme_trtype=loop is not supported in this test
nvme/048 (tr=loop) (Test queue count changes on reconnect) [not run]
nvme_trtype=loop is not supported in this test
nvme/051 (tr=loop) (test nvmet concurrent ns enable/disable) [passed]

    runtime  2.157s  ...  2.184s
nvme/052 (tr=loop) (Test file-ns creation/deletion under one subsystem) [passed]
    runtime  6.443s  ...  6.458s
nvme/054 (tr=loop) (Test the NVMe reservation feature)       [passed]
    runtime  0.937s  ...  0.933s
nvme/055 (tr=loop) (Test nvme write to a loop target ns just after ns is disabled) [passed]
    runtime  0.917s  ...  0.924s

nvme/056 (tr=loop) (enable zero copy offload and run rw traffic) [not run]
Remote target required but NVME_TARGET_CONTROL is not set nvme_trtype=loop is not supported in this test
kernel option ULP_DDP has not been enabled
module nvme_tcp does not have parameter ddp_offload
KERNELSRC not set
Kernel sources do not have tools/net/ynl/cli.py
NVME_IFACE not set
nvme/057 (tr=loop) (test nvme fabrics controller ANA failover during I/O) [passed]

    runtime  37.234s  ...  29.876s
nvme/058 (tr=loop) (test rapid namespace remapping)          [passed]
    runtime  8.093s  ...  7.385s

nvme/060 (tr=loop) (test nvme fabrics target reset) [not run]
nvme_trtype=loop is not supported in this test
nvme/061 (tr=loop) (test fabric target teardown and setup during I/O) [not run]
nvme_trtype=loop is not supported in this test
nvme/062 (tr=loop) (Create TLS-encrypted connections) [not run]
nvme_trtype=loop is not supported in this test
command tlshd is not available
systemctl unit 'tlshd' is missing
Install ktls-utils for tlshd
nvme/063 (tr=loop) (Create authenticated TCP connections with secure concatenation) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme_trtype=loop is not supported in this test
command tlshd is not available
systemctl unit 'tlshd' is missing
Install ktls-utils for tlshd
nvme/065 (test unmap write zeroes sysfs interface with nvmet devices) [not run]
driver scsi_debug is not available
++ for t in loop tcp
++ echo '################nvme_trtype=tcp############' ################nvme_trtype=tcp############
++ nvme_img_size=900M
++ nvme_num_iter=1
++ nvme_trtype=tcp
++ ./check nvme
nvme/002 (tr=tcp) (create many subsystems and test discovery) [not run]
nvme_trtype=tcp is not supported in this test
nvme/003 (tr=tcp) (test if we're sending keep-alives to a discovery controller) [passed]

    runtime  10.267s  ...  10.260s
nvme/004 (tr=tcp) (test nvme and nvmet UUID NS descriptors)  [passed]
    runtime  0.405s  ...  0.410s
nvme/005 (tr=tcp) (reset local loopback target)              [passed]
    runtime  0.507s  ...  0.505s
nvme/006 (tr=tcp bd=device) (create an NVMeOF target)        [passed]
    runtime  0.121s  ...  0.131s
nvme/006 (tr=tcp bd=file) (create an NVMeOF target)          [passed]
    runtime  0.096s  ...  0.093s
nvme/008 (tr=tcp bd=device) (create an NVMeOF host)          [passed]
    runtime  0.396s  ...  0.408s
nvme/008 (tr=tcp bd=file) (create an NVMeOF host)            [passed]
    runtime  0.378s  ...  0.384s
nvme/010 (tr=tcp bd=device) (run data verification fio job)  [passed]
    runtime  3.971s  ...  101.482s
nvme/010 (tr=tcp bd=file) (run data verification fio job)    [passed]
    runtime  0.984s  ...  149.695s
nvme/012 (tr=tcp bd=device) (run mkfs and data verification fio) [passed]
    runtime  2.220s  ...  111.416s
nvme/012 (tr=tcp bd=file) (run mkfs and data verification fio) [passed]
    runtime    ...  144.534s
nvme/014 (tr=tcp bd=device) (flush a command from host)      [passed]
    runtime  6.669s  ...  12.069s
nvme/014 (tr=tcp bd=file) (flush a command from host)        [passed]
    runtime  6.066s  ...  10.806s

nvme/016 (tr=tcp) (create/delete many NVMeOF block device-backed ns and test discovery) [not run]
nvme_trtype=tcp is not supported in this test
nvme/017 (tr=tcp) (create/delete many file-ns and test discovery) [not run]
nvme_trtype=tcp is not supported in this test
nvme/018 (tr=tcp) (unit test NVMe-oF out of range access on a file backend) [passed]

    runtime  0.374s  ...  0.382s
nvme/019 (tr=tcp bd=device) (test NVMe DSM Discard command)  [passed]
    runtime  0.410s  ...  0.414s
nvme/019 (tr=tcp bd=file) (test NVMe DSM Discard command)    [passed]
    runtime  0.362s  ...  0.383s
nvme/021 (tr=tcp bd=device) (test NVMe list command)         [passed]
    runtime  0.422s  ...  0.413s
nvme/021 (tr=tcp bd=file) (test NVMe list command)           [passed]
    runtime  0.383s  ...  0.383s
nvme/022 (tr=tcp bd=device) (test NVMe reset command)        [passed]
    runtime  0.522s  ...  0.510s
nvme/022 (tr=tcp bd=file) (test NVMe reset command)          [passed]
    runtime  0.473s  ...  0.488s
nvme/023 (tr=tcp bd=device) (test NVMe smart-log command)    [passed]
    runtime  0.405s  ...  0.390s
nvme/023 (tr=tcp bd=file) (test NVMe smart-log command)      [passed]
    runtime  0.368s  ...  0.350s
nvme/025 (tr=tcp bd=device) (test NVMe effects-log)          [passed]
    runtime  0.421s  ...  0.436s
nvme/025 (tr=tcp bd=file) (test NVMe effects-log)            [passed]
    runtime  0.385s  ...  0.382s
nvme/026 (tr=tcp bd=device) (test NVMe ns-descs)             [passed]
    runtime  0.407s  ...  0.397s
nvme/026 (tr=tcp bd=file) (test NVMe ns-descs)               [passed]
    runtime  0.354s  ...  0.367s
nvme/027 (tr=tcp bd=device) (test NVMe ns-rescan command)    [passed]
    runtime  0.432s  ...  0.446s
nvme/027 (tr=tcp bd=file) (test NVMe ns-rescan command)      [passed]
    runtime  0.411s  ...  0.407s
nvme/028 (tr=tcp bd=device) (test NVMe list-subsys)          [passed]
    runtime  0.395s  ...  0.393s
nvme/028 (tr=tcp bd=file) (test NVMe list-subsys)            [passed]
    runtime  0.367s  ...  0.362s
nvme/029 (tr=tcp) (test userspace IO via nvme-cli read/write interface) [passed]
    runtime  0.626s  ...  0.636s
nvme/030 (tr=tcp) (ensure the discovery generation counter is updated appropriately) [passed]
    runtime  0.376s  ...  0.384s
nvme/031 (tr=tcp) (test deletion of NVMeOF controllers immediately after setup) [passed]
    runtime  3.170s  ...  3.217s
nvme/038 (tr=tcp) (test deletion of NVMeOF subsystem without enabling) [passed]
    runtime  0.035s  ...  0.036s
nvme/040 (tr=tcp) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
    runtime  6.481s  ...  6.484s

nvme/041 (tr=tcp) (Create authenticated connections) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/042 (tr=tcp) (Test dhchap key types for authenticated connections) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/043 (tr=tcp) (Test hash and DH group variations for authenticated connections) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/044 (tr=tcp) (Test bi-directional authentication) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/045 (tr=tcp) (Test re-authentication) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
nvme/047 (tr=tcp) (test different queue types for fabric transports) [passed]

    runtime  1.943s  ...  1.936s
nvme/048 (tr=tcp) (Test queue count changes on reconnect)    [passed]
    runtime  5.548s  ...  5.519s
nvme/051 (tr=tcp) (test nvmet concurrent ns enable/disable)  [passed]
    runtime  2.197s  ...  2.234s
nvme/052 (tr=tcp) (Test file-ns creation/deletion under one subsystem) [not run]
    nvme_trtype=tcp is not supported in this test
nvme/054 (tr=tcp) (Test the NVMe reservation feature)        [passed]
    runtime  0.477s  ...  0.453s

nvme/055 (tr=tcp) (Test nvme write to a loop target ns just after ns is disabled) [not run]
nvme_trtype=tcp is not supported in this test
nvme/056 (tr=tcp) (enable zero copy offload and run rw traffic) [not run]
Remote target required but NVME_TARGET_CONTROL is not set
kernel option ULP_DDP has not been enabled
module nvme_tcp does not have parameter ddp_offload
KERNELSRC not set
Kernel sources do not have tools/net/ynl/cli.py
NVME_IFACE not set
nvme/057 (tr=tcp) (test nvme fabrics controller ANA failover during I/O) [passed]

    runtime  26.300s  ...  31.214s
nvme/058 (tr=tcp) (test rapid namespace remapping)           [passed]
    runtime  4.343s  ...  7.080s
nvme/060 (tr=tcp) (test nvme fabrics target reset)           [passed]
    runtime  19.562s  ...  19.475s
nvme/061 (tr=tcp) (test fabric target teardown and setup during I/O) [passed]
    runtime  8.648s  ...  8.586s

nvme/062 (tr=tcp) (Create TLS-encrypted connections) [not run]
command tlshd is not available
systemctl unit 'tlshd' is missing
Install ktls-utils for tlshd
nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation) [not run]
kernel option NVME_AUTH has not been enabled
kernel option NVME_TARGET_AUTH has not been enabled
nvme-fabrics does not support dhchap_ctrl_secret
command tlshd is not available
systemctl unit 'tlshd' is missing
Install ktls-utils for tlshd
nvme/065 (test unmap write zeroes sysfs interface with nvmet devices) [not run]
driver scsi_debug is not available
blktests (master) #

Signature · 1 line
2.40.0
#1 lore raw

PATCH 1/2 nvme-tcp: use __fput_sync() to avoid use-after-free on reset

When a controller reset is triggered via sysfs (by writing to /sys/class/nvme/<nvmedev>/reset_controller), the reset work tears down and re-establishes all queues. The socket release using fput() defers the actual cleanup to task_work delayed_fput workqueue. This deferred cleanup can race with the subsequent queue re-allocation during reset, potentially leading to use-after-free or resource conflicts.

Replace fput() with __fput_sync() to ensure synchronous socket release, guaranteeing that all socket resources are fully cleaned up before the function returns. This prevents races during controller reset where new queue setup may begin before the old socket is fully released.

* Call chain during reset:
nvme_reset_ctrl_work()

    -> nvme_tcp_teardown_ctrl()
      -> nvme_tcp_teardown_io_queues()
        -> nvme_tcp_free_io_queues()
          -> nvme_tcp_free_queue()       <-- fput() -> __fput_sync()
      -> nvme_tcp_teardown_admin_queue()
        -> nvme_tcp_free_admin_queue()
          -> nvme_tcp_free_queue()       <-- fput() -> __fput_sync()
    -> nvme_tcp_setup_ctrl()             <-- race with deferred fput

* The issue can be reproduced using blktests:

nvme_trtype=tcp ./check nvme/005
blktests (master) # nvme_trtype=tcp ./check nvme/005
nvme/005 (tr=tcp) (reset local loopback target) [failed]

    runtime  0.725s  ...  0.798s
    something found in dmesg:
    [  108.473940] run blktests nvme/005 at 2025-11-22 16:12:20

    [...]

    (See '/root/blktests/results/nodev_tr_tcp/nvme/005.dmesg' for the entire message)
blktests (master) # cat /root/blktests/results/nodev_tr_tcp/nvme/005.dmesg
[  108.473940] run blktests nvme/005 at 2025-11-22 16:12:20
[  108.526983] loop0: detected capacity change from 0 to 2097152
[  108.555606] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  108.572531] nvmet_tcp: enabling port 0 (127.0.0.1:4420)
[  108.613061] nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[  108.616832] nvme nvme0: creating 48 I/O queues.
[  108.630791] nvme nvme0: mapped 48/0/0 default/read/poll queues.
[  108.661892] nvme nvme0: new ctrl: NQN "blktests-subsystem-1", addr 127.0.0.1:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
[  108.746639] nvmet: Created nvm controller 2 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
[  108.748466] nvme nvme0: creating 48 I/O queues.
[  108.802984] nvme nvme0: mapped 48/0/0 default/read/poll queues.
[  108.829983] nvme nvme0: Removing ctrl: NQN "blktests-subsystem-1"
[  108.854288] block nvme0n1: no available path - failing I/O
[  108.854344] block nvme0n1: no available path - failing I/O
[  108.854373] Buffer I/O error on dev nvme0n1, logical block 1, async page read
[  108.891693] ======================================================
[  108.895912] WARNING: possible circular locking dependency detected
[  108.900184] 6.17.0nvme+ #3 Tainted: G                 N
[  108.903913] ------------------------------------------------------
[  108.908171] nvme/2734 is trying to acquire lock:
[  108.911957] ffff88810210e610 (set->srcu){.+.+}-{0:0}, at: __synchronize_srcu+0x17/0x170
[  108.917587]
               but task is already holding lock:
[  108.921570] ffff88813abea198 (&q->elevator_lock){+.+.}-{4:4}, at: elevator_change+0xa8/0x1c0
[  108.927361]
               which lock already depends on the new lock.
[  108.933018]
               the existing dependency chain (in reverse order) is:
[  108.938223]
               -> #4 (&q->elevator_lock){+.+.}-{4:4}:
[  108.942988]        __mutex_lock+0xa2/0x1150
[  108.945873]        elevator_change+0xa8/0x1c0
[  108.948925]        elv_iosched_store+0xdf/0x140
[  108.952043]        kernfs_fop_write_iter+0x16a/0x220
[  108.955367]        vfs_write+0x378/0x520
[  108.957598]        ksys_write+0x67/0xe0
[  108.959721]        do_syscall_64+0x76/0xbb0
[  108.962052]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  108.965145]
               -> #3 (&q->q_usage_counter(io)){++++}-{0:0}:
[  108.968923]        blk_alloc_queue+0x30e/0x350
[  108.972117]        blk_mq_alloc_queue+0x61/0xd0
[  108.974677]        scsi_alloc_sdev+0x2a0/0x3e0
[  108.977092]        scsi_probe_and_add_lun+0x1bd/0x430
[  108.979921]        __scsi_add_device+0x109/0x120
[  108.982504]        ata_scsi_scan_host+0x97/0x1c0
[  108.984365]        async_run_entry_fn+0x2d/0x130
[  108.986109]        process_one_work+0x20e/0x630
[  108.987830]        worker_thread+0x184/0x330
[  108.989473]        kthread+0x10a/0x250
[  108.990852]        ret_from_fork+0x297/0x300
[  108.992491]        ret_from_fork_asm+0x1a/0x30
[  108.994159]
               -> #2 (fs_reclaim){+.+.}-{0:0}:
[  108.996320]        fs_reclaim_acquire+0x99/0xd0
[  108.998058]        kmem_cache_alloc_node_noprof+0x4e/0x3c0
[  109.000123]        __alloc_skb+0x15f/0x190
[  109.002195]        tcp_send_active_reset+0x3f/0x1e0
[  109.004038]        tcp_disconnect+0x50b/0x720
[  109.005695]        __tcp_close+0x2b8/0x4b0
[  109.007227]        tcp_close+0x20/0x80
[  109.008663]        inet_release+0x31/0x60
[  109.010175]        __sock_release+0x3a/0xc0
[  109.011778]        sock_close+0x14/0x20
[  109.013263]        __fput+0xee/0x2c0
[  109.014673]        delayed_fput+0x31/0x50
[  109.016183]        process_one_work+0x20e/0x630
[  109.017897]        worker_thread+0x184/0x330
[  109.019543]        kthread+0x10a/0x250
[  109.020929]        ret_from_fork+0x297/0x300
[  109.022565]        ret_from_fork_asm+0x1a/0x30
[  109.024194]
               -> #1 (sk_lock-AF_INET-NVME){+.+.}-{0:0}:
[  109.026634]        lock_sock_nested+0x2e/0x70
[  109.028251]        tcp_sendmsg+0x1a/0x40
[  109.029783]        sock_sendmsg+0xed/0x110
[  109.031321]        nvme_tcp_try_send_cmd_pdu+0x13e/0x260 [nvme_tcp]
[  109.034263]        nvme_tcp_try_send+0xb3/0x330 [nvme_tcp]
[  109.036375]        nvme_tcp_queue_rq+0x342/0x3d0 [nvme_tcp]
[  109.038528]        blk_mq_dispatch_rq_list+0x297/0x800
[  109.040448]        __blk_mq_sched_dispatch_requests+0x3db/0x5f0
[  109.042677]        blk_mq_sched_dispatch_requests+0x29/0x70
[  109.044787]        blk_mq_run_work_fn+0x76/0x1b0
[  109.046535]        process_one_work+0x20e/0x630
[  109.048245]        worker_thread+0x184/0x330
[  109.049890]        kthread+0x10a/0x250
[  109.051331]        ret_from_fork+0x297/0x300
[  109.053024]        ret_from_fork_asm+0x1a/0x30
[  109.054740]
               -> #0 (set->srcu){.+.+}-{0:0}:
[  109.056850]        __lock_acquire+0x1468/0x2210
[  109.058614]        lock_sync+0xa5/0x110
[  109.060048]        __synchronize_srcu+0x49/0x170
[  109.061802]        elevator_switch+0xc9/0x330
[  109.063950]        elevator_change+0x128/0x1c0
[  109.065675]        elevator_set_none+0x4c/0x90
[  109.067316]        blk_unregister_queue+0xa8/0x110
[  109.069165]        __del_gendisk+0x14e/0x3c0
[  109.070824]        del_gendisk+0x75/0xa0
[  109.072328]        nvme_ns_remove+0xf2/0x230 [nvme_core]
[  109.074365]        nvme_remove_namespaces+0xf2/0x150 [nvme_core]
[  109.076652]        nvme_do_delete_ctrl+0x71/0x90 [nvme_core]
[  109.078775]        nvme_delete_ctrl_sync+0x3b/0x50 [nvme_core]
[  109.081009]        nvme_sysfs_delete+0x34/0x40 [nvme_core]
[  109.083082]        kernfs_fop_write_iter+0x16a/0x220
[  109.085009]        vfs_write+0x378/0x520
[  109.086539]        ksys_write+0x67/0xe0
[  109.087982]        do_syscall_64+0x76/0xbb0
[  109.089577]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  109.091665]
               other info that might help us debug this:
[  109.095478] Chain exists of:
                 set->srcu --> &q->q_usage_counter(io) --> &q->elevator_lock
[  109.099544]  Possible unsafe locking scenario:
[  109.101708]        CPU0                    CPU1
[  109.103402]        ----                    ----
[  109.105103]   lock(&q->elevator_lock);
[  109.106530]                                lock(&q->q_usage_counter(io));
[  109.109022]                                lock(&q->elevator_lock);
[  109.111391]   sync(set->srcu);
[  109.112586]
                *** DEADLOCK ***
[  109.114772] 5 locks held by nvme/2734:
[  109.116189]  #0: ffff888101925410 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x67/0xe0
[  109.119143]  #1: ffff88817a914e88 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x10f/0x220
[  109.123141]  #2: ffff8881046313f8 (kn->active#185){++++}-{0:0}, at: sysfs_remove_file_self+0x26/0x50
[  109.126543]  #3: ffff88810470e1d0 (&set->update_nr_hwq_lock){++++}-{4:4}, at: del_gendisk+0x6d/0xa0
[  109.129891]  #4: ffff88813abea198 (&q->elevator_lock){+.+.}-{4:4}, at: elevator_change+0xa8/0x1c0
[  109.133149]
               stack backtrace:
[  109.134817] CPU: 6 UID: 0 PID: 2734 Comm: nvme Tainted: G                 N  6.17.0nvme+ #3 PREEMPT(voluntary)
[  109.134819] Tainted: [N]=TEST
[  109.134820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[  109.134821] Call Trace:
[  109.134823]  <TASK>
[  109.134824]  dump_stack_lvl+0x75/0xb0
[  109.134828]  print_circular_bug+0x26a/0x330
[  109.134831]  check_noncircular+0x12f/0x150
[  109.134834]  __lock_acquire+0x1468/0x2210
[  109.134837]  ? __synchronize_srcu+0x17/0x170
[  109.134838]  lock_sync+0xa5/0x110
[  109.134840]  ? __synchronize_srcu+0x17/0x170
[  109.134842]  __synchronize_srcu+0x49/0x170
[  109.134843]  ? mark_held_locks+0x49/0x80
[  109.134845]  ? _raw_spin_unlock_irqrestore+0x2d/0x60
[  109.134847]  ? kvm_clock_get_cycles+0x14/0x30
[  109.134853]  ? ktime_get_mono_fast_ns+0x36/0xb0
[  109.134858]  elevator_switch+0xc9/0x330
[  109.134860]  elevator_change+0x128/0x1c0
[  109.134862]  ? kernfs_put.part.0+0x86/0x290
[  109.134864]  elevator_set_none+0x4c/0x90
[  109.134866]  blk_unregister_queue+0xa8/0x110
[  109.134868]  __del_gendisk+0x14e/0x3c0
[  109.134870]  del_gendisk+0x75/0xa0
[  109.134872]  nvme_ns_remove+0xf2/0x230 [nvme_core]
[  109.134879]  nvme_remove_namespaces+0xf2/0x150 [nvme_core]
[  109.134887]  nvme_do_delete_ctrl+0x71/0x90 [nvme_core]
[  109.134893]  nvme_delete_ctrl_sync+0x3b/0x50 [nvme_core]
[  109.134899]  nvme_sysfs_delete+0x34/0x40 [nvme_core]
[  109.134905]  kernfs_fop_write_iter+0x16a/0x220
[  109.134908]  vfs_write+0x378/0x520
[  109.134911]  ksys_write+0x67/0xe0
[  109.134913]  do_syscall_64+0x76/0xbb0
[  109.134915]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  109.134916] RIP: 0033:0x7fd68a737317
[  109.134917] Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[  109.134919] RSP: 002b:00007ffded1546d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[  109.134920] RAX: ffffffffffffffda RBX: 000000000054f7e0 RCX: 00007fd68a737317
[  109.134921] RDX: 0000000000000001 RSI: 00007fd68a855719 RDI: 0000000000000003
[  109.134921] RBP: 0000000000000003 R08: 0000000030407850 R09: 00007fd68a7cd4e0
[  109.134922] R10: 00007fd68a65b130 R11: 0000000000000246 R12: 00007fd68a855719
[  109.134923] R13: 00000000304074c0 R14: 00000000304074c0 R15: 0000000030408660
[  109.134926]  </TASK>
[  109.962756] Key type psk unregistered
---
 drivers/nvme/host/tcp.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +5 -4
index 1413788ca7d5..ba23ca465591 100644
@@ -9,6 +9,7 @@
 #include <linux/slab.h>
 #include <linux/err.h>
 #include <linux/crc32.h>
+#include <linux/file.h>
 #include <linux/nvme-tcp.h>
 #include <linux/nvme-keyring.h>
 #include <net/sock.h>
@@ -1442,8 +1443,8 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	page_frag_cache_drain(&queue->pf_cache);
 
 	noreclaim_flag = memalloc_noreclaim_save();
-	/* ->sock will be released by fput() */
-	fput(queue->sock->file);
+	/* ->sock will be released by __fput_sync() */
+	__fput_sync(queue->sock->file);
 	queue->sock = NULL;
 	memalloc_noreclaim_restore(noreclaim_flag);
 
@@ -1897,8 +1898,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 err_rcv_pdu:
 	kfree(queue->pdu);
 err_sock:
-	/* ->sock will be released by fput() */
-	fput(queue->sock->file);
+	/* ->sock will be released by __fput_sync() */
+	__fput_sync(queue->sock->file);
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
Signature · 1 line
2.40.0
#2 lore raw

Re: [PATCH BUF FIX 1/2] nvme-tcp: use __fput_sync() to avoid use-after-free on reset

Please add a comment why you need the sync variant. Otherwise this looks sane to me.

#3 lore raw

PATCH 2/2 nvme-multipath: clear BIO_QOS flags on requeue

In reply to Chaitanya Kulkarni

When a bio goes through the rq_qos infrastructure on a path's request queue, it gets BIO_QOS_THROTTLED or BIO_QOS_MERGED flags set. These flags indicate that rq_qos_done_bio() should be called on completion to update rq_qos accounting.

During path failover in nvme_failover_req(), the bio's bi_bdev is redirected from the failed path's disk to the multipath head's disk via bio_set_dev(). However, the BIO_QOS flags are not cleared.

When the bio eventually completes (either successfully via a new path or with an error via bio_io_error()), rq_qos_done_bio() checks for these flags and calls __rq_qos_done_bio(q->rq_qos, bio) where q is obtained from the bio's current bi_bdev - which is now the multipath head's queue, not the original path's queue.

The multipath head's queue does not have rq_qos enabled (q->rq_qos is NULL), but the code assumes that if BIO_QOS_* flags are set, q->rq_qos must be valid. This assumption is documented in block/blk-rq-qos.h:

"If a bio has BIO_QOS_xxx set, it implicitly implies that
q->rq_qos is present."

This breaks when a bio is moved between queues during NVMe multipath failover, leading to a NULL pointer dereference.

Execution Context timeline :-

* =====> dd process context
[USER] dd process
[SYSCALL] write() - dd process context
submit_bio() nvme_ns_head_submit_bio() - path selection blk_mq_submit_bio() #### QOS FLAGS SET HERE

[USER] dd waits or returns

==== I/O in flight on NVMe hardware =====

===== End of submission path ====
------------------------------------------------------

* dd ====> Interrupt context;
[IRQ] NVMe completion interrupt
nvme_irq()
nvme_complete_rq()
nvme_failover_req() ### BIO MOVED TO HEAD
spin_lock_irqsave (atomic section) bio_set_dev() changes bi_bdev ### BUG: QOS flags NOT cleared kblockd_schedule_work()

* Interrupt context =====> kblockd workqueue
[WQ] kblockd workqueue - kworker process

       nvme_requeue_work()                       
        submit_bio_noacct()                       
         nvme_ns_head_submit_bio()                 
          nvme_find_path() returns NULL             
           bio_io_error()                            
            bio_endio()                               
             rq_qos_done_bio()  ### CRASH ###

KERNEL PANIC / OOPS

Crash from blktests nvme/058 (rapid namespace remapping):

[ 1339.636033] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 1339.641025] nvme nvme4: rescanning namespaces.
[ 1339.642064] #PF: supervisor read access in kernel mode
[ 1339.642067] #PF: error_code(0x0000) - not-present page
[ 1339.642070] PGD 0 P4D 0
[ 1339.642073] Oops: Oops: 0000 [#1] SMP NOPTI
[ 1339.642078] CPU: 35 UID: 0 PID: 4579 Comm: kworker/35:2H
               Tainted: G   O     N  6.17.0-rc3nvme+ #5 PREEMPT(voluntary)
[ 1339.642084] Tainted: [O]=OOT_MODULE, [N]=TEST
[ 1339.673446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
	       BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 1339.682359] Workqueue: kblockd nvme_requeue_work [nvme_core]
[ 1339.686613] RIP: 0010:__rq_qos_done_bio+0xd/0x40
[ 1339.690161] Code: 75 dd 5b 5d 41 5c c3 cc cc cc cc 66 90 90 90 90 90 90 90
                     90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 f5
		     53 48 89 fb <48> 8b 03 48 8b 40 30 48 85 c0 74 0b 48 89 ee
		     48 89 df ff d0 0f 1f
[ 1339.703691] RSP: 0018:ffffc900066f3c90 EFLAGS: 00010202
[ 1339.706844] RAX: ffff888148b9ef00 RBX: 0000000000000000 RCX: 0000000000000000
[ 1339.711136] RDX: 00000000000001c0 RSI: ffff8882aaab8a80 RDI: 0000000000000000
[ 1339.715691] RBP: ffff8882aaab8a80 R08: 0000000000000000 R09: 0000000000000000
[ 1339.720472] R10: 0000000000000000 R11: fefefefefefefeff R12: ffff8882aa3b6010
[ 1339.724650] R13: 0000000000000000 R14: ffff8882338bcef0 R15: ffff8882aa3b6020
[ 1339.729029] FS:  0000000000000000(0000) GS:ffff88985c0cf000(0000) knlGS:0000000000000000
[ 1339.734525] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1339.738563] CR2: 0000000000000000 CR3: 0000000111045000 CR4: 0000000000350ef0
[ 1339.742750] DR0: ffffffff845ccbec DR1: ffffffff845ccbed DR2: ffffffff845ccbee
[ 1339.745630] DR3: ffffffff845ccbef DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 1339.748488] Call Trace:
[ 1339.749512]  <TASK>
[ 1339.750449]  bio_endio+0x71/0x2e0
[ 1339.751833]  nvme_ns_head_submit_bio+0x290/0x320 [nvme_core]
[ 1339.754073]  __submit_bio+0x222/0x5e0
[ 1339.755623]  ? rcu_is_watching+0xd/0x40
[ 1339.757201]  ? submit_bio_noacct_nocheck+0x131/0x370
[ 1339.759210]  submit_bio_noacct_nocheck+0x131/0x370
[ 1339.761189]  ? submit_bio_noacct+0x20/0x620
[ 1339.762849]  nvme_requeue_work+0x4b/0x60 [nvme_core]
[ 1339.764828]  process_one_work+0x20e/0x630
[ 1339.766528]  worker_thread+0x184/0x330
[ 1339.768129]  ? __pfx_worker_thread+0x10/0x10
[ 1339.769942]  kthread+0x10a/0x250
[ 1339.771263]  ? __pfx_kthread+0x10/0x10
[ 1339.772776]  ? __pfx_kthread+0x10/0x10
[ 1339.774381]  ret_from_fork+0x273/0x2e0
[ 1339.775948]  ? __pfx_kthread+0x10/0x10
[ 1339.777504]  ret_from_fork_asm+0x1a/0x30
[ 1339.779163]  </TASK>

Fix this by clearing both BIO_QOS_THROTTLED and BIO_QOS_MERGED flags when bios are redirected to the multipath head in nvme_failover_req(). This is consistent with the existing code that clears REQ_POLLED and REQ_NOWAIT flags when the bio changes queues.

---
 drivers/nvme/host/multipath.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/nvme/host/multipath.c +10 -0
index 3da980dc60d9..2535dba8ce1e 100644
@@ -168,6 +168,16 @@ void nvme_failover_req(struct request *req)
 		 * the flag to avoid spurious EAGAIN I/O failures.
 		 */
 		bio->bi_opf &= ~REQ_NOWAIT;
+		/*
+		 * BIO_QOS_THROTTLED and BIO_QOS_MERGED were set when the bio
+		 * went through the path's request queue rq_qos infrastructure.
+		 * The bio is now being redirected to the multipath head's
+		 * queue which may not have rq_qos enabled, so these flags are
+		 * no longer valid and must be cleared to prevent
+		 * rq_qos_done_bio() from dereferencing a NULL q->rq_qos.
+		 */
+		bio_clear_flag(bio, BIO_QOS_THROTTLED);
+		bio_clear_flag(bio, BIO_QOS_MERGED);
 	}
 	blk_steal_bios(&ns->head->requeue_list, req);
 	spin_unlock_irqrestore(&ns->head->requeue_lock, flags);
Signature · 1 line
2.40.0
#4 lore raw

Re: [PATCH BUG FIX 2/2] nvme-multipath: clear BIO_QOS flags on requeue

Quoted 132 lines · Chaitanya Kulkarni

When a bio goes through the rq_qos infrastructure on a path's request queue, it gets BIO_QOS_THROTTLED or BIO_QOS_MERGED flags set. These flags indicate that rq_qos_done_bio() should be called on completion to update rq_qos accounting.

During path failover in nvme_failover_req(), the bio's bi_bdev is redirected from the failed path's disk to the multipath head's disk via bio_set_dev(). However, the BIO_QOS flags are not cleared.

When the bio eventually completes (either successfully via a new path or with an error via bio_io_error()), rq_qos_done_bio() checks for these flags and calls __rq_qos_done_bio(q->rq_qos, bio) where q is obtained from the bio's current bi_bdev - which is now the multipath head's queue, not the original path's queue.

The multipath head's queue does not have rq_qos enabled (q->rq_qos is NULL), but the code assumes that if BIO_QOS_* flags are set, q->rq_qos must be valid. This assumption is documented in block/blk-rq-qos.h:

"If a bio has BIO_QOS_xxx set, it implicitly implies that
q->rq_qos is present."

This breaks when a bio is moved between queues during NVMe multipath failover, leading to a NULL pointer dereference.

Execution Context timeline :-

* =====> dd process context
[USER] dd process
[SYSCALL] write() - dd process context
submit_bio() nvme_ns_head_submit_bio() - path selection blk_mq_submit_bio() #### QOS FLAGS SET HERE

[USER] dd waits or returns

==== I/O in flight on NVMe hardware =====

===== End of submission path ====
------------------------------------------------------

* dd ====> Interrupt context;
[IRQ] NVMe completion interrupt
nvme_irq()
nvme_complete_rq()
nvme_failover_req() ### BIO MOVED TO HEAD
spin_lock_irqsave (atomic section) bio_set_dev() changes bi_bdev ### BUG: QOS flags NOT cleared kblockd_schedule_work()

* Interrupt context =====> kblockd workqueue
[WQ] kblockd workqueue - kworker process

       nvme_requeue_work()                       
        submit_bio_noacct()                       
         nvme_ns_head_submit_bio()                 
          nvme_find_path() returns NULL             
           bio_io_error()                            
            bio_endio()                               
             rq_qos_done_bio()  ### CRASH ###

KERNEL PANIC / OOPS

Crash from blktests nvme/058 (rapid namespace remapping):

[ 1339.636033] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 1339.641025] nvme nvme4: rescanning namespaces.
[ 1339.642064] #PF: supervisor read access in kernel mode
[ 1339.642067] #PF: error_code(0x0000) - not-present page
[ 1339.642070] PGD 0 P4D 0
[ 1339.642073] Oops: Oops: 0000 [#1] SMP NOPTI
[ 1339.642078] CPU: 35 UID: 0 PID: 4579 Comm: kworker/35:2H
               Tainted: G   O     N  6.17.0-rc3nvme+ #5 PREEMPT(voluntary)
[ 1339.642084] Tainted: [O]=OOT_MODULE, [N]=TEST
[ 1339.673446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
	       BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[ 1339.682359] Workqueue: kblockd nvme_requeue_work [nvme_core]
[ 1339.686613] RIP: 0010:__rq_qos_done_bio+0xd/0x40
[ 1339.690161] Code: 75 dd 5b 5d 41 5c c3 cc cc cc cc 66 90 90 90 90 90 90 90
                     90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 f5
		     53 48 89 fb <48> 8b 03 48 8b 40 30 48 85 c0 74 0b 48 89 ee
		     48 89 df ff d0 0f 1f
[ 1339.703691] RSP: 0018:ffffc900066f3c90 EFLAGS: 00010202
[ 1339.706844] RAX: ffff888148b9ef00 RBX: 0000000000000000 RCX: 0000000000000000
[ 1339.711136] RDX: 00000000000001c0 RSI: ffff8882aaab8a80 RDI: 0000000000000000
[ 1339.715691] RBP: ffff8882aaab8a80 R08: 0000000000000000 R09: 0000000000000000
[ 1339.720472] R10: 0000000000000000 R11: fefefefefefefeff R12: ffff8882aa3b6010
[ 1339.724650] R13: 0000000000000000 R14: ffff8882338bcef0 R15: ffff8882aa3b6020
[ 1339.729029] FS:  0000000000000000(0000) GS:ffff88985c0cf000(0000) knlGS:0000000000000000
[ 1339.734525] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1339.738563] CR2: 0000000000000000 CR3: 0000000111045000 CR4: 0000000000350ef0
[ 1339.742750] DR0: ffffffff845ccbec DR1: ffffffff845ccbed DR2: ffffffff845ccbee
[ 1339.745630] DR3: ffffffff845ccbef DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 1339.748488] Call Trace:
[ 1339.749512]  <TASK>
[ 1339.750449]  bio_endio+0x71/0x2e0
[ 1339.751833]  nvme_ns_head_submit_bio+0x290/0x320 [nvme_core]
[ 1339.754073]  __submit_bio+0x222/0x5e0
[ 1339.755623]  ? rcu_is_watching+0xd/0x40
[ 1339.757201]  ? submit_bio_noacct_nocheck+0x131/0x370
[ 1339.759210]  submit_bio_noacct_nocheck+0x131/0x370
[ 1339.761189]  ? submit_bio_noacct+0x20/0x620
[ 1339.762849]  nvme_requeue_work+0x4b/0x60 [nvme_core]
[ 1339.764828]  process_one_work+0x20e/0x630
[ 1339.766528]  worker_thread+0x184/0x330
[ 1339.768129]  ? __pfx_worker_thread+0x10/0x10
[ 1339.769942]  kthread+0x10a/0x250
[ 1339.771263]  ? __pfx_kthread+0x10/0x10
[ 1339.772776]  ? __pfx_kthread+0x10/0x10
[ 1339.774381]  ret_from_fork+0x273/0x2e0
[ 1339.775948]  ? __pfx_kthread+0x10/0x10
[ 1339.777504]  ret_from_fork_asm+0x1a/0x30
[ 1339.779163]  </TASK>

Fix this by clearing both BIO_QOS_THROTTLED and BIO_QOS_MERGED flags when bios are redirected to the multipath head in nvme_failover_req(). This is consistent with the existing code that clears REQ_POLLED and REQ_NOWAIT flags when the bio changes queues.

---
 drivers/nvme/host/multipath.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/nvme/host/multipath.c +0 -0
index 3da980dc60d9..2535dba8ce1e 100644
@@ -168,6 +168,16 @@ void nvme_failover_req(struct request *req)
 		 * the flag to avoid spurious EAGAIN I/O failures.
 		 */
 		bio->bi_opf &= ~REQ_NOWAIT;

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 180

Quoted 10 lines
drivers/nvme/host/multipath.c +10 -0
+		/*
+		 * BIO_QOS_THROTTLED and BIO_QOS_MERGED were set when the bio
+		 * went through the path's request queue rq_qos infrastructure.
+		 * The bio is now being redirected to the multipath head's
+		 * queue which may not have rq_qos enabled, so these flags are
+		 * no longer valid and must be cleared to prevent
+		 * rq_qos_done_bio() from dereferencing a NULL q->rq_qos.
+		 */
+		bio_clear_flag(bio, BIO_QOS_THROTTLED);
+		bio_clear_flag(bio, BIO_QOS_MERGED);

This really should go into blk_steal_bios instead. As should be the existing nowait/polled fixups..

#5 lore raw

Re: [PATCH BUG FIX 2/2] nvme-multipath: clear BIO_QOS flags on requeue

Quoted 22 lines
Quoted 22 lines

Fix this by clearing both BIO_QOS_THROTTLED and BIO_QOS_MERGED flags when bios are redirected to the multipath head in nvme_failover_req(). This is consistent with the existing code that clears REQ_POLLED and REQ_NOWAIT flags when the bio changes queues.

---
  drivers/nvme/host/multipath.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/nvme/host/multipath.c +4 -0
index 3da980dc60d9..2535dba8ce1e 100644
@@ -168,6 +168,16 @@ void nvme_failover_req(struct request *req)
  		 * the flag to avoid spurious EAGAIN I/O failures.
  		 */
  		bio->bi_opf &= ~REQ_NOWAIT;
+		/*
+		 * BIO_QOS_THROTTLED and BIO_QOS_MERGED were set when the bio
+		 * went through the path's request queue rq_qos infrastructure.
+		 * The bio is now being redirected to the multipath head's
Quoted 10 lines

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 180

Quoted 6 lines
drivers/nvme/host/multipath.c +6 -0
+		 * queue which may not have rq_qos enabled, so these flags are
+		 * no longer valid and must be cleared to prevent
+		 * rq_qos_done_bio() from dereferencing a NULL q->rq_qos.
+		 */
+		bio_clear_flag(bio, BIO_QOS_THROTTLED);
+		bio_clear_flag(bio, BIO_QOS_MERGED);

This really should go into blk_steal_bios instead. As should be the existing nowait/polled fixups..

even better, should I send a prep patch to move existing nowait/polled and second patch for QOS THROTTLED and MERGED that fixes the bug ?

or just single patch is fine ?

-ck

#6 lore raw

Re: [PATCH BUG FIX 2/2] nvme-multipath: clear BIO_QOS flags on requeue

Quoted 32 lines · Chaitanya Kulkarni
Quoted 22 lines
Quoted 22 lines

Fix this by clearing both BIO_QOS_THROTTLED and BIO_QOS_MERGED flags when bios are redirected to the multipath head in nvme_failover_req(). This is consistent with the existing code that clears REQ_POLLED and REQ_NOWAIT flags when the bio changes queues.

---
  drivers/nvme/host/multipath.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/nvme/host/multipath.c +4 -0
index 3da980dc60d9..2535dba8ce1e 100644
@@ -168,6 +168,16 @@ void nvme_failover_req(struct request *req)
  		 * the flag to avoid spurious EAGAIN I/O failures.
  		 */
  		bio->bi_opf &= ~REQ_NOWAIT;
+		/*
+		 * BIO_QOS_THROTTLED and BIO_QOS_MERGED were set when the bio
+		 * went through the path's request queue rq_qos infrastructure.
+		 * The bio is now being redirected to the multipath head's
Quoted 10 lines

drivers/nvme/host/multipath.c void nvme_failover_req(struct request *req) · line 180

Quoted 6 lines
drivers/nvme/host/multipath.c +6 -0
+		 * queue which may not have rq_qos enabled, so these flags are
+		 * no longer valid and must be cleared to prevent
+		 * rq_qos_done_bio() from dereferencing a NULL q->rq_qos.
+		 */
+		bio_clear_flag(bio, BIO_QOS_THROTTLED);
+		bio_clear_flag(bio, BIO_QOS_MERGED);

This really should go into blk_steal_bios instead. As should be the existing nowait/polled fixups..

Quoted 2 lines

even better, should I send a prep patch to move existing nowait/polled and second patch for QOS THROTTLED and MERGED that fixes the bug ?

Yes, we can probably move all of them including your new clear in a separate patch (and wait until the next merge window).