nvme-multipath: fix crash in nvme_mpath_clear_ctrl_paths
PATCH 1/1 nvme-multipath: fix crash in nvme_mpath_clear_ctrl_paths
nvme_mpath_clear_ctrl_paths() iterates through the ctrl->namespaces list while holding ctrl->scan_lock. This does not seem to be the correct way of protecting from concurrent list modification.
Specifically, nvme_scan_work() sorts ctrl->namespaces AFTER unlocking scan_lock.
This may result in the following (rare) crash in ctrl disconnect during scan_work:
BUG: kernel NULL pointer dereference, address: 0000000000000050
Oops: 0000 [#1] SMP PTI
CPU: 0 PID: 3995 Comm: nvme 5.3.5-050305-generic
RIP: 0010:nvme_mpath_clear_current_path+0xe/0x90 [nvme_core]
...
Call Trace:
nvme_mpath_clear_ctrl_paths+0x3c/0x70 [nvme_core]
nvme_remove_namespaces+0x35/0xe0 [nvme_core]
nvme_do_delete_ctrl+0x47/0x90 [nvme_core]
nvme_sysfs_delete+0x49/0x60 [nvme_core]
dev_attr_store+0x17/0x30
sysfs_kf_write+0x3e/0x50
kernfs_fop_write+0x11e/0x1a0
__vfs_write+0x1b/0x40
vfs_write+0xb9/0x1a0
ksys_write+0x67/0xe0
__x64_sys_write+0x1a/0x20
do_syscall_64+0x5a/0x130
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f8d02bfb154Fix:
After taking scan_lock in nvme_mpath_clear_ctrl_paths() down_read(&ctrl->namespaces_rwsem) as well to make list traversal safe. This will not cause deadlocks because taking scan_lock never happens while holding the namespaces_rwsem.
Moreover, scan work downs namespaces_rwsem in the same order.
Alternative: sort ctrl->namespaces in nvme_scan_work()
while still holding the scan_lock.
This would leave nvme_mpath_clear_ctrl_paths() without correct protection against ctrl->namespaces modification by anyone other than scan_work.
--- drivers/nvme/host/multipath.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/host/multipath.c +2 -0
@@ -159,9 +159,11 @@ void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)struct nvme_ns *ns;mutex_lock (&ctrl->scan_lock); +down_read (&ctrl->namespaces_rwsem);list_for_each_entry (ns, &ctrl->namespaces, list)if (nvme_mpath_clear_current_path (ns))kblockd_schedule_work (&ns->head->requeue_work); +up_read (&ctrl->namespaces_rwsem);mutex_unlock (&ctrl->scan_lock); }
Signature · 7 lines
2.14.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 33 lines
nvme_mpath_clear_ctrl_paths()iterates through thectrl->namespaceslist while holdingctrl->scan_lock. This does not seem to be the correct way of protecting from concurrent list modification.Specifically,
nvme_scan_work()sortsctrl->namespacesAFTER unlockingscan_lock.This may result in the following (rare) crash in ctrl disconnect during
scan_work:BUG: kernel NULL pointer dereference, address: 0000000000000050 Oops: 0000 [#1] SMP PTI CPU: 0 PID: 3995 Comm: nvme 5.3.5-050305-generic RIP: 0010:nvme_mpath_clear_current_path+0xe/0x90 [nvme_core] ... Call Trace: nvme_mpath_clear_ctrl_paths+0x3c/0x70 [nvme_core] nvme_remove_namespaces+0x35/0xe0 [nvme_core] nvme_do_delete_ctrl+0x47/0x90 [nvme_core] nvme_sysfs_delete+0x49/0x60 [nvme_core] dev_attr_store+0x17/0x30 sysfs_kf_write+0x3e/0x50 kernfs_fop_write+0x11e/0x1a0 __vfs_write+0x1b/0x40 vfs_write+0xb9/0x1a0 ksys_write+0x67/0xe0 __x64_sys_write+0x1a/0x20 do_syscall_64+0x5a/0x130 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f8d02bfb154
Quoted 6 lines
Fix:
After takingscan_lockinnvme_mpath_clear_ctrl_paths()down_read(&ctrl->namespaces_rwsem)as well to make list traversal safe. This will not cause deadlocks because takingscan_locknever happens while holding thenamespaces_rwsem.
Moreover, scan work downsnamespaces_rwsemin the same order.
Thanks Anton, this looks good to me,
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Looks good,
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme