Skip to contentSkip to conversation

nvme-multipath: fix crash in nvme_mpath_clear_ctrl_paths

PATCH In review Anton Eidelman 3 participants 3 messages Nov 2, 2019 – Nov 4, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-multipath: fix crash in nvme_mpath_clear_ctrl_paths

nvme_mpath_clear_ctrl_paths() iterates through the ctrl->namespaces list while holding ctrl->scan_lock. This does not seem to be the correct way of protecting from concurrent list modification.

Specifically, nvme_scan_work() sorts ctrl->namespaces AFTER unlocking scan_lock.

This may result in the following (rare) crash in ctrl disconnect during scan_work:

    BUG: kernel NULL pointer dereference, address: 0000000000000050
    Oops: 0000 [#1] SMP PTI
    CPU: 0 PID: 3995 Comm: nvme 5.3.5-050305-generic
    RIP: 0010:nvme_mpath_clear_current_path+0xe/0x90 [nvme_core]
    ...
    Call Trace:
     nvme_mpath_clear_ctrl_paths+0x3c/0x70 [nvme_core]
     nvme_remove_namespaces+0x35/0xe0 [nvme_core]
     nvme_do_delete_ctrl+0x47/0x90 [nvme_core]
     nvme_sysfs_delete+0x49/0x60 [nvme_core]
     dev_attr_store+0x17/0x30
     sysfs_kf_write+0x3e/0x50
     kernfs_fop_write+0x11e/0x1a0
     __vfs_write+0x1b/0x40
     vfs_write+0xb9/0x1a0
     ksys_write+0x67/0xe0
     __x64_sys_write+0x1a/0x20
     do_syscall_64+0x5a/0x130
     entry_SYSCALL_64_after_hwframe+0x44/0xa9
    RIP: 0033:0x7f8d02bfb154

Fix:
After taking scan_lock in nvme_mpath_clear_ctrl_paths() down_read(&ctrl->namespaces_rwsem) as well to make list traversal safe. This will not cause deadlocks because taking scan_lock never happens while holding the namespaces_rwsem.
Moreover, scan work downs namespaces_rwsem in the same order.

Alternative: sort ctrl->namespaces in nvme_scan_work()
while still holding the scan_lock.
This would leave nvme_mpath_clear_ctrl_paths() without correct protection against ctrl->namespaces modification by anyone other than scan_work.

---
 drivers/nvme/host/multipath.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/multipath.c +2 -0
index 682be6195a95..cb00afac516d 100644
@@ -159,9 +159,11 @@ void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl)
 	struct nvme_ns *ns;
 
 	mutex_lock(&ctrl->scan_lock);
+	down_read(&ctrl->namespaces_rwsem);
 	list_for_each_entry(ns, &ctrl->namespaces, list)
 		if (nvme_mpath_clear_current_path(ns))
 			kblockd_schedule_work(&ns->head->requeue_work);
+	up_read(&ctrl->namespaces_rwsem);
 	mutex_unlock(&ctrl->scan_lock);
 }
 
Signature · 7 lines
2.14.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 33 lines

nvme_mpath_clear_ctrl_paths() iterates through the ctrl->namespaces list while holding ctrl->scan_lock. This does not seem to be the correct way of protecting from concurrent list modification.

Specifically, nvme_scan_work() sorts ctrl->namespaces AFTER unlocking scan_lock.

This may result in the following (rare) crash in ctrl disconnect during scan_work:

     BUG: kernel NULL pointer dereference, address: 0000000000000050
     Oops: 0000 [#1] SMP PTI
     CPU: 0 PID: 3995 Comm: nvme 5.3.5-050305-generic
     RIP: 0010:nvme_mpath_clear_current_path+0xe/0x90 [nvme_core]
     ...
     Call Trace:
      nvme_mpath_clear_ctrl_paths+0x3c/0x70 [nvme_core]
      nvme_remove_namespaces+0x35/0xe0 [nvme_core]
      nvme_do_delete_ctrl+0x47/0x90 [nvme_core]
      nvme_sysfs_delete+0x49/0x60 [nvme_core]
      dev_attr_store+0x17/0x30
      sysfs_kf_write+0x3e/0x50
      kernfs_fop_write+0x11e/0x1a0
      __vfs_write+0x1b/0x40
      vfs_write+0xb9/0x1a0
      ksys_write+0x67/0xe0
      __x64_sys_write+0x1a/0x20
      do_syscall_64+0x5a/0x130
      entry_SYSCALL_64_after_hwframe+0x44/0xa9
     RIP: 0033:0x7f8d02bfb154
Quoted 6 lines

Fix:
After taking scan_lock in nvme_mpath_clear_ctrl_paths() down_read(&ctrl->namespaces_rwsem) as well to make list traversal safe. This will not cause deadlocks because taking scan_lock never happens while holding the namespaces_rwsem.
Moreover, scan work downs namespaces_rwsem in the same order.

Thanks Anton, this looks good to me,

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme