nvme-pci: avoid dereference of symbol from unloaded module

PATCH New Ming Lei 3 participants 7 messages Nov 2, 2017 – Nov 9, 2017 NVMe

Share & tools

Ming Lei

author

2017-11-02 09:36 UTC #0 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

The 'remove_work' may be scheduled to run after nvme_remove() returns since we can't simply cancel it in nvme_remove() for avoiding deadlock. Once nvme_remove() returns, this module(nvme) can be unloaded.

On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl which may point to nvme_pci_free_ctrl() in unloaded module.

This patch avoids this issue by holding the module refcount before scheduling 'remove_work'.

Signed-off-by Ming Lei <ming.lei at redhat.com>

---
 drivers/nvme/host/pci.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

drivers/nvme/host/pci.c +15 -1

index 3f5a04c586ce..3ed8fd7218d4 100644
@@ -2134,8 +2134,12 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev, int status)
 
 	kref_get(&dev->ctrl.kref);
 	nvme_dev_disable(dev, false);
-	if (!schedule_work(&dev->remove_work))
+
+	__module_get(THIS_MODULE);
+	if (!schedule_work(&dev->remove_work)) {
 		nvme_put_ctrl(&dev->ctrl);
+		module_put(THIS_MODULE);
+	}
 }
 
 static void nvme_reset_work(struct work_struct *work)

@@ -2231,10 +2235,20 @@ static void nvme_remove_dead_ctrl_work(struct work_struct *work)
 	struct nvme_dev *dev = container_of(work, struct nvme_dev, remove_work);
 	struct pci_dev *pdev = to_pci_dev(dev->dev);
 
+	/*
+	 * Inside nvme_remove(), we can't simply cancel this 'remove_work'
+	 * for avoiding deadlock, so this work function may be run after
+	 * nvme_remove() returns, and this module may have been removed
+	 * at that time. We have to get the module refcount before scheduling
+	 * 'remove_work', otherwise nvme_put_ctrl() may deference symbols
+	 * of the unloaded module via ctrl->ops.
+	 */
+
 	nvme_kill_queues(&dev->ctrl);
 	if (pci_get_drvdata(pdev))
 		device_release_driver(&pdev->dev);
 	nvme_put_ctrl(&dev->ctrl);
+	module_put(THIS_MODULE);
 }
 
 static int nvme_pci_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val)

Signature · 1 line

2.9.5

Sagi Grimberg

maintainer

2017-11-02 12:10 UTC #1 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

Quoted 11 lines

The 'remove_work' may be scheduled to run after nvme_remove() returns since we can't simply cancel it in nvme_remove() for avoiding deadlock. Once nvme_remove() returns, this module(nvme) can be unloaded.
On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl which may point to nvme_pci_free_ctrl() in unloaded module.
This patch avoids this issue by holding the module refcount before scheduling 'remove_work'.

Why not simply flushing all scheduled works in module exit?

Ming Lei

author

2017-11-02 12:29 UTC #2 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

Quoted 6 lines · Sagi Grimberg

Quoted 6 lines
The 'remove_work' may be scheduled to run after nvme_remove() returns since we can't simply cancel it in nvme_remove() for avoiding deadlock. Once nvme_remove() returns, this module(nvme) can be unloaded.

Quoted 7 lines

Quoted 5 lines
On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl which may point to nvme_pci_free_ctrl() in unloaded module.
This patch avoids this issue by holding the module refcount before scheduling 'remove_work'.
Why not simply flushing all scheduled works in module exit?

The nvme_wq is shared by all kinds of host(PCI, FC, ...), so not good to flush all queued work in 'nvme_wq' just before removing pci host module only.

Not mention 'remove_work' is actually scheduled by system wq instead of nvme_wq.

Signature · 1 line

Ming

Christoph Hellwig

2017-11-08 18:15 UTC #3 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

Quoted 13 lines · Ming Lei

Quoted 12 lines · Sagi Grimberg
Quoted 11 lines
The 'remove_work' may be scheduled to run after nvme_remove() returns since we can't simply cancel it in nvme_remove() for avoiding deadlock. Once nvme_remove() returns, this module(nvme) can be unloaded.
On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl which may point to nvme_pci_free_ctrl() in unloaded module.
This patch avoids this issue by holding the module refcount before scheduling 'remove_work'.

Quoted 8 lines

Quoted 1 line
Why not simply flushing all scheduled works in module exit?
The nvme_wq is shared by all kinds of host(PCI, FC, ...), so not good to flush all queued work in 'nvme_wq' just before removing pci host module only.
Not mention 'remove_work' is actually scheduled by system wq instead of nvme_wq.

Still seems better than messing with module reference counts.

Ming Lei

author

2017-11-09 01:13 UTC #4 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

Quoted 16 lines · Christoph Hellwig

Quoted 15 lines · Ming Lei
Quoted 13 lines · Sagi Grimberg
Quoted 11 lines
The 'remove_work' may be scheduled to run after nvme_remove() returns since we can't simply cancel it in nvme_remove() for avoiding deadlock. Once nvme_remove() returns, this module(nvme) can be unloaded.
On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl which may point to nvme_pci_free_ctrl() in unloaded module.
This patch avoids this issue by holding the module refcount before scheduling 'remove_work'.
Why not simply flushing all scheduled works in module exit?

Quoted 8 lines

Quoted 6 lines
The nvme_wq is shared by all kinds of host(PCI, FC, ...), so not good to flush all queued work in 'nvme_wq' just before removing pci host module only.
Not mention 'remove_work' is actually scheduled by system wq instead of nvme_wq.
Still seems better than messing with module reference counts.

If 'remove_work' can be queued via 'nvme_wq', I think it is doable to call flush_workqueue(nvme_wq) in nvme_exit(): drivers/nvme/host/pci.c.

Otherwise, it may not be a good idea to flush the global wq, please see comment of 'flush_scheduled_work()'.

Signature · 1 line

Ming

Sagi Grimberg

maintainer

2017-11-09 10:32 UTC #5 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

Quoted 3 lines

If 'remove_work' can be queued via 'nvme_wq', I think it is doable to call flush_workqueue(nvme_wq) in nvme_exit(): drivers/nvme/host/pci.c.

Lets do that.

Sagi Grimberg

maintainer

2017-11-09 10:31 UTC #6 lore raw

PATCH 1/1 nvme-pci: avoid dereference of symbol from unloaded module

In reply to Christoph Hellwig

Quoted 9 lines

Quoted 7 lines
The nvme_wq is shared by all kinds of host(PCI, FC, ...), so not good to flush all queued work in 'nvme_wq' just before removing pci host module only.
Not mention 'remove_work' is actually scheduled by system wq instead of nvme_wq.
Still seems better than messing with module reference counts.

And also does not change the behavior (failing module unload).