Skip to contentSkip to conversation

NVMe: Fix reset/remove race

PATCH Superseded Keith Busch 4 participants 5 messages Mar 22, 2016 – Apr 4, 2016 NVMe

lore
Share & tools
lore

Keith Busch

author
#0 lore raw

PATCH v2 1/1 NVMe: Fix reset/remove race

This fixes a scenario where device is present and being reset, but a request to unbind the driver occurs.

A previous patch series addressing a device failure removal scenario flushed reset_work after controller disable to unblock reset_work waiting on a completion that wouldn't occur. This isn't safe as-is. The broken scenario can potentially be induced with:

  modprobe nvme && modprobe -r nvme

To fix, the reset work is flushed immediately after setting the controller removing flag, and any subsequent reset will not proceed with controller initialization if the flag is set.

The controller status must be polled while active, so the watchdog timer is also left active until the controller is disabled to cleanup requests that may be stuck during namespace removal.

[Fixes: ff23a2a15a2117245b4599c1352343c8b8fb4c43]
Signed-off-by Keith Busch <keith.busch at intel.com>

---
v1->v2:
Removed the untested change on IO timeout handling that skipped queueing reset work.

 drivers/nvme/host/pci.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +4 -2
index 24ccda3..660ec84 100644
@@ -1859,6 +1859,9 @@ static void nvme_reset_work(struct work_struct *work)
 	if (dev->ctrl.ctrl_config & NVME_CC_ENABLE)
 		nvme_dev_disable(dev, false);
 
+	if (test_bit(NVME_CTRL_REMOVING, &dev->flags))
+		goto out;
+
 	set_bit(NVME_CTRL_RESETTING, &dev->flags);
 
 	result = nvme_pci_enable(dev);

@@ -2078,11 +2081,10 @@ static void nvme_remove(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
 
-	del_timer_sync(&dev->watchdog_timer);
-
 	set_bit(NVME_CTRL_REMOVING, &dev->flags);
 	pci_set_drvdata(pdev, NULL);
 	flush_work(&dev->async_work);
+	flush_work(&dev->reset_work);
 	flush_work(&dev->scan_work);
 	nvme_remove_namespaces(&dev->ctrl);
 	nvme_uninit_ctrl(&dev->ctrl);
Signature · 1 line
2.7.2
#1 lore raw

PATCH v2 1/1 NVMe: Fix reset/remove race

This looks ok:

Reviewed-by Christoph Hellwig <hch at lst.de>

But this just shows we'll finally need a real state machine for the NVMe controller instead of the flags. Sagi did a really nice one for the Fabrics driver, and I'll plan to lift it to common code for 4.7.

#2 lore raw

PATCH v2 1/1 NVMe: Fix reset/remove race

In reply to Keith Busch
Quoted 45 lines · On Dienstag, 22. M?rz 2016 15:49:35 CET Keith Busch wrote:

This fixes a scenario where device is present and being reset, but a request to unbind the driver occurs.

A previous patch series addressing a device failure removal scenario flushed reset_work after controller disable to unblock reset_work waiting on a completion that wouldn't occur. This isn't safe as-is. The broken scenario can potentially be induced with:

  modprobe nvme && modprobe -r nvme

To fix, the reset work is flushed immediately after setting the controller removing flag, and any subsequent reset will not proceed with controller initialization if the flag is set.

The controller status must be polled while active, so the watchdog timer is also left active until the controller is disabled to cleanup requests that may be stuck during namespace removal.

[Fixes: ff23a2a15a2117245b4599c1352343c8b8fb4c43]
Signed-off-by Keith Busch <keith.busch at intel.com>

---
v1->v2:
Removed the untested change on IO timeout handling that skipped queueing reset work.

 drivers/nvme/host/pci.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +3 -0
index 24ccda3..660ec84 100644
@@ -1859,6 +1859,9 @@ static void nvme_reset_work(struct work_struct *work)
 	if (dev->ctrl.ctrl_config & NVME_CC_ENABLE)
 		nvme_dev_disable(dev, false);
 
+	if (test_bit(NVME_CTRL_REMOVING, &dev->flags))
+		goto out;
+
 	set_bit(NVME_CTRL_RESETTING, &dev->flags);
 
 	result = nvme_pci_enable(dev);

@@ -2078,11 +2081,10 @@ static void nvme_remove(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);

drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2090

Quoted 10 lines
drivers/nvme/host/pci.c +1 -2
 
-	del_timer_sync(&dev->watchdog_timer);
-
 	set_bit(NVME_CTRL_REMOVING, &dev->flags);
 	pci_set_drvdata(pdev, NULL);
 	flush_work(&dev->async_work);
+	flush_work(&dev->reset_work);
 	flush_work(&dev->scan_work);
 	nvme_remove_namespaces(&dev->ctrl);
 	nvme_uninit_ctrl(&dev->ctrl);
Reviewed-by Johannes Thumshirn <jthumshirn at suse.de>
Signature · 6 lines
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
maintainer
#3 lore raw

PATCH v2 1/1 NVMe: Fix reset/remove race

In reply to Keith Busch
Quoted 43 lines · Keith Busch

This fixes a scenario where device is present and being reset, but a request to unbind the driver occurs.

A previous patch series addressing a device failure removal scenario flushed reset_work after controller disable to unblock reset_work waiting on a completion that wouldn't occur. This isn't safe as-is. The broken scenario can potentially be induced with:

   modprobe nvme && modprobe -r nvme

To fix, the reset work is flushed immediately after setting the controller removing flag, and any subsequent reset will not proceed with controller initialization if the flag is set.

The controller status must be polled while active, so the watchdog timer is also left active until the controller is disabled to cleanup requests that may be stuck during namespace removal.

[Fixes: ff23a2a15a2117245b4599c1352343c8b8fb4c43]
Signed-off-by Keith Busch <keith.busch at intel.com>

---
v1->v2:
Removed the untested change on IO timeout handling that skipped queueing reset work.

  drivers/nvme/host/pci.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +3 -0
index 24ccda3..660ec84 100644
@@ -1859,6 +1859,9 @@ static void nvme_reset_work(struct work_struct *work)
  	if (dev->ctrl.ctrl_config & NVME_CC_ENABLE)
  		nvme_dev_disable(dev, false);
 
+	if (test_bit(NVME_CTRL_REMOVING, &dev->flags))
+		goto out;
+
  	set_bit(NVME_CTRL_RESETTING, &dev->flags);
 
  	result = nvme_pci_enable(dev);

drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2087

Quoted 9 lines
drivers/nvme/host/pci.c +1 -2
@@ -2078,11 +2081,10 @@ static void nvme_remove(struct pci_dev *pdev)
  {
  	struct nvme_dev *dev = pci_get_drvdata(pdev);
 
-	del_timer_sync(&dev->watchdog_timer);
-
  	set_bit(NVME_CTRL_REMOVING, &dev->flags);
  	pci_set_drvdata(pdev, NULL);
  	flush_work(&dev->async_work);
+	flush_work(&dev->reset_work);

Do we need the same for scan_work? AFAICT it can still sneak in while we're removing...

author
#4 lore raw

PATCH v2 1/1 NVMe: Fix reset/remove race

Quoted 20 lines · sagig
Quoted 19 lines · Keith Busch
 drivers/nvme/host/pci.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/pci.c +3 -1
index 24ccda3..660ec84 100644
@@ -1859,6 +1859,9 @@ static void nvme_reset_work(struct work_struct *work)
 	if (dev->ctrl.ctrl_config & NVME_CC_ENABLE)
 		nvme_dev_disable(dev, false);
+	if (test_bit(NVME_CTRL_REMOVING, &dev->flags))
+		goto out;
+
 	set_bit(NVME_CTRL_RESETTING, &dev->flags);
 	result = nvme_pci_enable(dev);

@@ -2078,11 +2081,10 @@ static void nvme_remove(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
-	del_timer_sync(&dev->watchdog_timer);
Quoted 8 lines

drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2087

Quoted 5 lines
drivers/nvme/host/pci.c +1 -1
-
 	set_bit(NVME_CTRL_REMOVING, &dev->flags);
 	pci_set_drvdata(pdev, NULL);
 	flush_work(&dev->async_work);
+	flush_work(&dev->reset_work);

Do we need the same for scan_work? AFAICT it can still sneak in while we're removing...

It is flushed in the very next line not included in your reply. :)

scan_work isn't queued again when the "NVME_CTRL_REMOVING" flag is set, so we're safe from seeing that queue again.