Skip to contentSkip to conversation

nvme-pci: fix probe and remove race

PATCH New Sagi Grimberg 4 participants 9 messages Jul 19, 2019 – Jul 29, 2019 1 message not in the archive NVMe

lore
Share & tools
lore

Sagi Grimberg

author maintainer
#0 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

It is possible that nvme_remove() being ran concurrently with nvme_reset_work(), with following sequence:

nvme_probe()
nvme_init_ctrl()

    //set to NEW
  nvme_async_probe()
                                                      nvme_remove()
                                                        //can not change to
                                                        //DELETING from NEW
    nvme_reset_ctrl_sync()
        nvme_reset_ctrl()
          //change from NEW
          //to RESETTING
                                                       flush reset_work()
                                                       //not yet queued
          queue reset_work
            nvme_reset_work()
              ....                                     ....

With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
[mem 0xc0000000-0xc0003fff] not claimed

Fix this by waiting for the async probe to complete before allowing remove to make forward progress.

Reported-by Li Zhong <lizhongfs at gmail.com>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/pci.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +4 -1
index 4b508d5e45cf..50061abe49c6 100644
@@ -127,6 +127,7 @@ struct nvme_dev {
 	dma_addr_t host_mem_descs_dma;
 	struct nvme_host_mem_buf_desc *host_mem_descs;
 	void **host_mem_desc_bufs;
+	async_cookie_t async_probe;
 };
 
 static int io_queue_depth_set(const char *val, const struct kernel_param *kp)
@@ -2765,7 +2766,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
 
 	nvme_get_ctrl(&dev->ctrl);
-	async_schedule(nvme_async_probe, dev);
+	dev->async_probe = async_schedule(nvme_async_probe, dev);
 
 	return 0;
 
@@ -2810,6 +2811,8 @@ static void nvme_remove(struct pci_dev *pdev)
 {
 	struct nvme_dev *dev = pci_get_drvdata(pdev);
 
+	/* wait for async probe to complete */
+	async_synchronize_cookie(dev->async_probe + 1);
 	nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
 	pci_set_drvdata(pdev, NULL);
 
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

Quoted 26 lines · Sagi Grimberg

It is possible that nvme_remove() being ran concurrently with nvme_reset_work(), with following sequence:

nvme_probe()
nvme_init_ctrl()

    //set to NEW
  nvme_async_probe()
                                                      nvme_remove()
                                                        //can not change to
                                                        //DELETING from NEW
    nvme_reset_ctrl_sync()
        nvme_reset_ctrl()
          //change from NEW
          //to RESETTING
                                                       flush reset_work()
                                                       //not yet queued
          queue reset_work
            nvme_reset_work()
              ....                                     ....

With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
[mem 0xc0000000-0xc0003fff] not claimed

Quoted 5 lines

Fix this by waiting for the async probe to complete before allowing remove to make forward progress.

Reported-by Li Zhong <lizhongfs at gmail.com>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>

This looks good to me.

Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>

Thanks, Sagi.

A message that is not in the archive (sent off-list or not yet received): CAOSXXT7z4+pScQ+Kf0VauTCvPdRDEXX=H7jQN-Dkk=M2hkTFsA@mail.gmail.com

author maintainer
#3 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

Quoted 30 lines · Keith Busch
Quoted 9 lines · On Fri, Jul 19, 2019 at 1:43 PM Sagi Grimberg <sagi at grimberg.me…

It is possible that nvme_remove() being ran concurrently with nvme_reset_work(), with following sequence:

nvme_probe()
? nvme_init_ctrl()
? ? //set to NEW
? nvme_async_probe()
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? nvme_remove() ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? //can not

change to

Quoted 1 line

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? //DELETING

from NEW

Quoted 16 lines

? ? nvme_reset_ctrl_sync()
? ? ? ? nvme_reset_ctrl()
? ? ? ? ? //change from NEW
? ? ? ? ? //to RESETTING
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?flush reset_work() ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?//not yet queued ? ? ? ? ? queue reset_work
? ? ? ? ? ? nvme_reset_work()
? ? ? ? ? ? ? .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ....

With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
?[mem 0xc0000000-0xc0003fff] not claimed

Quoted 8 lines
Quoted 2 lines

Fix this by waiting for the async probe to complete before allowing remove to make forward progress.

Hi Sagi,

The only problem is the async probe may be stuck if its device was hot removed, and needs nvme_remove to make it unstuck immediately. Otherwise it waits for timeout work to fix it up, which can be a very long time.

Given that this is an error case, I don't think that the timeout is a big issue.

Quoted 2 lines

We only really need to wait for the state to not be new? And if so, is the attached ok?

I don't have a firm reproducer, I was hoping that Li would have since he reported the problem.

This would work because nvme_dev_disable is serialized with the shutdown_lock...

#4 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

In reply to Sagi Grimberg
Quoted 22 lines · Sagi Grimberg
drivers/nvme/host/pci.c +2 -1
index 4b508d5e45cf..50061abe49c6 100644
@@ -127,6 +127,7 @@ struct nvme_dev {
  	dma_addr_t host_mem_descs_dma;
  	struct nvme_host_mem_buf_desc *host_mem_descs;
  	void **host_mem_desc_bufs;
+	async_cookie_t async_probe;
  };
 
  static int io_queue_depth_set(const char *val, const struct kernel_param *kp)
@@ -2765,7 +2766,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
  	dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
 
  	nvme_get_ctrl(&dev->ctrl);
-	async_schedule(nvme_async_probe, dev);
+	dev->async_probe = async_schedule(nvme_async_probe, dev);
 
  	return 0;
 

drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2817

Quoted 7 lines
drivers/nvme/host/pci.c +2 -0
@@ -2810,6 +2811,8 @@ static void nvme_remove(struct pci_dev *pdev)
  {
  	struct nvme_dev *dev = pci_get_drvdata(pdev);
 
+	/* wait for async probe to complete */
+	async_synchronize_cookie(dev->async_probe + 1);
  	nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
  	pci_set_drvdata(pdev, NULL);

Hi Sagi,

Does the async_synchronize_cookie() call wait until all previously started probes have finished? In other words, does the change in nvme_remove() introduce a dependency between probe and remove calls of different NVMe devices? Is that dependency important? If not, can it be avoided to introduce that dependency?

Thanks,

Bart.

author maintainer
#5 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

Quoted 31 lines
Quoted 22 lines
drivers/nvme/host/pci.c +1 -0
index 4b508d5e45cf..50061abe49c6 100644
@@ -127,6 +127,7 @@ struct nvme_dev {
 ????? dma_addr_t host_mem_descs_dma;
 ????? struct nvme_host_mem_buf_desc *host_mem_descs;
 ????? void **host_mem_desc_bufs;
+??? async_cookie_t async_probe;
 ? };

? static int io_queue_depth_set(const char *val, const struct kernel_param *kp)

drivers/nvme/host/pci.c +1 -1
@@ -2765,7 +2766,7 @@ static int nvme_probe(struct pci_dev *pdev, 
 const struct pci_device_id *id)
 ????? dev_info(dev->ctrl.device, "pci function %s\n", 
 dev_name(&pdev->dev));
 ????? nvme_get_ctrl(&dev->ctrl);
-??? async_schedule(nvme_async_probe, dev);
+??? dev->async_probe = async_schedule(nvme_async_probe, dev);
 ????? return 0;

drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2816

Quoted 6 lines
drivers/nvme/host/pci.c +2 -0
@@ -2810,6 +2811,8 @@ static void nvme_remove(struct pci_dev *pdev)
 ? {
 ????? struct nvme_dev *dev = pci_get_drvdata(pdev);
+??? /* wait for async probe to complete */
+??? async_synchronize_cookie(dev->async_probe + 1);
 ????? nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING);
 ????? pci_set_drvdata(pdev, NULL);

Hi Sagi,

Quoted 5 lines

Does the async_synchronize_cookie() call wait until all previously started probes have finished? In other words, does the change in nvme_remove() introduce a dependency between probe and remove calls of different NVMe devices? Is that dependency important? If not, can it be avoided to introduce that dependency?

It does create dependency as they are in the same domain I assume. Given that nvme_remove() is really a hot-unplug operation (or shutdown) I am not sure if this dependency is a limitation in any real use-case though..

maintainer
#6 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

In reply to Sagi Grimberg
Quoted 26 lines · Sagi Grimberg

It is possible that nvme_remove() being ran concurrently with nvme_reset_work(), with following sequence:

nvme_probe()
nvme_init_ctrl()

    //set to NEW
  nvme_async_probe()
                                                      nvme_remove()
                                                        //can not change to
                                                        //DELETING from NEW
    nvme_reset_ctrl_sync()
        nvme_reset_ctrl()
          //change from NEW
          //to RESETTING
                                                       flush reset_work()
                                                       //not yet queued
          queue reset_work
            nvme_reset_work()
              ....                                     ....

With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
[mem 0xc0000000-0xc0003fff] not claimed

Quoted 5 lines

Fix this by waiting for the async probe to complete before allowing remove to make forward progress.

Reported-by Li Zhong <lizhongfs at gmail.com>
Signed-off-by Sagi Grimberg <sagi at grimberg.me>

I still think we'd prefer not adding that async domain dependency and relying on timeout to unstuck a hot-removal. So how about we schedule the reset work in probe and have the async part just flush the reset and scan work?

---

drivers/nvme/host/pci.c +2 -1
index db160cee42ad..0c2c4b0c6655 100644
@@ -2695,7 +2695,7 @@ static void nvme_async_probe(void *data, async_cookie_t cookie)
 {
 	struct nvme_dev *dev = data;
 
-	nvme_reset_ctrl_sync(&dev->ctrl);
+	flush_work(&dev->ctrl.reset_work);
 	flush_work(&dev->ctrl.scan_work);
 	nvme_put_ctrl(&dev->ctrl);
 }
@@ -2761,6 +2761,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
 	dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
 
+	nvme_reset_ctrl(&dev->ctrl);
 	nvme_get_ctrl(&dev->ctrl);
 	async_schedule(nvme_async_probe, dev);
 

--

author maintainer
#7 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

Quoted 21 lines

I still think we'd prefer not adding that async domain dependency and relying on timeout to unstuck a hot-removal. So how about we schedule the reset work in probe and have the async part just flush the reset and scan work?

---

drivers/nvme/host/pci.c +1 -1
index db160cee42ad..0c2c4b0c6655 100644
@@ -2695,7 +2695,7 @@ static void nvme_async_probe(void *data, async_cookie_t cookie)
  {
  	struct nvme_dev *dev = data;
 
-	nvme_reset_ctrl_sync(&dev->ctrl);
+	flush_work(&dev->ctrl.reset_work);
  	flush_work(&dev->ctrl.scan_work);
  	nvme_put_ctrl(&dev->ctrl);
  }

drivers/nvme/host/pci.c static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 2765

Quoted 5 lines
drivers/nvme/host/pci.c +1 -0
@@ -2761,6 +2761,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 
  	dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
 
+	nvme_reset_ctrl(&dev->ctrl);
  	nvme_get_ctrl(&dev->ctrl);

I think you need to get the ref first and then fire the work right?

maintainer
#8 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

Quoted 17 lines · Sagi Grimberg
Quoted 17 lines

I still think we'd prefer not adding that async domain dependency and relying on timeout to unstuck a hot-removal. So how about we schedule the reset work in probe and have the async part just flush the reset and scan work?

---

drivers/nvme/host/pci.c +1 -1
index db160cee42ad..0c2c4b0c6655 100644
@@ -2695,7 +2695,7 @@ static void nvme_async_probe(void *data, async_cookie_t cookie)
  {
  	struct nvme_dev *dev = data;
-	nvme_reset_ctrl_sync(&dev->ctrl);
+	flush_work(&dev->ctrl.reset_work);
  	flush_work(&dev->ctrl.scan_work);
Quoted 8 lines

drivers/nvme/host/pci.c static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 2763

Quoted 6 lines
drivers/nvme/host/pci.c +1 -0
  	nvme_put_ctrl(&dev->ctrl);
  }
@@ -2761,6 +2761,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)
  	dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev));
+	nvme_reset_ctrl(&dev->ctrl);
  	nvme_get_ctrl(&dev->ctrl);

I think you need to get the ref first and then fire the work right?

That ref order doesn't actually matter here. We can't call nvme_remove during the synchronous part of probe, and the extra ref is just for the async_schedule callback.

author maintainer
#9 lore raw

PATCH 1/1 nvme-pci: fix probe and remove race

Quoted 6 lines
Quoted 2 lines

I think you need to get the ref first and then fire the work right?

That ref order doesn't actually matter here. We can't call nvme_remove during the synchronous part of probe, and the extra ref is just for the async_schedule callback.

Makes sense,

Can you send a proper patch?