nvme-pci: fix probe and remove race
Sagi Grimberg
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
It is possible that nvme_remove() being ran concurrently with nvme_reset_work(), with following sequence:
nvme_probe()
nvme_init_ctrl()
//set to NEW
nvme_async_probe()
nvme_remove()
//can not change to
//DELETING from NEW
nvme_reset_ctrl_sync()
nvme_reset_ctrl()
//change from NEW
//to RESETTING
flush reset_work()
//not yet queued
queue reset_work
nvme_reset_work()
.... ....With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
[mem 0xc0000000-0xc0003fff] not claimed
Fix this by waiting for the async probe to complete before allowing remove to make forward progress.
--- drivers/nvme/host/pci.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +4 -1
@@ -127,6 +127,7 @@ struct nvme_dev {dma_addr_t host_mem_descs_dma;struct nvme_host_mem_buf_desc *host_mem_descs;void **host_mem_desc_bufs; +async_cookie_t async_probe; };static int io_queue_depth_set (const char *val,const struct kernel_param *kp)
@@ -2765,7 +2766,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)dev_info (dev->ctrl.device,"pci function %s\n" ,dev_name (&pdev->dev));nvme_get_ctrl (&dev->ctrl); -async_schedule (nvme_async_probe, dev); + dev->async_probe =async_schedule (nvme_async_probe, dev);return 0 ;
@@ -2810,6 +2811,8 @@ static void nvme_remove(struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); +/* wait for async probe to complete */ +async_synchronize_cookie (dev->async_probe +1 );nvme_change_ctrl_state (&dev->ctrl,NVME_CTRL_DELETING );pci_set_drvdata (pdev,NULL );
Signature · 1 line
2.17.1
PATCH 1/1 nvme-pci: fix probe and remove race
Quoted 26 lines · Sagi Grimberg
It is possible that
nvme_remove()being ran concurrently withnvme_reset_work(), with following sequence:
nvme_probe()
nvme_init_ctrl()//set to NEW nvme_async_probe() nvme_remove() //can not change to //DELETING from NEW nvme_reset_ctrl_sync() nvme_reset_ctrl() //change from NEW //to RESETTING flush reset_work() //not yet queued queue reset_work nvme_reset_work() .... ....With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
[mem 0xc0000000-0xc0003fff] not claimed
Quoted 5 lines
Fix this by waiting for the async probe to complete before allowing remove to make forward progress.
Reported-by Li Zhong <lizhongfs at gmail.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>
This looks good to me.
Thanks, Sagi.
A message that is not in the archive (sent off-list or not yet received): CAOSXXT7z4+pScQ+Kf0VauTCvPdRDEXX=H7jQN-Dkk=M2hkTFsA@mail.gmail.com
Sagi Grimberg
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
Quoted 30 lines · Keith Busch
Quoted 9 lines · On Fri, Jul 19, 2019 at 1:43 PM Sagi Grimberg <sagi at grimberg.me…
It is possible that
nvme_remove()being ran concurrently withnvme_reset_work(), with following sequence:
nvme_probe()
?nvme_init_ctrl()
? ? //set to NEW
?nvme_async_probe()
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?nvme_remove()? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? //can notchange to
Quoted 1 line
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? //DELETING
from NEW
Quoted 16 lines
? ?
nvme_reset_ctrl_sync()
? ? ? ?nvme_reset_ctrl()
? ? ? ? ? //change from NEW
? ? ? ? ? //to RESETTING
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?flushreset_work()? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?//not yet queued ? ? ? ? ? queuereset_work
? ? ? ? ? ?nvme_reset_work()
? ? ? ? ? ? ? .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ....With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
?[mem 0xc0000000-0xc0003fff] not claimed
Quoted 8 lines
Quoted 2 lines
Fix this by waiting for the async probe to complete before allowing remove to make forward progress.
Hi Sagi,
The only problem is the async probe may be stuck if its device was hot removed, and needs
nvme_removeto make it unstuck immediately. Otherwise it waits for timeout work to fix it up, which can be a very long time.
Given that this is an error case, I don't think that the timeout is a big issue.
Quoted 2 lines
We only really need to wait for the state to not be new? And if so, is the attached ok?
I don't have a firm reproducer, I was hoping that Li would have since he reported the problem.
This would work because nvme_dev_disable is serialized with the shutdown_lock...
PATCH 1/1 nvme-pci: fix probe and remove race
In reply to Sagi GrimbergQuoted 22 lines · Sagi Grimberg
drivers/nvme/host/pci.c +2
-1@@ -127,6 +127,7 @@ struct nvme_dev {dma_addr_t host_mem_descs_dma;struct nvme_host_mem_buf_desc *host_mem_descs;void **host_mem_desc_bufs; +async_cookie_t async_probe; };static int io_queue_depth_set (const char *val,const struct kernel_param *kp)@@ -2765,7 +2766,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)dev_info (dev->ctrl.device,"pci function %s\n" ,dev_name (&pdev->dev));nvme_get_ctrl (&dev->ctrl); -async_schedule (nvme_async_probe, dev); + dev->async_probe =async_schedule (nvme_async_probe, dev);return 0 ;
drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2817
Quoted 7 lines
drivers/nvme/host/pci.c +2
-0@@ -2810,6 +2811,8 @@ static void nvme_remove(struct pci_dev *pdev) {struct nvme_dev *dev =pci_get_drvdata (pdev); +/* wait for async probe to complete */ +async_synchronize_cookie (dev->async_probe +1 );nvme_change_ctrl_state (&dev->ctrl,NVME_CTRL_DELETING );pci_set_drvdata (pdev,NULL );
Hi Sagi,
Does the async_synchronize_cookie() call wait until all previously started probes have finished? In other words, does the change in nvme_remove() introduce a dependency between probe and remove calls of different NVMe devices? Is that dependency important? If not, can it be avoided to introduce that dependency?
Thanks,
Bart.
Sagi Grimberg
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
Quoted 31 lines
Quoted 22 lines
drivers/nvme/host/pci.c +1
-0@@ -127,6 +127,7 @@ struct nvme_dev { ????? dma_addr_t host_mem_descs_dma; ????? struct nvme_host_mem_buf_desc *host_mem_descs; ????? void **host_mem_desc_bufs; +??? async_cookie_t async_probe; ? };? static int
io_queue_depth_set(const char *val, const struct kernel_param *kp)drivers/nvme/host/pci.c +1
-1@@ -2765,7 +2766,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) ????? dev_info(dev->ctrl.device, "pci function %s\n", dev_name(&pdev->dev)); ????? nvme_get_ctrl(&dev->ctrl); -??? async_schedule(nvme_async_probe, dev); +??? dev->async_probe = async_schedule(nvme_async_probe, dev); ????? return 0;drivers/nvme/host/pci.c static void nvme_remove(struct pci_dev *pdev) · line 2816
Quoted 6 lines
drivers/nvme/host/pci.c +2
-0@@ -2810,6 +2811,8 @@ static void nvme_remove(struct pci_dev *pdev) ? { ????? struct nvme_dev *dev = pci_get_drvdata(pdev); +??? /* wait for async probe to complete */ +??? async_synchronize_cookie(dev->async_probe + 1); ????? nvme_change_ctrl_state(&dev->ctrl, NVME_CTRL_DELETING); ????? pci_set_drvdata(pdev, NULL);Hi Sagi,
Quoted 5 lines
Does the
async_synchronize_cookie()call wait until all previously started probes have finished? In other words, does the change innvme_remove()introduce a dependency between probe and remove calls of different NVMe devices? Is that dependency important? If not, can it be avoided to introduce that dependency?
It does create dependency as they are in the same domain I assume. Given that nvme_remove() is really a hot-unplug operation (or shutdown) I am not sure if this dependency is a limitation in any real use-case though..
Keith Busch
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
In reply to Sagi GrimbergQuoted 26 lines · Sagi Grimberg
It is possible that
nvme_remove()being ran concurrently withnvme_reset_work(), with following sequence:
nvme_probe()
nvme_init_ctrl()//set to NEW nvme_async_probe() nvme_remove() //can not change to //DELETING from NEW nvme_reset_ctrl_sync() nvme_reset_ctrl() //change from NEW //to RESETTING flush reset_work() //not yet queued queue reset_work nvme_reset_work() .... ....With the above running concurrently, then it is possible to cause some strange issues, like kernel crash with illegal memory accessing or something like:
kernel: pci 0000:00:1f.0: can't enable device: BAR 0
[mem 0xc0000000-0xc0003fff] not claimed
Quoted 5 lines
Fix this by waiting for the async probe to complete before allowing remove to make forward progress.
Reported-by Li Zhong <lizhongfs at gmail.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>
I still think we'd prefer not adding that async domain dependency and relying on timeout to unstuck a hot-removal. So how about we schedule the reset work in probe and have the async part just flush the reset and scan work?
---
drivers/nvme/host/pci.c +2 -1
@@ -2695,7 +2695,7 @@ static void nvme_async_probe(void *data, async_cookie_t cookie) {struct nvme_dev *dev = data; -nvme_reset_ctrl_sync (&dev->ctrl); +flush_work (&dev->ctrl.reset_work);flush_work (&dev->ctrl.scan_work);nvme_put_ctrl (&dev->ctrl); }
@@ -2761,6 +2761,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)dev_info (dev->ctrl.device,"pci function %s\n" ,dev_name (&pdev->dev)); +nvme_reset_ctrl (&dev->ctrl);nvme_get_ctrl (&dev->ctrl);async_schedule (nvme_async_probe, dev);
--
Sagi Grimberg
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
Quoted 21 lines
I still think we'd prefer not adding that async domain dependency and relying on timeout to unstuck a hot-removal. So how about we schedule the reset work in probe and have the async part just flush the reset and scan work?
---
drivers/nvme/host/pci.c +1
-1@@ -2695,7 +2695,7 @@ static void nvme_async_probe(void *data, async_cookie_t cookie) {struct nvme_dev *dev = data; -nvme_reset_ctrl_sync (&dev->ctrl); +flush_work (&dev->ctrl.reset_work);flush_work (&dev->ctrl.scan_work);nvme_put_ctrl (&dev->ctrl); }
drivers/nvme/host/pci.c static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 2765
Quoted 5 lines
drivers/nvme/host/pci.c +1
-0@@ -2761,6 +2761,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)dev_info (dev->ctrl.device,"pci function %s\n" ,dev_name (&pdev->dev)); +nvme_reset_ctrl (&dev->ctrl);nvme_get_ctrl (&dev->ctrl);
I think you need to get the ref first and then fire the work right?
Keith Busch
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
Quoted 17 lines · Sagi Grimberg
Quoted 17 lines
I still think we'd prefer not adding that async domain dependency and relying on timeout to unstuck a hot-removal. So how about we schedule the reset work in probe and have the async part just flush the reset and scan work?
---
drivers/nvme/host/pci.c +1
-1@@ -2695,7 +2695,7 @@ static void nvme_async_probe(void *data, async_cookie_t cookie) {struct nvme_dev *dev = data; -nvme_reset_ctrl_sync (&dev->ctrl); +flush_work (&dev->ctrl.reset_work);flush_work (&dev->ctrl.scan_work);
Quoted 8 lines
drivers/nvme/host/pci.c static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 2763
Quoted 6 lines
drivers/nvme/host/pci.c +1
-0nvme_put_ctrl (&dev->ctrl); }@@ -2761,6 +2761,7 @@ static int nvme_probe(struct pci_dev *pdev, const struct pci_device_id *id)dev_info (dev->ctrl.device,"pci function %s\n" ,dev_name (&pdev->dev)); +nvme_reset_ctrl (&dev->ctrl);nvme_get_ctrl (&dev->ctrl);I think you need to get the ref first and then fire the work right?
That ref order doesn't actually matter here. We can't call nvme_remove during the synchronous part of probe, and the extra ref is just for the async_schedule callback.
Sagi Grimberg
maintainerPATCH 1/1 nvme-pci: fix probe and remove race
Quoted 6 lines
Quoted 2 lines
I think you need to get the ref first and then fire the work right?
That ref order doesn't actually matter here. We can't call
nvme_removeduring the synchronous part of probe, and the extra ref is just for theasync_schedulecallback.
Makes sense,
Can you send a proper patch?