nvme: fix race condition between remove and scan_work
RFC PATCH 1/1 nvme: fix race condition between remove and scan_work
There is a race condition between nvme_remove and nvme_scan_work, as following:
sysfs_kr_wirtenvme_remove nvme_queue_scan timeout_worker
queue scan_work
nvme_scan_work
set NVME_CTRL_DELETINGflush_work(&ctrl->scan_work)wait_for_completion scan_work
nvme_scan_ns_list
nvme_validate_ns
nvme_submit_sync_cmd(admin_q)
IO timeout
nvme_timeout
nvme_dev_disable
nvme_suspend_queue(admin_q)
blk_mq_quiesce_queue
nvme_reset_ctrl
return -EBUSY
nvme_scan_ns_sequential
nvme_submit_sync_cmd(admin_q)
IO cannnot issue, as admin_q suspend
wait_for_completion_io_timeout
nvme_remove() wait for nvme_scan_work() finish, and scan_work wait for io complete. When a previous IO issued in scan_work has timeout, nvme_timeout() will suspend admin queue and make the queue in quiesce state.
Since nvme controller state is 'NVME_CTRL_DELETING', nvme_reset_ctrl() can not start reset work to recovery admin queue. Then, the next io request issued to admin_q will be stalled in blk-mq 'ctx' queue forever. As a result, both of remove and scan_work threads are stuck!
We fix this problem by adding manual recovery admin queue after nvme_reset_ctrl return -EBUSY. Since ctrl.admin_q has been flags 'NVMEQ_ENABLED', all requests issused will return as io error.
--- drivers/nvme/host/pci.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/nvme/host/pci.c +8 -1
@@ -1272,6 +1272,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)struct request *abort_req;struct nvme_command cmd;u32 csts =readl (dev->bar +NVME_REG_CSTS ); +int error =0 ;/* If PCI error recovery process is happening, we cannot reset or * the recovery mechanism will surely fail.
@@ -1329,7 +1330,13 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)"I/O %d QID %d timeout, reset controller\n" , req->tag, nvmeq->qid);nvme_dev_disable (dev, false); -nvme_reset_ctrl (&dev->ctrl); +/* +* If reset ctrl fail, we need to drain all requests in ctx +* and elevator, avoiding io stuck forever. +*/ + error =nvme_reset_ctrl (&dev->ctrl); +if (error) +blk_mq_unquiesce_queue (dev->ctrl.admin_q);nvme_req (req)->flags |=NVME_REQ_CANCELLED ;return BLK_EH_DONE ;
Signature · 1 line
2.16.2.dirty
RFC PATCH 1/1 nvme: fix race condition between remove and scan_work
ping ?
Quoted 78 lines · Yufen Yu
There is a race condition between
nvme_removeandnvme_scan_work, as following:
sysfs_kr_wirtenvme_removenvme_queue_scantimeout_worker
queuescan_work
nvme_scan_workset
NVME_CTRL_DELETINGflush_work(&ctrl->scan_work)wait_for_completionscan_work
nvme_scan_ns_list
nvme_validate_ns
nvme_submit_sync_cmd(admin_q)
IO timeout
nvme_timeout
nvme_dev_disable
nvme_suspend_queue(admin_q)
blk_mq_quiesce_queue
nvme_reset_ctrl
return-EBUSY
nvme_scan_ns_sequential
nvme_submit_sync_cmd(admin_q)
IO cannnot issue, asadmin_qsuspend
wait_for_completion_io_timeout
nvme_remove()wait fornvme_scan_work()finish, andscan_workwait for io complete. When a previous IO issued inscan_workhas timeout,nvme_timeout()will suspend admin queue and make the queue in quiesce state.Since nvme controller state is '
NVME_CTRL_DELETING',nvme_reset_ctrl()can not start reset work to recovery admin queue. Then, the next io request issued toadmin_qwill be stalled in blk-mq 'ctx' queue forever. As a result, both of remove andscan_workthreads are stuck!We fix this problem by adding manual recovery admin queue after
nvme_reset_ctrlreturn-EBUSY. Since ctrl.admin_q has been flags 'NVMEQ_ENABLED', all requests issused will return as io error.Cc Keith Busch <kbusch at kernel.org>Cc Sagi Grimberg <sagi at grimberg.me>Cc James Smart <james.smart at broadcom.com>Cc Bart Van Assche <bvanassche at acm.org>Signed-off-by Yufen Yu <yuyufen at huawei.com>--- drivers/nvme/host/pci.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/nvme/host/pci.c +8
-1@@ -1272,6 +1272,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)struct request *abort_req;struct nvme_command cmd;u32 csts =readl (dev->bar +NVME_REG_CSTS ); +int error =0 ;/* If PCI error recovery process is happening, we cannot reset or * the recovery mechanism will surely fail. @@ -1329,7 +1330,13 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)"I/O %d QID %d timeout, reset controller\n" , req->tag, nvmeq->qid);nvme_dev_disable (dev, false); -nvme_reset_ctrl (&dev->ctrl); +/* +* If reset ctrl fail, we need to drain all requests in ctx +* and elevator, avoiding io stuck forever. +*/ + error =nvme_reset_ctrl (&dev->ctrl); +if (error) +blk_mq_unquiesce_queue (dev->ctrl.admin_q);nvme_req (req)->flags |=NVME_REQ_CANCELLED ;return BLK_EH_DONE ;
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: fix race condition between remove and scan_work
In reply to Yufen YuQuoted 22 lines · Yufen Yu
There is a race condition between
nvme_removeandnvme_scan_work, as following:
sysfs_kr_wirtenvme_removenvme_queue_scantimeout_worker
queuescan_work
nvme_scan_workset
NVME_CTRL_DELETINGflush_work(&ctrl->scan_work)wait_for_completionscan_work
nvme_scan_ns_list
nvme_validate_ns
nvme_submit_sync_cmd(admin_q)
IO timeout
nvme_timeoutnvme_dev_disable
nvme_suspend_queue(admin_q)
blk_mq_quiesce_queue
Quoted 8 lines
nvme_reset_ctrlreturn-EBUSY
nvme_scan_ns_sequentialnvme_submit_sync_cmd(admin_q)
IO cannnot issue, asadmin_qsuspend
wait_for_completion_io_timeout
Should we still attempt nvme_scan_ns_sequential() if nvme_scan_ns_list() failed?
I still think that I/Os cannot hang forever when issued concurrently with controller removal...
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1275
Quoted 9 lines
drivers/nvme/host/pci.c +1
-0@@ -1272,6 +1272,7 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)struct request *abort_req;struct nvme_command cmd;u32 csts =readl (dev->bar +NVME_REG_CSTS ); +int error =0 ;
This can move to the usage scope right?
Quoted 5 lines
drivers/nvme/host/pci.c +0
-0/* If PCI error recovery process is happening, we cannot reset or * the recovery mechanism will surely fail. @@ -1329,7 +1330,13 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) "I/O %d QID %d timeout, reset controller\n",
drivers/nvme/host/pci.c static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1339
Quoted 10 lines
drivers/nvme/host/pci.c +7
-1req->tag, nvmeq->qid);nvme_dev_disable (dev, false); -nvme_reset_ctrl (&dev->ctrl); +/* +* If reset ctrl fail, we need to drain all requests in ctx +* and elevator, avoiding io stuck forever. +*/ + error =nvme_reset_ctrl (&dev->ctrl); +if (error) +blk_mq_unquiesce_queue (dev->ctrl.admin_q);
Is it just DELETING state that is acceptable here? or can we meet other states that fail transition to RESETTING (CONNECTING/DEAD)?
Keith Busch
maintainerRFC PATCH 1/1 nvme: fix race condition between remove and scan_work
Quoted 9 lines · Sagi Grimberg
Quoted 9 lines
/* If PCI error recovery process is happening, we cannot reset or
* the recovery mechanism will surely fail.(quoted diff) +2
-1@@ -1329,7 +1330,13 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) "I/O %d QID %d timeout, reset controller\n", req->tag, nvmeq->qid); nvme_dev_disable(dev, false); - nvme_reset_ctrl(&dev->ctrl); + /* + * If reset ctrl fail, we need to drain all requests in ctx
Quoted 8 lines
Review static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved) · line 1339
Quoted 5 lines
(quoted diff) +5
-0+ * and elevator, avoiding io stuck forever. + */ + error = nvme_reset_ctrl(&dev->ctrl); + if (error) + blk_mq_unquiesce_queue(dev->ctrl.admin_q);Is it just DELETING state that is acceptable here? or can we meet other states that fail transition to RESETTING (CONNECTING/DEAD)?
It could be connecting or already scheduled resetting, in which case we wouldn't want to unquiesce.
When we do want to unquiesce, though, we also want to do that to the IO queues, not just the admin queue. Untested below, but this might be in the right direction:
---
drivers/nvme/host/pci.c +8 -1
@@ -1315,6 +1315,10 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)nvme_dev_disable (dev, false);nvme_req (req)->flags |=NVME_REQ_CANCELLED ;return BLK_EH_DONE ; +case NVME_CTRL_DELETING : +nvme_dev_disable (dev, true); +nvme_req (req)->flags |=NVME_REQ_CANCELLED ; +return BLK_EH_DONE ;default :break ; }
@@ -2438,8 +2442,11 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)* must flush all entered requests to their failed completion to avoid * deadlocking blk-mq hot-cpu notifier. */ -if (shutdown) +if (shutdown) {nvme_start_queues (&dev->ctrl); +if (dev->ctrl.admin_q) +blk_mq_unquiesce_queue (dev->ctrl.admin_q); + }mutex_unlock (&dev->shutdown_lock); }
--
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: fix race condition between remove and scan_work
Quoted 10 lines
Quoted 3 lines
Is it just DELETING state that is acceptable here? or can we meet other states that fail transition to RESETTING (CONNECTING/DEAD)?
It could be connecting or already scheduled resetting, in which case we wouldn't want to unquiesce.
When we do want to unquiesce, though, we also want to do that to the IO queues, not just the admin queue. Untested below, but this might be in the right direction:
This looks good Keith, Yufen, does that address your race?
Quoted 37 lines
---
drivers/nvme/host/pci.c +8
-2@@ -1315,6 +1315,10 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)nvme_dev_disable (dev, false);nvme_req (req)->flags |=NVME_REQ_CANCELLED ;return BLK_EH_DONE ; +case NVME_CTRL_DELETING : +nvme_dev_disable (dev, true); +nvme_req (req)->flags |=NVME_REQ_CANCELLED ; +return BLK_EH_DONE ;default :break ; }@@ -2438,8 +2442,11 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)* must flush all entered requests to their failed completion to avoid * deadlocking blk-mq hot-cpu notifier. */ -if (shutdown) +if (shutdown) {nvme_start_queues (&dev->ctrl); +if (dev->ctrl.admin_q) +blk_mq_unquiesce_queue (dev->ctrl.admin_q); + }mutex_unlock (&dev->shutdown_lock); } --_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
RFC PATCH 1/1 nvme: fix race condition between remove and scan_work
Quoted 4 lines · Sagi Grimberg
Quoted 4 lines
Quoted 3 lines
Is it just DELETING state that is acceptable here? or can we meet other states that fail transition to RESETTING (CONNECTING/DEAD)?
Quoted 8 lines
Quoted 6 lines
It could be connecting or already scheduled resetting, in which case we wouldn't want to unquiesce.
When we do want to unquiesce, though, we also want to do that to the IO queues, not just the admin queue. Untested below, but this might be in the right direction:
This looks good Keith, Yufen, does that address your race?
Thanks a lot for your response and suggestion.
The follow patch can address our problem.:-)
Yufen
thanks
Quoted 43 lines
Quoted 35 lines
---
drivers/nvme/host/pci.c +4
-0@@ -1315,6 +1315,10 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)nvme_dev_disable (dev, false);nvme_req (req)->flags |=NVME_REQ_CANCELLED ;return BLK_EH_DONE ; +case NVME_CTRL_DELETING : +nvme_dev_disable (dev, true); +nvme_req (req)->flags |=NVME_REQ_CANCELLED ; +return BLK_EH_DONE ;default :break ;}drivers/nvme/host/pci.c +0
-0@@ -2438,8 +2442,11 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) * must flush all entered requests to their failed completionto avoid
* deadlocking blk-mq hot-cpu notifier.
*/drivers/nvme/host/pci.c +4
-1-if (shutdown) +if (shutdown) {nvme_start_queues (&dev->ctrl); +if (dev->ctrl.admin_q) +blk_mq_unquiesce_queue (dev->ctrl.admin_q); + }mutex_unlock (&dev->shutdown_lock); } --Quoted 5 lines
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme.
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: fix race condition between remove and scan_work
Quoted 5 lines
Quoted 2 lines
This looks good Keith, Yufen, does that address your race?
Thanks a lot for your response and suggestion.
The follow patch can address our problem.:-)
Keith, care to send a patch?