nvme-rdma: fix timeout handler
Sagi Grimberg
maintainerPATCH v2 1/1 nvme-rdma: fix timeout handler
Currently, we have several problems with the timeout handler:
1. If we timeout on the controller establishment flow, we will hang because we don't execute the error recovery (and we shouldn't because the create_ctrl flow needs to fail and cleanup on its own)
2. We might also hang if we get a disconnet on a queue while the controller is already deleting. This racy flow can cause the controller disable/shutdown admin command to hang.
We cannot complete a timed out request from the timeout handler without mutual exclusion from the teardown flow (e.g. nvme_rdma_error_recovery_work). So we serialize it in the timeout handler and teardown io and admin queues to guarantee that no one races with us from completing the request.
---
This is a slightly different version that looks more like pci behavior as we teardown the controller if we are connection or deleting in the timeout handler.
drivers/nvme/host/rdma.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
drivers/nvme/host/rdma.c +18 -8
@@ -1681,18 +1681,28 @@ static enum blk_eh_timer_return nvme_rdma_timeout(struct request *rq, bool reserved) {struct nvme_rdma_request *req =blk_mq_rq_to_pdu (rq); +struct nvme_rdma_queue *queue = req->queue; +struct nvme_rdma_ctrl *ctrl = queue->ctrl; -dev_warn (req->queue->ctrl->ctrl.device, -"I/O %d QID %d timeout, reset controller\n" , - rq->tag,nvme_rdma_queue_idx (req->queue)); +dev_warn (ctrl->ctrl.device,"I/O %d QID %d timeout\n" , + rq->tag,nvme_rdma_queue_idx (queue)); -/* queue error recovery */ -nvme_rdma_error_recovery (req->queue->ctrl); +if (ctrl->ctrl.state !=NVME_CTRL_LIVE ) { +/* +* teardown immediately if controller times out while starting +* or we are already started error recovery. all outstanding +* requests are completed on shutdown, so we return BLK_EH_DONE. +*/ +flush_work (&ctrl->err_work); +nvme_rdma_teardown_io_queues (ctrl, false); +nvme_rdma_teardown_admin_queue (ctrl, false); +return BLK_EH_DONE ; + } -/* fail with DNR on cmd timeout */ -nvme_req (rq)->status =NVME_SC_ABORT_REQ |NVME_SC_DNR ; +dev_warn (ctrl->ctrl.device,"starting error recovery\n" ); +nvme_rdma_error_recovery (ctrl); -return BLK_EH_DONE ; +return BLK_EH_RESET_TIMER ; } static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
Signature · 1 line
2.17.1
PATCH v2 1/1 nvme-rdma: fix timeout handler
Quoted 15 lines · Sagi Grimberg
Currently, we have several problems with the timeout handler:
1. If we timeout on the controller establishment flow, we will hang because we don't execute the error recovery (and we shouldn't because thecreate_ctrlflow needs to fail and cleanup on its own)
2. We might also hang if we get a disconnet on a queue while the controller is already deleting. This racy flow can cause the controller disable/shutdown admin command to hang.We cannot complete a timed out request from the timeout handler without mutual exclusion from the teardown flow (e.g.
nvme_rdma_error_recovery_work). So we serialize it in the timeout handler and teardown io and admin queues to guarantee that no one races with us from completing the request.
Quoted 6 lines
Reported-by Jaesoo Lee <jalee at purestorage.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>---
This is a slightly different version that looks more like pci behavior as we teardown the controller if we are connection or deleting in the timeout handler.
This looks good to me.
Quoted 44 lines
drivers/nvme/host/rdma.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)drivers/nvme/host/rdma.c +18
-8@@ -1681,18 +1681,28 @@ static enum blk_eh_timer_return nvme_rdma_timeout(struct request *rq, bool reserved) {struct nvme_rdma_request *req =blk_mq_rq_to_pdu (rq); +struct nvme_rdma_queue *queue = req->queue; +struct nvme_rdma_ctrl *ctrl = queue->ctrl; -dev_warn (req->queue->ctrl->ctrl.device, -"I/O %d QID %d timeout, reset controller\n" , - rq->tag,nvme_rdma_queue_idx (req->queue)); +dev_warn (ctrl->ctrl.device,"I/O %d QID %d timeout\n" , + rq->tag,nvme_rdma_queue_idx (queue)); -/* queue error recovery */ -nvme_rdma_error_recovery (req->queue->ctrl); +if (ctrl->ctrl.state !=NVME_CTRL_LIVE ) { +/* +* teardown immediately if controller times out while starting +* or we are already started error recovery. all outstanding +* requests are completed on shutdown, so we return BLK_EH_DONE. +*/ +flush_work (&ctrl->err_work); +nvme_rdma_teardown_io_queues (ctrl, false); +nvme_rdma_teardown_admin_queue (ctrl, false); +return BLK_EH_DONE ; + } -/* fail with DNR on cmd timeout */ -nvme_req (rq)->status =NVME_SC_ABORT_REQ |NVME_SC_DNR ; +dev_warn (ctrl->ctrl.device,"starting error recovery\n" ); +nvme_rdma_error_recovery (ctrl); -return BLK_EH_DONE ; +return BLK_EH_RESET_TIMER ; } static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx,
Sagi Grimberg
maintainerPATCH v2 1/1 nvme-rdma: fix timeout handler
In reply to Sagi GrimbergJaesoo, can you verify that this patch fixes the issue you were seeing?
A message that is not in the archive (sent off-list or not yet received): CAJX3CtgkGX_1eUU_x=v+VrUwvEA6ZwOgiPdps_-sPc5+JUFBqg@mail.gmail.com
PATCH v2 1/1 nvme-rdma: fix timeout handler
I have tested the code (backported to v4.19.9) and verified it works for below two scenarios I had originally reported.
* Scenario 1
1. issued connect nvme-cli command for nvme-rdma target with 500 namespaces
2. after 3 seconds while the scan process is still going, issued disconnect command
* Scenario 2
1. connected nvme-rdma target with 4 namespaces
2. ran IO traffic over the namespaces
3. changed MTU to 568 for the roce interface at the target
Thanks,
Jaesoo Lee.
Quoted 8 lines · Jaesoo Lee
Sure. I will test and post the results ASAP.
Quoted 3 lines · On Wed, Jan 16, 2019, 5:27 PM Sagi Grimberg <[email protected] wrote:
Jaesoo, can you verify that this patch fixes the issue you were seeing?
Sagi Grimberg
maintainerPATCH v2 1/1 nvme-rdma: fix timeout handler
Quoted 9 lines
I have tested the code (backported to v4.19.9) and verified it works for below two scenarios I had originally reported.
* Scenario 1
1. issued connect nvme-cli command for nvme-rdma target with 500 namespaces
2. after 3 seconds while the scan process is still going, issued disconnect command
Quoted 4 lines
* Scenario 2
1. connected nvme-rdma target with 4 namespaces
2. ran IO traffic over the namespaces
3. changed MTU to 568 for the roce interface at the target
Thanks!
Christoph Hellwig
maintainerPATCH v2 1/1 nvme-rdma: fix timeout handler
In reply to Sagi Grimbergdrivers/nvme/host/rdma.c static enum blk_eh_timer_return · line 1694
Quoted 5 lines
drivers/nvme/host/rdma.c +5
-0+if (ctrl->ctrl.state !=NVME_CTRL_LIVE ) { +/* +* teardown immediately if controller times out while starting +* or we are already started error recovery. all outstanding +* requests are completed on shutdown, so we return BLK_EH_DONE.
Nitpick: please start sentences with upper case letters.
Otherwise this looks fine:
PATCH v2 1/1 nvme-rdma: fix timeout handler
Quoted 9 lines
drivers/nvme/host/rdma.c static enum blk_eh_timer_return · line 1694
Quoted 6 lines
drivers/nvme/host/rdma.c +5
-0+if (ctrl->ctrl.state !=NVME_CTRL_LIVE ) { +/* +* teardown immediately if controller times out while starting +* or we are already started error recovery. all outstanding +* requests are completed on shutdown, so we return BLK_EH_DONE. Nitpick: please start sentences with upper case letters.
Quoted 3 lines
Otherwise this looks fine:
Reviewed-by Christoph Hellwig <hch at lst.de>
Applied to nvme-5.0 with the comment fix. Thanks!