v4.16-rc2 nvme_rdma ib_destroy_qp() warns about MRs
Hello,
With the v4.16-rc2 nvme_rdma driver on top of the rdma_rxe driver the following kernel warning appeared in the kernel log:
CPU: 3 PID: 152 Comm: kworker/u8:3 Not tainted 4.16.0-rc2-dbg+ #3 Workqueue: nvme-wq nvme_rdma_error_recovery_work [nvme_rdma] RIP: 0010:ib_destroy_qp+0x177/0x1a0 [ib_core] Call Trace: nvme_rdma_destroy_queue_ib+0x32/0x70 [nvme_rdma] nvme_rdma_free_queue+0x2e/0x90 [nvme_rdma] nvme_rdma_destroy_io_queues+0x5d/0xb0 [nvme_rdma] nvme_rdma_error_recovery_work+0x4c/0xb0 [nvme_rdma] process_one_work+0x20b/0x6a0 worker_thread+0x35/0x380 kthread+0x117/0x130 ret_from_fork+0x24/0x30
Does this mean that the nvme_rdma driver calls ib_destroy_qp() before all MRs associated with the QP have been destroyed?
Thanks,
Bart.
Quoted 18 lines · Bart Van Assche
Hello,
With the v4.16-rc2
nvme_rdmadriver on top of therdma_rxedriver the following kernel warning appeared in the kernel log:CPU: 3 PID: 152 Comm: kworker/u8:3 Not tainted 4.16.0-rc2-dbg+ #3 Workqueue: nvme-wq nvme_rdma_error_recovery_work [nvme_rdma] RIP: 0010:ib_destroy_qp+0x177/0x1a0 [ib_core] Call Trace: nvme_rdma_destroy_queue_ib+0x32/0x70 [nvme_rdma] nvme_rdma_free_queue+0x2e/0x90 [nvme_rdma] nvme_rdma_destroy_io_queues+0x5d/0xb0 [nvme_rdma] nvme_rdma_error_recovery_work+0x4c/0xb0 [nvme_rdma] process_one_work+0x20b/0x6a0 worker_thread+0x35/0x380 kthread+0x117/0x130 ret_from_fork+0x24/0x30
Quoted 2 lines
Does this mean that the
nvme_rdmadriver callsib_destroy_qp()before all MRs associated with the QP have been destroyed?
I guess it happens to you in some port toggling during IO (or some other recovery scenario).
The answer is yes, and I've started looking into it and started a thread:
https://www.mail-archive.com/linux-block at vger.kernel.org/msg18559.html
Quoted 5 lines
Thanks,
Bart.
Thanks,
-Max.
Quoted 6 lines
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7C64b749f9d50d4cc8f0c808d57b1ce8a6%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636550286469377682&sdata=DGnEOBEX7Y3GyHByLA68wFb8nCtYVsX1kYtyoHgelTA%3D&reserved=0
Sagi Grimberg
maintainerQuoted 2 lines
Hello,
Hi Bart,
Quoted 15 lines
With the v4.16-rc2
nvme_rdmadriver on top of therdma_rxedriver the following kernel warning appeared in the kernel log:CPU: 3 PID: 152 Comm: kworker/u8:3 Not tainted 4.16.0-rc2-dbg+ #3 Workqueue: nvme-wq nvme_rdma_error_recovery_work [nvme_rdma] RIP: 0010:ib_destroy_qp+0x177/0x1a0 [ib_core] Call Trace: nvme_rdma_destroy_queue_ib+0x32/0x70 [nvme_rdma] nvme_rdma_free_queue+0x2e/0x90 [nvme_rdma] nvme_rdma_destroy_io_queues+0x5d/0xb0 [nvme_rdma] nvme_rdma_error_recovery_work+0x4c/0xb0 [nvme_rdma] process_one_work+0x20b/0x6a0 worker_thread+0x35/0x380 kthread+0x117/0x130 ret_from_fork+0x24/0x30
Thanks for reporting.
Quoted 2 lines
Does this mean that the
nvme_rdmadriver callsib_destroy_qp()before all MRs associated with the QP have been destroyed?
That's the warning... But I'm having troubles understanding how can this be a nvme-rdma issue. We only allocate in .queue_rq if we passed which means that the queue has READY on, and before we destroy the qp only after we:
1. quiesced all the request queues
2. cancel all started requests (which trigger nvme_rdma_complete_request that returns the mr to the pool)
So the only way I see that we can get here, is if blk_mq_complete_request does not call __blk_mq_complete_request.
This can happen when:
--
/*
* If @rq->aborted_gstate equals the current instance, timeout is
* claiming @rq and we lost. This is synchronized through
* hctx_lock(). See blk_mq_timeout_work() for details. *
* Completion path never blocks and we can directly use RCU here
* instead of hctx_lock() which can be either RCU or SRCU.
* However, that would complicate paths which want to synchronize
* against us. Let stay in sync with the issue path so that
* hctx_lock() covers both issue and completion paths. */
hctx_lock(hctx, &srcu_idx);
if (blk_mq_rq_aborted_gstate(rq) != rq->gstate)
__blk_mq_complete_request(rq);
hctx_unlock(hctx, srcu_idx);
--
Does this mean that the block driver must not assume that .complete will be called on a timed out request for sure?
Is this easy to reproduce Bart? Does this patch help? --
drivers/nvme/host/rdma.c +5 -1
@@ -1585,6 +1585,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved)"I/O %d QID %d timeout, reset controller\n" , rq->tag, nvme_rdma_queue_idx(req->queue)); +if (req->mr) { +ib_mr_pool_put (queue->qp, &queue->qp->rdma_mrs, req->mr); + req->mr =NULL ; + } +/* queue error recovery */ nvme_rdma_error_recovery (req->queue->ctrl); --
Sagi Grimberg
maintainerQuoted 68 lines · Sagi Grimberg
Quoted 2 lines
Hello,
Hi Bart,
Quoted 15 lines
With the v4.16-rc2
nvme_rdmadriver on top of therdma_rxedriver the following kernel warning appeared in the kernel log:CPU: 3 PID: 152 Comm: kworker/u8:3 Not tainted 4.16.0-rc2-dbg+ #3 Workqueue: nvme-wq nvme_rdma_error_recovery_work [nvme_rdma] RIP: 0010:ib_destroy_qp+0x177/0x1a0 [ib_core] Call Trace: ? nvme_rdma_destroy_queue_ib+0x32/0x70 [nvme_rdma] ? nvme_rdma_free_queue+0x2e/0x90 [nvme_rdma] ? nvme_rdma_destroy_io_queues+0x5d/0xb0 [nvme_rdma] ? nvme_rdma_error_recovery_work+0x4c/0xb0 [nvme_rdma] ? process_one_work+0x20b/0x6a0 ? worker_thread+0x35/0x380 ? kthread+0x117/0x130 ? ret_from_fork+0x24/0x30Thanks for reporting.
Quoted 3 lines
Does this mean that the
nvme_rdmadriver callsib_destroy_qp()before all MRs
associated with the QP have been destroyed?That's the warning... But I'm having troubles understanding how can this be a nvme-rdma issue. We only allocate in .queue_rq if we passed which means that the queue has READY on, and before we destroy the qp only after we:
1. quiesced all the request queues
2. cancel all started requests (which triggernvme_rdma_complete_requestthat returns the mr to the pool)So the only way I see that we can get here, is if
blk_mq_complete_requestdoes not call__blk_mq_complete_request.This can happen when:
Signature · 22 lines
??????? /* ???????? * If @rq->aborted_gstate equals the current instance, timeout is ???????? * claiming @rq and we lost.? This is synchronized through ???????? * hctx_lock().? See blk_mq_timeout_work() for details. ???????? * ???????? * Completion path never blocks and we can directly use RCU here ???????? * instead of hctx_lock() which can be either RCU or SRCU. ???????? * However, that would complicate paths which want to synchronize ???????? * against us.? Let stay in sync with the issue path so that ???????? * hctx_lock() covers both issue and completion paths. ???????? */ ??????? hctx_lock(hctx, &srcu_idx); ??????? if (blk_mq_rq_aborted_gstate(rq) != rq->gstate) ??????????????? __blk_mq_complete_request(rq); ??????? hctx_unlock(hctx, srcu_idx); -- Does this mean that the block driver must not assume that .complete will be called on a timed out request for sure? Is this easy to reproduce Bart? Does this patch help? --drivers/nvme/host/rdma.c +0
-0
drivers/nvme/host/rdma.c nvme_rdma_timeout(struct request *rq, bool reserved) · line 1594
Quoted 10 lines
drivers/nvme/host/rdma.c +5
-0@@ -1585,6 +1585,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved) ???????????????? "I/O %d QID %d timeout, reset controller\n", ???????????????? rq->tag, nvme_rdma_queue_idx(req->queue)); +?????? if (req->mr) { +?????????????? ib_mr_pool_put(queue->qp, &queue->qp->rdma_mrs, req->mr); +?????????????? req->mr = NULL; +?????? } + ??????? /* queue error recovery */ ??????? nvme_rdma_error_recovery(req->queue->ctrl);
Now with a patch that actually compiles :)
--
drivers/nvme/host/rdma.c +2 -0
@@ -1585,6 +1585,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved)"I/O %d QID %d timeout, reset controller\n" , rq->tag, nvme_rdma_queue_idx(req->queue)); +if (req->mr) { + ib_mr_pool_put(req->queue->qp,
&req->queue->qp->rdma_mrs, req->mr);
+ req->mr = NULL;
+ }
+
/* queue error recovery */
nvme_rdma_error_recovery(req->queue->ctrl);--
hi Sagi,
I tried this patch but it still doesn't fix the issue.
this is the same issue I reported and discussed with Ming:
https://www.mail-archive.com/linux-block at vger.kernel.org/msg18559.html
we must somehow drain the commands that were pushed between blk_mq_quiesce_queue to blk_mq_unquiesce_queue. The timeout expires after we destroy the QP and there are MR missing in the pool.
I'll get back to this after SDC EMEA :)
-Max.
Quoted 110 lines · Sagi Grimberg
Quoted 78 lines · Sagi Grimberg
Quoted 2 lines
Hello,
Hi Bart,
Quoted 15 lines
With the v4.16-rc2
nvme_rdmadriver on top of therdma_rxedriver the following kernel warning appeared in the kernel log:CPU: 3 PID: 152 Comm: kworker/u8:3 Not tainted 4.16.0-rc2-dbg+ #3 Workqueue: nvme-wq nvme_rdma_error_recovery_work [nvme_rdma] RIP: 0010:ib_destroy_qp+0x177/0x1a0 [ib_core] Call Trace: ? nvme_rdma_destroy_queue_ib+0x32/0x70 [nvme_rdma] ? nvme_rdma_free_queue+0x2e/0x90 [nvme_rdma] ? nvme_rdma_destroy_io_queues+0x5d/0xb0 [nvme_rdma] ? nvme_rdma_error_recovery_work+0x4c/0xb0 [nvme_rdma] ? process_one_work+0x20b/0x6a0 ? worker_thread+0x35/0x380 ? kthread+0x117/0x130 ? ret_from_fork+0x24/0x30Thanks for reporting.
Quoted 3 lines
Does this mean that the
nvme_rdmadriver callsib_destroy_qp()before all MRs
associated with the QP have been destroyed?That's the warning... But I'm having troubles understanding how can this be a nvme-rdma issue. We only allocate in .queue_rq if we passed which means that the queue has READY on, and before we destroy the qp only after we:
1. quiesced all the request queues
2. cancel all started requests (which triggernvme_rdma_complete_requestthat returns the mr to the pool)So the only way I see that we can get here, is if
blk_mq_complete_requestdoes not call__blk_mq_complete_request.This can happen when:
Signature · 24 lines
???????? /* ????????? * If @rq->aborted_gstate equals the current instance, timeout is ????????? * claiming @rq and we lost.? This is synchronized through ????????? * hctx_lock().? See blk_mq_timeout_work() for details. ????????? * ????????? * Completion path never blocks and we can directly use RCU here ????????? * instead of hctx_lock() which can be either RCU or SRCU. ????????? * However, that would complicate paths which want to synchronize ????????? * against us.? Let stay in sync with the issue path so that ????????? * hctx_lock() covers both issue and completion paths. ????????? */ ???????? hctx_lock(hctx, &srcu_idx); ???????? if (blk_mq_rq_aborted_gstate(rq) != rq->gstate) ???????????????? __blk_mq_complete_request(rq); ???????? hctx_unlock(hctx, srcu_idx); -- Does this mean that the block driver must not assume that .complete will be called on a timed out request for sure? Is this easy to reproduce Bart? Does this patch help? --drivers/nvme/host/rdma.c +5
-0@@ -1585,6 +1585,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved) ????????????????? "I/O %d QID %d timeout, reset controller\n", ????????????????? rq->tag, nvme_rdma_queue_idx(req->queue)); +?????? if (req->mr) { +?????????????? ib_mr_pool_put(queue->qp, &queue->qp->rdma_mrs, req->mr); +?????????????? req->mr = NULL; +?????? } +Quoted 3 lines
???????? /* queue error recovery */
????????nvme_rdma_error_recovery(req->queue->ctrl);Now with a patch that actually compiles :)
drivers/nvme/host/rdma.c +5
-0@@ -1585,6 +1585,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved) ???????????????? "I/O %d QID %d timeout, reset controller\n", ???????????????? rq->tag, nvme_rdma_queue_idx(req->queue)); +?????? if (req->mr) { +?????????????? ib_mr_pool_put(req->queue->qp, &req->queue->qp->rdma_mrs, req->mr); +?????????????? req->mr = NULL; +?????? } + ??????? /* queue error recovery */ ??????? nvme_rdma_error_recovery(req->queue->ctrl);Signature · 4 lines
_______________________________________________ Linux-nvme mailing list Linux-nvme at lists.infradead.org https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7Cf5862c14e9174ef33a8c08d57c7a8817%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636551788090800209&sdata=QTzZ4zaOYXajnh1HVIaIYhnjOE3t3TUr9hGSU2xW%2FPI%3D&reserved=0
Sagi Grimberg
maintainerQuoted 1 line
hi Sagi,
Hi Max,
Quoted 3 lines
I tried this patch but it still doesn't fix the issue.
this is the same issue I reported and discussed with Ming:
https://www.mail-archive.com/linux-block at vger.kernel.org/msg18559.html
Didn't follow the thread, wasn't CC'd on it.
Quoted 3 lines
we must somehow drain the commands that were pushed between
blk_mq_quiesce_queuetoblk_mq_unquiesce_queue. The timeout expires after we destroy the QP and there are MR missing in the pool.
Its a bug if .queue_rq is being invoked when the queue is quiesced. Have you actually seen this happen? IFF this is really happening, we can stop the rdma queues before we iterate on started requests (see a patch I suggested to Alon). With this, even requests that sneak in when the queues are quiesced will fail on nvme_rdma_is_ready(). but as said, I'm surprised this is even happening.
Quoted 1 line · Sagi Grimberg
Is this easy to reproduce Bart?
Yes, this behavior is 100% reproducible.
Quoted 7 lines
Does this patch help?
drivers/nvme/host/rdma.c +0
-0
drivers/nvme/host/rdma.c nvme_rdma_timeout(struct request *rq, bool reserved) · line 1594
Quoted 10 lines
drivers/nvme/host/rdma.c +5
-0@@ -1585,6 +1585,11 @@ nvme_rdma_timeout(struct request *rq, bool reserved) ???????????????? "I/O %d QID %d timeout, reset controller\n", ???????????????? rq->tag, nvme_rdma_queue_idx(req->queue)); +?????? if (req->mr) { +?????????????? ib_mr_pool_put(queue->qp, &queue->qp->rdma_mrs, req->mr); +?????????????? req->mr = NULL; +?????? } + ??????? /* queue error recovery */ ??????? nvme_rdma_error_recovery(req->queue->ctrl);
Hello Sagi,
Has this patch perhaps been superseded by the BLK_EH_RESET_TIMER patch that you sent to Alon?
Thanks,
Bart.