avoid race between time out and tear down
cover avoid race between time out and tear down
Avoid race between time out and tear down for rdma and tcp.
Chao Leng (3):
nvme-core: introduce sync io queues
nvme-rdma: avoid race between time out and tear down
nvme-tcp: avoid race between time out and tear down
drivers/nvme/host/core.c | 8 ++++++-- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/rdma.c | 12 ++---------- drivers/nvme/host/tcp.c | 14 +++----------- 4 files changed, 12 insertions(+), 23 deletions(-)
Signature · 7 lines
2.16.4 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 2 lines
Avoid race between time out and tear down for rdma and tcp.
This patchset overall looks good, but we still need the patch that avoids double completion:
--
drivers/nvme/host/tcp.c +1 -1
@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq)/* fence other contexts that may complete the command */ mutex_lock (&to_tcp_ctrl (ctrl)->teardown_lock);nvme_tcp_stop_queue (ctrl,nvme_tcp_queue_id (req->queue)); -if (!blk_mq_request_completed (rq)) { +if (blk_mq_request_started (rq) && !blk_mq_request_completed (rq)) {nvme_req (rq)->status =NVME_SC_HOST_ABORTED_CMD ;blk_mq_complete_request_sync (rq); }
--
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 12 lines · Sagi Grimberg
Quoted 2 lines
Avoid race between time out and tear down for rdma and tcp.
This patchset overall looks good, but we still need the patch that avoids double completion:
drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static void nvme_tcp_complete_timed_out(struct request *rq) · line 2178
Quoted 5 lines
drivers/nvme/host/tcp.c +1
-1@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq) /* fence other contexts that may complete the command */ mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue)); - if (!blk_mq_request_completed(rq)) { + if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {
Yes, this patch is need. and samely for nvme_cancel_request. This will fix the race with asynchronous completion.
drivers/nvme/host/core.c +1 -1
@@ -338,7 +338,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved) "Cancelling I/O %d", req->tag);/* don't abort one completed request */ -if (blk_mq_request_completed (req)) +if (blk_mq_request_completed (req) || !blk_mq_request_started (rq))return true;nvme_req (req)->status =NVME_SC_HOST_ABORTED_CMD ;
Quoted 5 lines
drivers/nvme/host/tcp.c +0
-0nvme_req(rq)->status = NVME_SC_HOST_ABORTED_CMD; blk_mq_complete_request_sync(rq); }Signature · 1 line
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 35 lines
Quoted 12 lines
Quoted 2 lines
Avoid race between time out and tear down for rdma and tcp.
This patchset overall looks good, but we still need the patch that avoids double completion:
drivers/nvme/host/tcp.c +0
-0drivers/nvme/host/tcp.c static void nvme_tcp_complete_timed_out(struct · line 2180
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-1@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq) /* fence other contexts that may complete the command */ mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue)); - if (!blk_mq_request_completed(rq)) { + if (blk_mq_request_started(rq) && !blk_mq_request_completed (rq)) {Yes, this patch is need. and samely for
nvme_cancel_request. This will fix the race with asynchronous completion.drivers/nvme/host/core.c +1
-1@@ -338,7 +338,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved) "Cancelling I/O %d", req->tag); /* don't abort one completed request */ - if (blk_mq_request_completed(req)) + if (blk_mq_request_completed(req) || !blk_mq_request_started(rq)) return true;
Quoted 1 line
nvme_req(req)->status =
NVME_SC_HOST_ABORTED_CMD;
This one is unneeded because blk_mq_tagset_busy_iter checks that the request has started...
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 32 lines · Sagi Grimberg
Quoted 32 lines
Quoted 12 lines
Quoted 2 lines
Avoid race between time out and tear down for rdma and tcp.
This patchset overall looks good, but we still need the patch that avoids double completion:
drivers/nvme/host/tcp.c +0
-0drivers/nvme/host/tcp.c static void nvme_tcp_complete_timed_out(struct request *rq) · line 2178
Quoted 5 lines
drivers/nvme/host/tcp.c +1
-1@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq) /* fence other contexts that may complete the command */ mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue)); - if (!blk_mq_request_completed(rq)) { + if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {Yes, this patch is need. and samely for
nvme_cancel_request. This will fix the race with asynchronous completion.drivers/nvme/host/core.c +1
-1@@ -338,7 +338,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved) "Cancelling I/O %d", req->tag); /* don't abort one completed request */ - if (blk_mq_request_completed(req)) + if (blk_mq_request_completed(req) || !blk_mq_request_started(rq)) return true;
Quoted 4 lines
drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 344
Quoted 1 line
drivers/nvme/host/core.c +0
-0nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;This one is unneeded because
blk_mq_tagset_busy_iterchecks that the request has started...
Yes, it is already checked.
Quoted 1 line
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 12 lines
Quoted 2 lines
Avoid race between time out and tear down for rdma and tcp.
This patchset overall looks good, but we still need the patch that avoids double completion:
drivers/nvme/host/tcp.c +0
-0drivers/nvme/host/tcp.c static void nvme_tcp_complete_timed_out(struct · line 2180
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-1@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq) /* fence other contexts that may complete the command */ mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue)); - if (!blk_mq_request_completed(rq)) { + if (blk_mq_request_started(rq) && !blk_mq_request_completed (rq)) {Yes, this patch is need. and samely for
nvme_cancel_request. This will fix the race with asynchronous completion.drivers/nvme/host/core.c +1
-1@@ -338,7 +338,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved) "Cancelling I/O %d", req->tag); /* don't abort one completed request */ - if (blk_mq_request_completed(req)) + if (blk_mq_request_completed(req) || !blk_mq_request_started (rq)) return true;
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
nvme_req(req)->status =
NVME_SC_HOST_ABORTED_CMD;This one is unneeded because
blk_mq_tagset_busy_iterchecks that the request has started...Yes, it is already checked.
Can you add the attached patches and resend?
You can also add my:
From 53c3b144d71f7ab96619f917571fe63dbd3a023c Mon Sep 17 00:00:00 2001
From: Sagi Grimberg <[email protected]>
Date: Wed, 21 Oct 2020 01:51:29 -0700
Subject: [PATCH 2/2] nvme-rdma: fix possible double completion for timed out
requests
If error recovery ran before the timeout handler did and fully completed the timed out request, it will appear as not started (rq state is MQ_RQ_IDLE), however the timeout handler only verifies that the request was not completed (rq state is MQ_RQ_COMPLETE).
Check and make sure that the request is both started and did not yet complete.
Example trace:
------------[ cut here ]------------ refcount_t: underflow; use-after-free. WARNING: CPU: 6 PID: 45 at lib/refcount.c:28 refcount_warn_saturate+0xa6/0xf0 Workqueue: kblockd blk_mq_timeout_work RIP: 0010:refcount_warn_saturate+0xa6/0xf0 RSP: 0018:ffffb71b80837dc8 EFLAGS: 00010292 RAX: 0000000000000026 RBX: 0000000000000000 RCX: ffff93f37dcd8d08 RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff93f37dcd8d00 RBP: ffff93f37a812400 R08: 00000203c5221fce R09: ffffffffa7fffbc4 R10: 0000000000000477 R11: 000000000002835c R12: ffff93f37a8124e8 R13: ffff93f37a2b0000 R14: ffffb71b80837e70 R15: ffff93f37a2b0000 FS: 0000000000000000(0000) GS:ffff93f37dcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005637b4137028 CR3: 000000007c1be000 CR4: 00000000000006e0 Call Trace: blk_mq_check_expired+0x109/0x1b0 blk_mq_queue_tag_busy_iter+0x1b8/0x330 ? blk_poll+0x300/0x300 blk_mq_timeout_work+0x44/0xe0 process_one_work+0x1b4/0x370 worker_thread+0x53/0x3e0 ? process_one_work+0x370/0x370 kthread+0x11b/0x140 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 7d137e36e23c0d19 ]---
--- drivers/nvme/host/rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/rdma.c +1 -1
@@ -1973,7 +1973,7 @@ static void nvme_rdma_complete_timed_out(struct request *rq)/* fence other contexts that may complete the command */ mutex_lock (&ctrl->teardown_lock);nvme_rdma_stop_queue (queue); -if (!blk_mq_request_completed (rq)) { +if (blk_mq_request_started (rq) && !blk_mq_request_completed (rq)) {nvme_req (rq)->status =NVME_SC_HOST_ABORTED_CMD ;blk_mq_complete_request_sync (rq); }
Signature · 1 line
2.25.1
From d9ccd0baf15779d3dcdd488577eda8077bb7cc21 Mon Sep 17 00:00:00 2001
From: Sagi Grimberg <[email protected]>
Date: Wed, 21 Oct 2020 01:45:34 -0700
Subject: [PATCH 1/2] nvme-tcp: fix possible double completion for timed out
requests
If error recovery ran before the timeout handler did and fully completed the timed out request, it will appear as not started (rq state is MQ_RQ_IDLE), however the timeout handler only verifies that the request was not completed (rq state is MQ_RQ_COMPLETE).
Check and make sure that the request is both started and did not yet complete.
Example trace:
------------[ cut here ]------------ refcount_t: underflow; use-after-free. WARNING: CPU: 6 PID: 45 at lib/refcount.c:28 refcount_warn_saturate+0xa6/0xf0 Workqueue: kblockd blk_mq_timeout_work RIP: 0010:refcount_warn_saturate+0xa6/0xf0 RSP: 0018:ffffb71b80837dc8 EFLAGS: 00010292 RAX: 0000000000000026 RBX: 0000000000000000 RCX: ffff93f37dcd8d08 RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff93f37dcd8d00 RBP: ffff93f37a812400 R08: 00000203c5221fce R09: ffffffffa7fffbc4 R10: 0000000000000477 R11: 000000000002835c R12: ffff93f37a8124e8 R13: ffff93f37a2b0000 R14: ffffb71b80837e70 R15: ffff93f37a2b0000 FS: 0000000000000000(0000) GS:ffff93f37dcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005637b4137028 CR3: 000000007c1be000 CR4: 00000000000006e0 Call Trace: blk_mq_check_expired+0x109/0x1b0 blk_mq_queue_tag_busy_iter+0x1b8/0x330 ? blk_poll+0x300/0x300 blk_mq_timeout_work+0x44/0xe0 process_one_work+0x1b4/0x370 worker_thread+0x53/0x3e0 ? process_one_work+0x370/0x370 kthread+0x11b/0x140 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 7d137e36e23c0d19 ]---
--- drivers/nvme/host/tcp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/tcp.c +1 -1
@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq)/* fence other contexts that may complete the command */ mutex_lock (&to_tcp_ctrl (ctrl)->teardown_lock);nvme_tcp_stop_queue (ctrl,nvme_tcp_queue_id (req->queue)); -if (!blk_mq_request_completed (rq)) { +if (blk_mq_request_started (rq) && !blk_mq_request_completed (rq)) {nvme_req (rq)->status =NVME_SC_HOST_ABORTED_CMD ;blk_mq_complete_request_sync (rq); }
Signature · 1 line
2.25.1
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 36 lines · Sagi Grimberg
Quoted 36 lines
Quoted 36 lines
Quoted 25 lines
Quoted 12 lines
Quoted 2 lines
Avoid race between time out and tear down for rdma and tcp.
This patchset overall looks good, but we still need the patch that avoids double completion:
drivers/nvme/host/tcp.c +0
-0drivers/nvme/host/tcp.c static void nvme_tcp_complete_timed_out(struct request *rq) · line 2178
Quoted 5 lines
drivers/nvme/host/tcp.c +1
-1@@ -2175,7 +2175,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq) /* fence other contexts that may complete the command */ mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue)); - if (!blk_mq_request_completed(rq)) { + if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq)) {Yes, this patch is need. and samely for
nvme_cancel_request. This will fix the race with asynchronous completion.drivers/nvme/host/core.c +0
-0drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 344
Quoted 8 lines
drivers/nvme/host/core.c +1
-1@@ -338,7 +338,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved) "Cancelling I/O %d", req->tag); /* don't abort one completed request */ - if (blk_mq_request_completed(req)) + if (blk_mq_request_completed(req) || !blk_mq_request_started(rq)) return true; nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;This one is unneeded because
blk_mq_tagset_busy_iterchecks that the request has started...
Quoted 6 lines
Quoted 1 line
Yes, it is already checked.
Can you add the attached patches and resend?
You can also add my:
Reviewed-by Sagi Grimberg <[email protected]>
ok, I will do it later.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme