Handle update hardware queues and queue freeze more carefully
Subsystems (2)
v4 cover Handle update hardware queues and queue freeze more carefully
[forgot to add the linux-nvme mailing list]
Hi,
This update version makes sure the unfreeze call is done when recreating the queues. I was able to reproduce hanging I/Os when we go into error recovery mode for FC and TCP [1]. Unfortunatly, I don't have access to a RDMA setup to verify but as the code is identically to the TCP, RDMA looks to like to suffer from the same problem.
Thanks,
Daniel
[1] https://lore.kernel.org/linux-nvme/[email protected]/
https://lore.kernel.org/linux-nvme/[email protected]/
v1:
- https://lore.kernel.org/linux-nvme/[email protected]/
v2:
- https://lore.kernel.org/linux-nvme/[email protected]/
- reviewed tags collected
- added 'update hardware queues' for all transport
- added fix for fc hanger in nvme_wait_freeze_timeout
v3:
- https://lore.kernel.org/linux-nvme/[email protected]/
- dropped 'nvme-fc: Freeze queues before destroying them'
- added James' two patches
v4:
- added 'nvme-*: Unfreeze queues on reconnect'
- added Hannes' reviewed tags
Daniel Wagner (5):
nvme-fc: Update hardware queues before using them
nvme-rdma: Update number of hardware queues before using them
nvme-fc: Wait with a timeout for queue to freeze
nvme-tcp: Unfreeze queues on reconnect
nvme-rdma: Unfreeze queues on reconnect
Hannes Reinecke (1):
nvme-tcp: Update number of hardware queues before using them
James Smart (2):
nvme-fc: avoid race between time out and tear down
nvme-fc: fix controller reset hang during traffic
drivers/nvme/host/fc.c | 28 +++++++++++++++++++--------- drivers/nvme/host/rdma.c | 15 ++++++++------- drivers/nvme/host/tcp.c | 18 +++++++++--------- 3 files changed, 36 insertions(+), 25 deletions(-)
Signature · 1 line
2.29.2
PATCH v4 1/8 nvme-fc: Update hardware queues before using them
In case the number of hardware queues changes, do the update the tagset and ctx to hctx first before using the mapping to recreate and connnect the IO queues.
--- drivers/nvme/host/fc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/nvme/host/fc.c +8 -8
@@ -2951,14 +2951,6 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)if (ctrl->ctrl.queue_count ==1 )return 0 ; - ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); -if (ret) - goto out_free_io_queues; - - ret =nvme_fc_connect_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); -if (ret) - goto out_delete_hw_queues; -if (prior_ioq_cnt != nr_io_queues) { dev_info(ctrl->ctrl.device, "reconnect: revising io queue count from %d to %d\n",
@@ -2968,6 +2960,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)nvme_unfreeze (&ctrl->ctrl); } + ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); +if (ret) + goto out_free_io_queues; + + ret =nvme_fc_connect_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); +if (ret) + goto out_delete_hw_queues; +return 0 ; out_delete_hw_queues:
Signature · 1 line
2.29.2
Re: [PATCH v4 1/8] nvme-fc: Update hardware queues before using them
Quoted 37 lines · Daniel Wagner
In case the number of hardware queues changes, do the update the tagset and ctx to hctx first before using the mapping to recreate and connnect the IO queues.
Reviewed-by James Smart <[email protected]>Reviewed-by Ming Lei <[email protected]>Reviewed-by Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)drivers/nvme/host/fc.c +1
-8@@ -2951,14 +2951,6 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)if (ctrl->ctrl.queue_count ==1 )return 0 ; - ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); -if (ret) - goto out_free_io_queues; - - ret =nvme_fc_connect_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); -if (ret) - goto out_delete_hw_queues; -if (prior_ioq_cnt != nr_io_queues) { dev_info(ctrl->ctrl.device, "reconnect: revising io queue count from %d to %d\n",@@ -2968,6 +2960,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)nvme_unfreeze (&ctrl->ctrl); } + ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 );
drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2973
Quoted 10 lines
drivers/nvme/host/fc.c +7
-0+if (ret) + goto out_free_io_queues; + + ret =nvme_fc_connect_io_queues (ctrl, ctrl->ctrl.sqsize +1 ); +if (ret) + goto out_delete_hw_queues; +return 0 ; out_delete_hw_queues:
Signature · 6 lines
Himanshu Madhani Oracle Linux Engineering _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v4 2/8 nvme-tcp: Update number of hardware queues before using them
In reply to Daniel WagnerFrom: Hannes Reinecke <[email protected]>
When the number of hardware queues changes during resetting we should update the tagset first before using it.
--- drivers/nvme/host/tcp.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
drivers/nvme/host/tcp.c +6 -8
@@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove)static int nvme_tcp_configure_io_queues (struct nvme_ctrl *ctrl,bool new) {int ret; +u32 prior_q_cnt = ctrl->queue_count; ret =nvme_tcp_alloc_io_queues (ctrl);if (ret)
@@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) ret =PTR_ERR (ctrl->connect_q); goto out_free_tag_set; } - } - - ret =nvme_tcp_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { -nvme_start_queues (ctrl); + }else if (prior_q_cnt != ctrl->queue_count) {if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) {/* * If we timed out waiting for freeze we are likely to
@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)nvme_unfreeze (ctrl); } + ret =nvme_tcp_start_io_queues (ctrl); +if (ret) + goto out_cleanup_connect_q; +return 0 ; out_wait_freeze_timed_out:
Signature · 1 line
2.29.2
Sagi Grimberg
maintainerRe: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Quoted 27 lines
From: Hannes Reinecke <[email protected]>
When the number of hardware queues changes during resetting we should update the tagset first before using it.
Signed-off-by Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/tcp.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)drivers/nvme/host/tcp.c +1
-0@@ -1789,6 +1789,7 @@ static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove)static int nvme_tcp_configure_io_queues (struct nvme_ctrl *ctrl,bool new) {int ret; +u32 prior_q_cnt = ctrl->queue_count; ret =nvme_tcp_alloc_io_queues (ctrl);if (ret)@@ -1806,14 +1807,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) ret =PTR_ERR (ctrl->connect_q); goto out_free_tag_set;
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810
Quoted 10 lines
drivers/nvme/host/tcp.c +1
-8} - } - - ret =nvme_tcp_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { -nvme_start_queues (ctrl); + }else if (prior_q_cnt != ctrl->queue_count) {
So if the queue count did not change we don't wait to make sure the queue g_usage_counter ref made it to zero? What guarantees that it did?
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1828
Quoted 11 lines
drivers/nvme/host/tcp.c +4
-0if (!nvme_wait_freeze_timeout (ctrl,NVME_IO_TIMEOUT )) {/* * If we timed out waiting for freeze we are likely to @@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)nvme_unfreeze (ctrl); } + ret =nvme_tcp_start_io_queues (ctrl); +if (ret) + goto out_cleanup_connect_q; +
Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?
If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress.
Quoted 4 lines
drivers/nvme/host/tcp.c +0
-0return 0 ; out_wait_freeze_timed_out:
Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Hi Sagi,
Quoted 11 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-6- ret =nvme_tcp_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { -nvme_start_queues (ctrl); + }else if (prior_q_cnt != ctrl->queue_count) {So if the queue count did not change we don't wait to make sure the queue
g_usage_counterref made it to zero? What guarantees that it did?
Hmm, good point. we should always call nvme_wait_freeze_timeout() for !new queues. Is this what you are implying?
Quoted 5 lines
Quoted 5 lines
if (!nvme_wait_freeze_timeout(ctrl,
NVME_IO_TIMEOUT)) {
/*
* If we timed out waiting for freeze we are likely todrivers/nvme/host/tcp.c +0
-0@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)nvme_unfreeze (ctrl);
Quoted 8 lines
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1828
Quoted 5 lines
drivers/nvme/host/tcp.c +4
-0} + ret =nvme_tcp_start_io_queues (ctrl); +if (ret) + goto out_cleanup_connect_q; +Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?
Not sure if this classifies as heavy I/O (on 80 CPU machine)
fio --rw=readwrite --name=test --filename=/dev/nvme16n1 --size=50M \
--direct=1 --bs=4k --numjobs=40 --group_reporting --runtime=4h \
--time_basedand then I installed iptables rules to block the traffic on the controller side. With this test it is pretty easily to get the host hanging. Let me know what test you would like to see from me. I am glad to try to get them running.
Quoted 3 lines
If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress.
So the unfreeze should happen after the connect call? What about the newly created queues? Do they not suffer from the same problem? Isn't the NVME_TCP_Q_LIVE flag not enough?
Daniel
Sagi Grimberg
maintainerRe: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Quoted 15 lines · Daniel Wagner
Hi Sagi,
Quoted 11 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-6- ret =nvme_tcp_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { -nvme_start_queues (ctrl); + }else if (prior_q_cnt != ctrl->queue_count) {So if the queue count did not change we don't wait to make sure the queue
g_usage_counterref made it to zero? What guarantees that it did?
Quoted 2 lines
Hmm, good point. we should always call
nvme_wait_freeze_timeout()for !new queues. Is this what you are implying?
I think we should always wait for the freeze to complete.
Quoted 18 lines
Quoted 15 lines
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1827
Quoted 12 lines
if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) { /* * If we timed out waiting for freeze we are likely todrivers/nvme/host/tcp.c +4
-0@@ -1828,6 +1822,10 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new)nvme_unfreeze (ctrl); } + ret =nvme_tcp_start_io_queues (ctrl); +if (ret) + goto out_cleanup_connect_q; +Did you test this with both heavy I/O, reset loop and ifdown/ifup loop?
Not sure if this classifies as heavy I/O (on 80 CPU machine)
Quoted 8 lines
fio --rw=readwrite --name=test --filename=/dev/nvme16n1 --size=50M \ --direct=1 --bs=4k --numjobs=40 --group_reporting --runtime=4h \ --time_basedand then I installed iptables rules to block the traffic on the controller side. With this test it is pretty easily to get the host hanging. Let me know what test you would like to see from me. I am glad to try to get them running.
Lets add iodepth=128
Quoted 7 lines
Quoted 3 lines
If we unquiesce and unfreeze before we start the queues the pending I/Os may resume before the connect and not allow the connect to make forward progress.
So the unfreeze should happen after the connect call? What about the newly created queues? Do they not suffer from the same problem? Isn't the
NVME_TCP_Q_LIVEflag not enough?
Q_LIVE will protect against the transport itself from queueing, however when multipath is not used, the transport will return BLK_STS_RESOURCE which will immediately trigger re-submission, in an endless loop, and that can prevent forward progress. It is also consistent with what nvme-pci does.
Keith Busch
maintainerRe: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Quoted 18 lines · Sagi Grimberg
Quoted 15 lines · Daniel Wagner
Hi Sagi,
Quoted 11 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-6- ret =nvme_tcp_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { -nvme_start_queues (ctrl); + }else if (prior_q_cnt != ctrl->queue_count) {So if the queue count did not change we don't wait to make sure the queue
g_usage_counterref made it to zero? What guarantees that it did?
Quoted 4 lines
Quoted 2 lines
Hmm, good point. we should always call
nvme_wait_freeze_timeout()for !new queues. Is this what you are implying?I think we should always wait for the freeze to complete.
Don't the queues need to be started in order for the freeze to complete? Any enqueued requests on the quiesced queues will never complete this way, so the wait_freeze() will be stuck, right? If so, I think the nvme_start_queues() was in the correct place already.
Sagi Grimberg
maintainerRe: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Quoted 20 lines
Quoted 20 lines · Daniel Wagner
Quoted 17 lines
Hi Sagi,
Quoted 11 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) · line 1810
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-6- ret =nvme_tcp_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { -nvme_start_queues (ctrl); + }else if (prior_q_cnt != ctrl->queue_count) {So if the queue count did not change we don't wait to make sure the queue
g_usage_counterref made it to zero? What guarantees that it did?Hmm, good point. we should always call
nvme_wait_freeze_timeout()for !new queues. Is this what you are implying?
Quoted 6 lines
Quoted 1 line
I think we should always wait for the freeze to complete.
Don't the queues need to be started in order for the freeze to complete? Any enqueued requests on the quiesced queues will never complete this way, so the
wait_freeze()will be stuck, right? If so, I think thenvme_start_queues()was in the correct place already.
Exactly what I was trying to point out (poorly though)
Re: [PATCH v4 2/8] nvme-tcp: Update number of hardware queues before using them
Quoted 8 lines · Sagi Grimberg
Quoted 6 lines
Quoted 1 line
I think we should always wait for the freeze to complete.
Don't the queues need to be started in order for the freeze to complete? Any enqueued requests on the quiesced queues will never complete this way, so the
wait_freeze()will be stuck, right? If so, I think thenvme_start_queues()was in the correct place already.Exactly what I was trying to point out (poorly though)
Thanks for explaining. I think I got the general idea what the different states are doing and what the transitions are now. (famous last words).
Anyway, the first three patches are the result of debugging the case of 'prior_ioq_cnt != nr_io_queues'. Starting the queues before updating the number of queues lookes strange.
I suppose in the case 'prior_ioq_cnt > nr_io_queues', nvme_tcp_start_io_queues() should be successful and we do the blk_mq_update_nr_hw_queues(). In the other case we should land in the error recovery.
Wouldn't it make sense to always exercise the error recovery path if we detect 'prior_ioq_cnt != nr_io_queues' and reduce the number of things which can go wrong?
Daniel
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v4 3/8 nvme-rdma: Update number of hardware queues before using them
In reply to Daniel WagnerWhen the number of hardware queues changes during resetting we should update the tagset first before using it.
--- drivers/nvme/host/rdma.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/nvme/host/rdma.c +6 -7
@@ -965,6 +965,7 @@ static void nvme_rdma_destroy_io_queues(struct nvme_rdma_ctrl *ctrl,static int nvme_rdma_configure_io_queues (struct nvme_rdma_ctrl *ctrl,bool new) {int ret; +u32 prior_q_cnt = ctrl->ctrl.queue_count; ret =nvme_rdma_alloc_io_queues (ctrl);if (ret)
@@ -982,13 +983,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) ret =PTR_ERR (ctrl->ctrl.connect_q); goto out_free_tag_set; } - } - - ret =nvme_rdma_start_io_queues (ctrl); -if (ret) - goto out_cleanup_connect_q; - -if (!new) { + }else if (prior_q_cnt != ctrl->ctrl.queue_count) {nvme_start_queues (&ctrl->ctrl);if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) {/*
@@ -1004,6 +999,10 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new)nvme_unfreeze (&ctrl->ctrl); } + ret =nvme_rdma_start_io_queues (ctrl); +if (ret) + goto out_cleanup_connect_q; +return 0 ; out_wait_freeze_timed_out:
Signature · 1 line
2.29.2
PATCH v4 4/8 nvme-fc: Wait with a timeout for queue to freeze
In reply to Daniel WagnerDo not wait indifinitly for all queues to freeze. Instead use a timeout and abort the operation if we get stuck.
--- drivers/nvme/host/fc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +8 -1
@@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)dev_info (ctrl->ctrl.device,"reconnect: revising io queue count from %d to %d\n" , prior_ioq_cnt, nr_io_queues); -nvme_wait_freeze (&ctrl->ctrl); +if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) { +/* +* If we timed out waiting for freeze we are likely to +* be stuck. Fail the controller initialization just +* to be safe. +*/ +return -ENODEV ; + }blk_mq_update_nr_hw_queues (&ctrl->tag_set, nr_io_queues);nvme_unfreeze (&ctrl->ctrl); }
Signature · 1 line
2.29.2
Re: [PATCH v4 4/8] nvme-fc: Wait with a timeout for queue to freeze
Quoted 21 lines · Daniel Wagner
Do not wait indifinitly for all queues to freeze. Instead use a timeout and abort the operation if we get stuck.
Reviewed-by James Smart <[email protected]>Reviewed-by Hannes Reinecke <[email protected]>Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/nvme/host/fc.c +1
-1@@ -2955,7 +2955,14 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)dev_info (ctrl->ctrl.device,"reconnect: revising io queue count from %d to %d\n" , prior_ioq_cnt, nr_io_queues); -nvme_wait_freeze (&ctrl->ctrl); +if (!nvme_wait_freeze_timeout (&ctrl->ctrl,NVME_IO_TIMEOUT )) {
drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2968
Quoted 10 lines
drivers/nvme/host/fc.c +7
-0+/* +* If we timed out waiting for freeze we are likely to +* be stuck. Fail the controller initialization just +* to be safe. +*/ +return -ENODEV ; + }blk_mq_update_nr_hw_queues (&ctrl->tag_set, nr_io_queues);nvme_unfreeze (&ctrl->ctrl); }
Signature · 6 lines
Himanshu Madhani Oracle Linux Engineering _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v4 5/8 nvme-fc: avoid race between time out and tear down
In reply to Daniel WagnerFrom: James Smart <[email protected]>
To avoid race between time out and tear down, in tear down process, first we quiesce the queue, and then delete the timer and cancel the time out work for the queue.
This patch merges the admin and io sync ops into the queue teardown logic as shown in the RDMA patch 3017013dcc "nvme-rdma: avoid race between time out and tear down". There is no teardown_lock in nvme-fc.
--- drivers/nvme/host/fc.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/host/fc.c +2 -0
@@ -2487,6 +2487,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)*/ if (ctrl->ctrl.queue_count >1 ) {nvme_stop_queues (&ctrl->ctrl); +nvme_sync_io_queues (&ctrl->ctrl);blk_mq_tagset_busy_iter (&ctrl->tag_set, nvme_fc_terminate_exchange, &ctrl->ctrl);blk_mq_tagset_wait_completed_request (&ctrl->tag_set);
@@ -2510,6 +2511,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)* clean up the admin queue. Same thing as above. */ blk_mq_quiesce_queue (ctrl->ctrl.admin_q); +blk_sync_queue (ctrl->ctrl.admin_q);blk_mq_tagset_busy_iter (&ctrl->admin_tag_set, nvme_fc_terminate_exchange, &ctrl->ctrl);blk_mq_tagset_wait_completed_request (&ctrl->admin_tag_set);
Signature · 1 line
2.29.2
Re: [PATCH v4 5/8] nvme-fc: avoid race between time out and tear down
Quoted 34 lines · Daniel Wagner
From: James Smart <[email protected]>
To avoid race between time out and tear down, in tear down process, first we quiesce the queue, and then delete the timer and cancel the time out work for the queue.
This patch merges the admin and io sync ops into the queue teardown logic as shown in the RDMA patch
3017013dcc"nvme-rdma: avoid race between time out and tear down". There is noteardown_lockin nvme-fc.Signed-off-by James Smart <[email protected]>CC Chao Leng <[email protected]>Tested-by Daniel Wagner <[email protected]>[dwagner: updated commit id referenced in commit message]Reviewed-by Daniel Wagner <[email protected]>Reviewed-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/fc.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/host/fc.c +1
-0@@ -2487,6 +2487,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)*/ if (ctrl->ctrl.queue_count >1 ) {nvme_stop_queues (&ctrl->ctrl); +nvme_sync_io_queues (&ctrl->ctrl);blk_mq_tagset_busy_iter (&ctrl->tag_set, nvme_fc_terminate_exchange, &ctrl->ctrl);blk_mq_tagset_wait_completed_request (&ctrl->tag_set);
drivers/nvme/host/fc.c __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues) · line 2517
Quoted 7 lines
drivers/nvme/host/fc.c +1
-0@@ -2510,6 +2511,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)* clean up the admin queue. Same thing as above. */ blk_mq_quiesce_queue (ctrl->ctrl.admin_q); +blk_sync_queue (ctrl->ctrl.admin_q);blk_mq_tagset_busy_iter (&ctrl->admin_tag_set, nvme_fc_terminate_exchange, &ctrl->ctrl);blk_mq_tagset_wait_completed_request (&ctrl->admin_tag_set);
Signature · 1 line
Himanshu Madhani Oracle Linux Engineering
PATCH v4 6/8 nvme-fc: fix controller reset hang during traffic
In reply to Daniel WagnerFrom: James Smart <[email protected]>
commit fe35ec58f0d3 ("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We call blk_mq_update_nr_hw_queues which may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.
So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code). blk_mq_update_nr_hw_queues will be called only after we are sure that the queue was already frozen.
This follows to how the pci driver handles resets.
This patch added logic introduced in commit 9f98772ba307 "nvme-rdma: fix controller reset hang during traffic".
[dwagner: call nvme_unfreeze() unconditionally in
nvme_fc_recreate_io_queues() to match the nvme_start_freeze()]
--- drivers/nvme/host/fc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/fc.c +2 -1
@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)* (but with error status). */ if (ctrl->ctrl.queue_count >1 ) { +nvme_start_freeze (&ctrl->ctrl);nvme_stop_queues (&ctrl->ctrl);nvme_sync_io_queues (&ctrl->ctrl); blk_mq_tagset_busy_iter(&ctrl->tag_set,
@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)return -ENODEV ; }blk_mq_update_nr_hw_queues (&ctrl->tag_set, nr_io_queues); -nvme_unfreeze (&ctrl->ctrl); } +nvme_unfreeze (&ctrl->ctrl); ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 );if (ret)
Signature · 1 line
2.29.2
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Quoted 43 lines · Daniel Wagner
From: James Smart <[email protected]>
commit
fe35ec58f0d3("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We callblk_mq_update_nr_hw_queueswhich may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code).
blk_mq_update_nr_hw_queueswill be called only after we are sure that the queue was already frozen.This follows to how the pci driver handles resets.
This patch added logic introduced in commit
9f98772ba307"nvme-rdma: fix controller reset hang during traffic".Signed-off-by James Smart <[email protected]>CC Sagi Grimberg <[email protected]>[dwagner: call
nvme_unfreeze()unconditionally in
nvme_fc_recreate_io_queues()to match thenvme_start_freeze()]Tested-by Daniel Wagner <[email protected]>Reviewed-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/fc.c +1
-0@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)* (but with error status). */ if (ctrl->ctrl.queue_count >1 ) { +nvme_start_freeze (&ctrl->ctrl);nvme_stop_queues (&ctrl->ctrl);nvme_sync_io_queues (&ctrl->ctrl); blk_mq_tagset_busy_iter(&ctrl->tag_set,
drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2974
Quoted 9 lines
drivers/nvme/host/fc.c +1
-1@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)return -ENODEV ; }blk_mq_update_nr_hw_queues (&ctrl->tag_set, nr_io_queues); -nvme_unfreeze (&ctrl->ctrl); } +nvme_unfreeze (&ctrl->ctrl); ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 );if (ret)
Signature · 6 lines
Himanshu Madhani Oracle Linux Engineering _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
In reply to Daniel WagnerQuoted 43 lines · Daniel Wagner
From: James Smart <[email protected]>
commit
fe35ec58f0d3("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We callblk_mq_update_nr_hw_queueswhich may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code).
blk_mq_update_nr_hw_queueswill be called only after we are sure that the queue was already frozen.This follows to how the pci driver handles resets.
This patch added logic introduced in commit
9f98772ba307"nvme-rdma: fix controller reset hang during traffic".Signed-off-by James Smart <[email protected]>CC Sagi Grimberg <[email protected]>[dwagner: call
nvme_unfreeze()unconditionally in
nvme_fc_recreate_io_queues()to match thenvme_start_freeze()]Tested-by Daniel Wagner <[email protected]>Reviewed-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/fc.c +1
-0@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)* (but with error status). */ if (ctrl->ctrl.queue_count >1 ) { +nvme_start_freeze (&ctrl->ctrl);nvme_stop_queues (&ctrl->ctrl);nvme_sync_io_queues (&ctrl->ctrl); blk_mq_tagset_busy_iter(&ctrl->tag_set,
drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2974
Quoted 9 lines
drivers/nvme/host/fc.c +1
-1@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)return -ENODEV ; }blk_mq_update_nr_hw_queues (&ctrl->tag_set, nr_io_queues); -nvme_unfreeze (&ctrl->ctrl); } +nvme_unfreeze (&ctrl->ctrl); ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 );if (ret)
There still is now an imbalance, as we're always calling 'nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.
This might lead to an imbalance on the mq_freeze_depth counter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of the if() condition to avoid the imbalance?
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Quoted 7 lines · Hannes Reinecke
There still is now an imbalance, as we're always calling '
nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.This might lead to an imbalance on the
mq_freeze_depthcounter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of theif()condition to avoid the imbalance?
What about something like nmve_is_frozen() which would avoid the tracking of the queue state open coded as it is right?
Sagi Grimberg
maintainerRe: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Quoted 6 lines · Hannes Reinecke
Quoted 4 lines
There still is now an imbalance, as we're always calling '
nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.
Quoted 6 lines
Quoted 3 lines
This might lead to an imbalance on the
mq_freeze_depthcounter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of theif()condition to avoid the imbalance?What about something like
nmve_is_frozen()which would avoid the tracking of the queue state open coded as it is right?
Why is there a conditional freeze?
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Quoted 1 line · Sagi Grimberg
Why is there a conditional freeze?
The freeze only happens if I/O queues have been created/started. If I understand this correctly, this is the corner case we were only able to establish the admin queue and fail later to setup the I/O queues.
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
In reply to Hannes ReineckeQuoted 57 lines · Hannes Reinecke
Quoted 43 lines · Daniel Wagner
From: James Smart <[email protected]>
commit
fe35ec58f0d3("block: update hctx map when use multiple maps") exposed an issue where we may hang trying to wait for queue freeze during I/O. We callblk_mq_update_nr_hw_queueswhich may attempt to freeze the queue. However we never started queue freeze when starting the reset, which means that we have inflight pending requests that entered the queue that we will not complete once the queue is quiesced.So start a freeze before we quiesce the queue, and unfreeze the queue after we successfully connected the I/O queues (the unfreeze is already present in the code).
blk_mq_update_nr_hw_queueswill be called only after we are sure that the queue was already frozen.This follows to how the pci driver handles resets.
This patch added logic introduced in commit
9f98772ba307"nvme-rdma: fix controller reset hang during traffic".Signed-off-by James Smart <[email protected]>CC Sagi Grimberg <[email protected]>[dwagner: call
nvme_unfreeze()unconditionally in
nvme_fc_recreate_io_queues()to match thenvme_start_freeze()]Tested-by Daniel Wagner <[email protected]>Reviewed-by Daniel Wagner <[email protected]>--- drivers/nvme/host/fc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/fc.c +1
-0@@ -2486,6 +2486,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)* (but with error status). */ if (ctrl->ctrl.queue_count >1 ) { +nvme_start_freeze (&ctrl->ctrl);nvme_stop_queues (&ctrl->ctrl);nvme_sync_io_queues (&ctrl->ctrl); blk_mq_tagset_busy_iter(&ctrl->tag_set,drivers/nvme/host/fc.c nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl) · line 2974
Quoted 9 lines
drivers/nvme/host/fc.c +1
-1@@ -2966,8 +2967,8 @@ nvme_fc_recreate_io_queues(struct nvme_fc_ctrl *ctrl)return -ENODEV ; }blk_mq_update_nr_hw_queues (&ctrl->tag_set, nr_io_queues); -nvme_unfreeze (&ctrl->ctrl); } +nvme_unfreeze (&ctrl->ctrl); ret =nvme_fc_create_hw_io_queues (ctrl, ctrl->ctrl.sqsize +1 );if (ret)There still is now an imbalance, as we're always calling '
nvme_unfreeze()' (irrespective on the number of queues), but will only call 'nvme_start_freeze()' if we have more than one queue.
Quoted 5 lines
This might lead to an imbalance on the
mq_freeze_depthcounter. Wouldn't it be better to move the call to 'nvme_start_freeze()' out of theif()condition to avoid the imbalance?Cheers,
Daniel,
try this. Moves the location of the freeze and will always pair with the unfreeze.
fc.c.new renamed from fc.c +5 -1
@@ -2965,9 +2965,10 @@ nvme_fc_recreate_io_queues(struct nvme_f prior_ioq_cnt, nr_io_queues); nvme_wait_freeze(&ctrl->ctrl); blk_mq_update_nr_hw_queues(&ctrl->tag_set, nr_io_queues); - nvme_unfreeze(&ctrl->ctrl); } + nvme_unfreeze(&ctrl->ctrl); + return 0; out_delete_hw_queues:
@@ -3206,6 +3207,9 @@ nvme_fc_delete_association(struct nvme_f ctrl->iocnt = 0; spin_unlock_irqrestore(&ctrl->lock, flags); + if (ctrl->ctrl.queue_count > 1) + nvme_start_freeze(&ctrl->ctrl); + __nvme_fc_abort_outstanding_ios(ctrl, false); /* kill the aens as they are a separate path */
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Quoted 2 lines · James Smart
try this. Moves the location of the freeze and will always pair with the unfreeze.
Yes, this works. Do you send a proper patch out or should I just pick up this patch and add a commit message?
Re: [PATCH v4 6/8] nvme-fc: fix controller reset hang during traffic
Quoted 6 lines · Daniel Wagner
Quoted 2 lines · James Smart
try this. Moves the location of the freeze and will always pair with the unfreeze.
Yes, this works. Do you send a proper patch out or should I just pick up this patch and add a commit message?
Forget it, the commit message is still the same. Let me respin this series.
PATCH v4 7/8 nvme-tcp: Unfreeze queues on reconnect
In reply to Daniel WagnerDuring the queue teardown in nvme_tcp_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.
--- drivers/nvme/host/tcp.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +3 -1
@@ -1819,9 +1819,11 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) }blk_mq_update_nr_hw_queues (ctrl->tagset, ctrl->queue_count -1 ); -nvme_unfreeze (ctrl); } +if (!new) +nvme_unfreeze (ctrl); + ret =nvme_tcp_start_io_queues (ctrl);if (ret) goto out_cleanup_connect_q;
Signature · 1 line
2.29.2
PATCH v4 8/8 nvme-rdma: Unfreeze queues on reconnect
In reply to Daniel WagnerDuring the queue teardown in nvme_rdma_teardown_io_queues() freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.
--- drivers/nvme/host/rdma.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/host/rdma.c +2 -0
@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, error =PTR_ERR (ctrl->ctrl.admin_q); goto out_cleanup_fabrics_q; } + }else { +nvme_unfreeze (&ctrl->ctrl); } error =nvme_rdma_start_queue (ctrl,0 );
Signature · 1 line
2.29.2
Re: [PATCH v4 8/8] nvme-rdma: Unfreeze queues on reconnect
Quoted 17 lines · Daniel Wagner
During the queue teardown in
nvme_rdma_teardown_io_queues()freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/rdma.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/host/rdma.c +0
-0
drivers/nvme/host/rdma.c static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · line 908
Quoted 8 lines
drivers/nvme/host/rdma.c +2
-0@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, error =PTR_ERR (ctrl->ctrl.admin_q); goto out_cleanup_fabrics_q; } + }else { +nvme_unfreeze (&ctrl->ctrl); } error =nvme_rdma_start_queue (ctrl,0 );
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), GF: Felix Imendörffer
Sagi Grimberg
maintainerRe: [PATCH v4 8/8] nvme-rdma: Unfreeze queues on reconnect
In reply to Daniel WagnerQuoted 17 lines
During the queue teardown in
nvme_rdma_teardown_io_queues()freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/rdma.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/host/rdma.c +0
-0
drivers/nvme/host/rdma.c static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · line 905
Quoted 5 lines
drivers/nvme/host/rdma.c +2
-0@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, error =PTR_ERR (ctrl->ctrl.admin_q); goto out_cleanup_fabrics_q; } + }else { +nvme_unfreeze (&ctrl->ctrl);
That seems misplaced.. unfreezing the I/O queues when setting up the admin queue?
Re: [PATCH v4 8/8] nvme-rdma: Unfreeze queues on reconnect
Hi Sagi,
Quoted 12 lines · Sagi Grimberg
Quoted 12 lines
During the queue teardown in
nvme_rdma_teardown_io_queues()freeze is called unconditionally. When we reconnect we need to pair the freeze with an unfreeze to avoid hanging I/Os. For newly created connection this is not needed.Fixes 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic")Signed-off-by Daniel Wagner <[email protected]>--- drivers/nvme/host/rdma.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 13 lines
drivers/nvme/host/rdma.c static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, · line 905
Quoted 10 lines
drivers/nvme/host/rdma.c +2
-0@@ -901,6 +901,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, error =PTR_ERR (ctrl->ctrl.admin_q); goto out_cleanup_fabrics_q; } + }else { +nvme_unfreeze (&ctrl->ctrl);That seems misplaced.. unfreezing the I/O queues when setting up the admin queue?
Indeed. After looking again on it, this should be almost identically to the tcp.c fix in nvme_rdma_configure_io_queues.
BTW, I am working on getting a RDMA test setup running. Hopefully I have all the right licenses on the array.
Daniel