Host reconnecting need more than 60s to start after nvmetcli clear on target

DISCUSSION Yi Zhang 3 participants 7 messages Sep 8, 2017 – Sep 21, 2017 1 message not in the archive NVMe

Share & tools

A message that is not in the archive (sent off-list or not yet received): [email protected]

Christoph Hellwig

2017-09-18 15:58 UTC #2 lore raw

Quoted 6 lines · Yi Zhang

Hi
I found this issue on latest 4.13, is it for designed? I cannot reproduce it on 4.12.
Here is the log from host:
4.13

Quoted 8 lines

[  637.246798] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  637.436315] nvme nvme0: creating 40 I/O queues.
[  637.939988] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  645.319803] nvme nvme0: rescanning

-->need more than 60 seconds to start reconnect

[  706.073551] nvme nvme0: Reconnecting in 10 seconds...

How did you initiate the reconnect? Cable drop?

Yi Zhang

author

2017-09-19 09:09 UTC #3 lore raw

Quoted 12 lines · Christoph Hellwig

Quoted 11 lines · Yi Zhang
Hi
I found this issue on latest 4.13, is it for designed? I cannot reproduce it on 4.12.
Here is the log from host:
4.13
[  637.246798] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  637.436315] nvme nvme0: creating 40 I/O queues.
[  637.939988] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  645.319803] nvme nvme0: rescanning

Quoted 4 lines

Quoted 3 lines
-->need more than 60 seconds to start reconnect
[  706.073551] nvme nvme0: Reconnecting in 10 seconds...
How did you initiate the reconnect? Cable drop?

Just execute "nvmetcli clear" on target side, and check the log on host side.

Quoted 4 lines

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Christoph Hellwig

2017-09-19 14:18 UTC #4 lore raw

Quoted 15 lines · Yi Zhang

Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Yi Zhang
Hi
I found this issue on latest 4.13, is it for designed? I cannot reproduce it on 4.12.
Here is the log from host:
4.13
[  637.246798] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  637.436315] nvme nvme0: creating 40 I/O queues.
[  637.939988] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  645.319803] nvme nvme0: rescanning

Quoted 7 lines

Quoted 4 lines
Quoted 3 lines
-->need more than 60 seconds to start reconnect
[  706.073551] nvme nvme0: Reconnecting in 10 seconds...
How did you initiate the reconnect? Cable drop?
Just execute "nvmetcli clear" on target side, and check the log on host side.

Ok. 60 seconds is when the first commands will time out, so that's
expected. The NVMeoF protocol has no way to notify the host that
a connection went away, so if you ren't on a protocol that supports link up/down notifications we'll have to wait for timeouts.

Yi Zhang

author

2017-09-20 00:50 UTC #5 lore raw

Quoted 17 lines · Christoph Hellwig

Quoted 16 lines · Yi Zhang
Quoted 14 lines · Christoph Hellwig
Quoted 13 lines · Yi Zhang
Hi
I found this issue on latest 4.13, is it for designed? I cannot reproduce it on 4.12.
Here is the log from host:
4.13
[  637.246798] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  637.436315] nvme nvme0: creating 40 I/O queues.
[  637.939988] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  645.319803] nvme nvme0: rescanning
-->need more than 60 seconds to start reconnect

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
Quoted 1 line
[  706.073551] nvme nvme0: Reconnecting in 10 seconds...
How did you initiate the reconnect? Cable drop?
Just execute "nvmetcli clear" on target side, and check the log on host side.
Ok. 60 seconds is when the first commands will time out, so that's
expected. The NVMeoF protocol has no way to notify the host that
a connection went away, so if you ren't on a protocol that supports link up/down notifications we'll have to wait for timeouts.

Got, thanks Christoph.

Quoted 4 lines

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Sagi Grimberg

maintainer

2017-09-20 11:26 UTC #6 lore raw

In reply to Christoph Hellwig

Quoted 17 lines

Quoted 17 lines
Quoted 16 lines
Quoted 7 lines
Hi
I found this issue on latest 4.13, is it for designed? I cannot reproduce it on 4.12.
Here is the log from host:
4.13
Quoted 8 lines
[  637.246798] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  637.436315] nvme nvme0: creating 40 I/O queues.
[  637.939988] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  645.319803] nvme nvme0: rescanning
-->need more than 60 seconds to start reconnect
[  706.073551] nvme nvme0: Reconnecting in 10 seconds...
How did you initiate the reconnect? Cable drop?

Quoted 7 lines

Quoted 2 lines
Just execute "nvmetcli clear" on target side, and check the log on host side.
Ok. 60 seconds is when the first commands will time out, so that's
expected. The NVMeoF protocol has no way to notify the host that
a connection went away, so if you ren't on a protocol that supports link up/down notifications we'll have to wait for timeouts.

That's not entirely true.

Yes there is no clear indication, but the keep-alive should expire faster than 60 seconds (it actually 5 seconds by default). The point here is that its not really a cable pull, its removal of the subsystem and the namespaces just before that trigger a rescan.

In rdma error recovery we first of all call nvme_stop_ctrl() which flush the scan_work and that waits for the identify to exhaust (60 seconds admin timeout). But in error recovery, we should really call full stop_ctrl, we just need to stop the keep-alive so it will get out of the way...

Does this fix your issue?
--

drivers/nvme/host/rdma.c +0 -0

index 4460ec3a2c0f..2d2afb5e8102 100644
@@ -966,7 +966,7 @@ static void nvme_rdma_error_recovery_work(struct

work_struct *work)
struct nvme_rdma_ctrl *ctrl = container_of(work,
struct nvme_rdma_ctrl, err_work);

- nvme_stop_ctrl(&ctrl->ctrl);
+ nvme_stop_keep_alive(ctrl);

         if (ctrl->ctrl.queue_count > 1) {
                 nvme_stop_queues(&ctrl->ctrl);

This was the original code, I replaced it (incorrectly I think) when introducing nvme_stop_ctrl.

Yi Zhang

author

2017-09-21 03:25 UTC #7 lore raw

Quoted 35 lines · Sagi Grimberg

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 15 lines
Hi
I found this issue on latest 4.13, is it for designed? I cannot reproduce it on 4.12.
Here is the log from host:
4.13
[  637.246798] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  637.436315] nvme nvme0: creating 40 I/O queues.
[  637.939988] nvme nvme0: new ctrl: NQN "testnqn", addr 
172.31.0.90:4420
[  645.319803] nvme nvme0: rescanning
Quoted 3 lines
-->need more than 60 seconds to start reconnect
[  706.073551] nvme nvme0: Reconnecting in 10 seconds...
How did you initiate the reconnect? Cable drop?
Quoted 7 lines
Quoted 2 lines
Just execute "nvmetcli clear" on target side, and check the log on host side.
Ok. 60 seconds is when the first commands will time out, so that's
expected. The NVMeoF protocol has no way to notify the host that
a connection went away, so if you ren't on a protocol that supports link up/down notifications we'll have to wait for timeouts.
That's not entirely true.
Yes there is no clear indication, but the keep-alive should expire faster than 60 seconds (it actually 5 seconds by default). The point here is that its not really a cable pull, its removal of the subsystem and the namespaces just before that trigger a rescan.

Quoted 7 lines

In rdma error recovery we first of all call nvme_stop_ctrl() which flush the scan_work and that waits for the identify to exhaust (60 seconds admin timeout). But in error recovery, we should really call full stop_ctrl, we just need to stop the keep-alive so it will get out of the way...
Does this fix your issue?

Hi Sagi

Your patch works, actually we should use
nvme_stop_keep_alive(&ctrl->ctrl) instead nvme_stop_keep_alive(ctrl). :)

Here is the log:

[  599.979081] nvme nvme0: new ctrl: NQN 
"nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.90:4420
[  600.116311] nvme nvme0: creating 40 I/O queues.
[  600.630916] nvme nvme0: new ctrl: NQN "testnqn", addr 172.31.0.90:4420
[  606.107455] nvme nvme0: rescanning
[  606.265619] nvme nvme0: Reconnecting in 10 seconds...
[  616.367326] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  616.374831] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  616.381262] nvme nvme0: Failed reconnect attempt 1
[  616.386626] nvme nvme0: Reconnecting in 10 seconds...
[  626.595572] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  626.603073] nvme nvme0: rdma_resolve_addr wait failed (-104).
[  626.609507] nvme nvme0: Failed reconnect attempt 2
[  626.614899] nvme nvme0: Reconnecting in 10 seconds...
[  636.835354] nvme nvme0: Connect rejected: status 8 (invalid service ID).
[  636.842856] nvme nvme0: rdma_resolve_addr wait failed (-104).

Thanks
Yi

Quoted 24 lines

drivers/nvme/host/rdma.c +1 -1

index 4460ec3a2c0f..2d2afb5e8102 100644
@@ -966,7 +966,7 @@ static void nvme_rdma_error_recovery_work(struct 
 work_struct *work)
        struct nvme_rdma_ctrl *ctrl = container_of(work,
                        struct nvme_rdma_ctrl, err_work);
 
-       nvme_stop_ctrl(&ctrl->ctrl);
+       nvme_stop_keep_alive(ctrl);
 
        if (ctrl->ctrl.queue_count > 1) {

                nvme_stop_queues(&ctrl->ctrl);

Signature · 7 lines

This was the original code, I replaced it (incorrectly I think) when
introducing nvme_stop_ctrl.

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme