nvme-tcp: Prevent infinite loop if socket closes during CONNECTING state
PATCH 1/1 nvme-tcp: Prevent infinite loop if socket closes during CONNECTING state
There is a potential race condition that can occur if
the target closes the socket while the host is in the CONNECTING state.
If the socket's state changes to TCP_CLOSE, the nvme_tcp_state_change() function is invoked. However, nvme_tcp_error_recovery() is unable to transition the controller state to NVME_CTRL_RESETTING because the controller is still in the CONNECTING state. As a result, error recovery is bypassed, and the controller incorrectly transitions to the LIVE state with closed sockets.
Subsequent attempts by the host to communicate with the target will result in an infinite loop.
Fix the bug by checking for the -EPIPE error code, which indicates that the connection was closed by the target. If this condition is detected, ensure the error recovery process is initiated to correctly handle the disconnection.
--- drivers/nvme/host/tcp.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +2 -0
@@ -1261,6 +1261,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) "failed to send request %d\n", ret);nvme_tcp_fail_request (queue->request);nvme_tcp_done_send_req (queue); +if (ret == -EPIPE ) +nvme_tcp_error_recovery (&queue->ctrl->ctrl); } out:memalloc_noreclaim_restore (noreclaim_flag);
Signature · 1 line
2.43.5
Quoted 28 lines · Maurizio Lombardi
There is a potential race condition that can occur if
the target closes the socket while the host is in the CONNECTING state.If the socket's state changes to
TCP_CLOSE, thenvme_tcp_state_change()function is invoked. However,nvme_tcp_error_recovery()is unable to transition the controller state toNVME_CTRL_RESETTINGbecause the controller is still in the CONNECTING state. As a result, error recovery is bypassed, and the controller incorrectly transitions to the LIVE state with closed sockets.Subsequent attempts by the host to communicate with the target will result in an infinite loop.
Fix the bug by checking for the
-EPIPEerror code, which indicates that the connection was closed by the target. If this condition is detected, ensure the error recovery process is initiated to correctly handle the disconnection.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/tcp.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) · line 1268
Quoted 8 lines
drivers/nvme/host/tcp.c +2
-0@@ -1261,6 +1261,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) "failed to send request %d\n", ret);nvme_tcp_fail_request (queue->request);nvme_tcp_done_send_req (queue); +if (ret == -EPIPE ) +nvme_tcp_error_recovery (&queue->ctrl->ctrl); } out:memalloc_noreclaim_restore (noreclaim_flag);
This feels a bit weird; wouldn't it be better to allow transitions from 'CONNECTING' to 'RESETTING'? Wouldn't the fix the issue, too? But yeah, this is a real issue; I've seen it in my tests occasionally, too.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
út 21. 1. 2025 v 15:11 odesílatel Hannes Reinecke <[email protected]> napsal:
Quoted 5 lines
This feels a bit weird; wouldn't it be better to allow transitions from 'CONNECTING' to 'RESETTING'? Wouldn't the fix the issue, too? But yeah, this is a real issue; I've seen it in my tests occasionally, too.
Yes, it would fix the bug.
I didn't do that because it could potentially have an impact on all the drivers and I wanted to avoid regressions.
Maurizio
Sagi Grimberg
maintainerQuoted 28 lines · Maurizio Lombardi
There is a potential race condition that can occur if
the target closes the socket while the host is in the CONNECTING state.If the socket's state changes to
TCP_CLOSE, thenvme_tcp_state_change()function is invoked. However,nvme_tcp_error_recovery()is unable to transition the controller state toNVME_CTRL_RESETTINGbecause the controller is still in the CONNECTING state. As a result, error recovery is bypassed, and the controller incorrectly transitions to the LIVE state with closed sockets.Subsequent attempts by the host to communicate with the target will result in an infinite loop.
Fix the bug by checking for the
-EPIPEerror code, which indicates that the connection was closed by the target. If this condition is detected, ensure the error recovery process is initiated to correctly handle the disconnection.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/nvme/host/tcp.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) · line 1265
Quoted 5 lines
drivers/nvme/host/tcp.c +2
-0@@ -1261,6 +1261,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) "failed to send request %d\n", ret);nvme_tcp_fail_request (queue->request);nvme_tcp_done_send_req (queue); +if (ret == -EPIPE ) +nvme_tcp_error_recovery (&queue->ctrl->ctrl);
Won't nvme_tcp_error_recover() fail to transition the state and exit before doing anything?
čt 23. 1. 2025 v 23:33 odesílatel Sagi Grimberg <[email protected]> napsal:
Quoted 4 lines
Won't
nvme_tcp_error_recover()fail to transition the state and exit before doing anything?
No, because it's called in nvme_tcp_try_send() when the call to sock_sendmsg() fails,
so the controller has already transitioned to LIVE state.
I will modify the commit message to make it clearer.
Maurizio