Skip to contentSkip to conversation

nvme-tcp: Prevent infinite loop if socket closes during CONNECTING state

PATCH Superseded Maurizio Lombardi 3 participants 5 messages Jan 21, 2025 – Jan 24, 2025 NVMe

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-tcp: Prevent infinite loop if socket closes during CONNECTING state

There is a potential race condition that can occur if
the target closes the socket while the host is in the CONNECTING state.

If the socket's state changes to TCP_CLOSE, the nvme_tcp_state_change() function is invoked. However, nvme_tcp_error_recovery() is unable to transition the controller state to NVME_CTRL_RESETTING because the controller is still in the CONNECTING state. As a result, error recovery is bypassed, and the controller incorrectly transitions to the LIVE state with closed sockets.

Subsequent attempts by the host to communicate with the target will result in an infinite loop.

Fix the bug by checking for the -EPIPE error code, which indicates that the connection was closed by the target. If this condition is detected, ensure the error recovery process is initiated to correctly handle the disconnection.

---
 drivers/nvme/host/tcp.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +2 -0
index 841238f38fdd..d0bb225dca23 100644
@@ -1261,6 +1261,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
 			"failed to send request %d\n", ret);
 		nvme_tcp_fail_request(queue->request);
 		nvme_tcp_done_send_req(queue);
+		if (ret == -EPIPE)
+			nvme_tcp_error_recovery(&queue->ctrl->ctrl);
 	}
 out:
 	memalloc_noreclaim_restore(noreclaim_flag);
Signature · 1 line
2.43.5
#1 lore raw
Quoted 28 lines · Maurizio Lombardi

There is a potential race condition that can occur if
the target closes the socket while the host is in the CONNECTING state.

If the socket's state changes to TCP_CLOSE, the nvme_tcp_state_change() function is invoked. However, nvme_tcp_error_recovery() is unable to transition the controller state to NVME_CTRL_RESETTING because the controller is still in the CONNECTING state. As a result, error recovery is bypassed, and the controller incorrectly transitions to the LIVE state with closed sockets.

Subsequent attempts by the host to communicate with the target will result in an infinite loop.

Fix the bug by checking for the -EPIPE error code, which indicates that the connection was closed by the target. If this condition is detected, ensure the error recovery process is initiated to correctly handle the disconnection.

---
  drivers/nvme/host/tcp.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +0 -0
index 841238f38fdd..d0bb225dca23 100644

drivers/nvme/host/tcp.c static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) · line 1268

Quoted 8 lines
drivers/nvme/host/tcp.c +2 -0
@@ -1261,6 +1261,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
  			"failed to send request %d\n", ret);
  		nvme_tcp_fail_request(queue->request);
  		nvme_tcp_done_send_req(queue);
+		if (ret == -EPIPE)
+			nvme_tcp_error_recovery(&queue->ctrl->ctrl);
  	}
  out:
  	memalloc_noreclaim_restore(noreclaim_flag);

This feels a bit weird; wouldn't it be better to allow transitions from 'CONNECTING' to 'RESETTING'? Wouldn't the fix the issue, too? But yeah, this is a real issue; I've seen it in my tests occasionally, too.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#2 lore raw

út 21. 1. 2025 v 15:11 odesílatel Hannes Reinecke <[email protected]> napsal:

Quoted 5 lines

This feels a bit weird; wouldn't it be better to allow transitions from 'CONNECTING' to 'RESETTING'? Wouldn't the fix the issue, too? But yeah, this is a real issue; I've seen it in my tests occasionally, too.

Yes, it would fix the bug.
I didn't do that because it could potentially have an impact on all the drivers and I wanted to avoid regressions.

Maurizio

maintainer
#3 lore raw
In reply to Maurizio Lombardi
Quoted 28 lines · Maurizio Lombardi

There is a potential race condition that can occur if
the target closes the socket while the host is in the CONNECTING state.

If the socket's state changes to TCP_CLOSE, the nvme_tcp_state_change() function is invoked. However, nvme_tcp_error_recovery() is unable to transition the controller state to NVME_CTRL_RESETTING because the controller is still in the CONNECTING state. As a result, error recovery is bypassed, and the controller incorrectly transitions to the LIVE state with closed sockets.

Subsequent attempts by the host to communicate with the target will result in an infinite loop.

Fix the bug by checking for the -EPIPE error code, which indicates that the connection was closed by the target. If this condition is detected, ensure the error recovery process is initiated to correctly handle the disconnection.

---
  drivers/nvme/host/tcp.c | 2 ++
  1 file changed, 2 insertions(+)
drivers/nvme/host/tcp.c +0 -0
index 841238f38fdd..d0bb225dca23 100644

drivers/nvme/host/tcp.c static int nvme_tcp_try_send(struct nvme_tcp_queue *queue) · line 1265

Quoted 5 lines
drivers/nvme/host/tcp.c +2 -0
@@ -1261,6 +1261,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
  			"failed to send request %d\n", ret);
  		nvme_tcp_fail_request(queue->request);
  		nvme_tcp_done_send_req(queue);
+		if (ret == -EPIPE)
+			nvme_tcp_error_recovery(&queue->ctrl->ctrl);

Won't nvme_tcp_error_recover() fail to transition the state and exit before doing anything?

#4 lore raw

čt 23. 1. 2025 v 23:33 odesílatel Sagi Grimberg <[email protected]> napsal:

Quoted 4 lines

Won't nvme_tcp_error_recover() fail to transition the state and exit before doing anything?

No, because it's called in nvme_tcp_try_send() when the call to sock_sendmsg() fails,
so the controller has already transitioned to LIVE state.

I will modify the commit message to make it clearer.

Maurizio