Skip to contentSkip to conversation

nvme-tcp: fence TCP socket on transport error

PATCH Superseded Chris Leech 4 participants 8 messages Mar 20, 2023 – Apr 5, 2023 NVMe

lore
Share & tools
lore
#0 lore raw

cover nvme-tcp: fence TCP socket on transport error

Even after "160f3549a907 nvme-tcp: fix UAF when detecting digest errors", I don't think the queue->rd_enabled flag is fencing the TCP socket from further receive processing.

io_work can be re-queued while running, and if it's pending when a receive error occurs it will bypass the queue->rd_enabled checks that should prevent queueing in nvme_tcp_data_ready. Actually, it looks like nvme_tcp_write_space and nvme_tcp_queue_request would schedule io_work anyway, which will always call nvme_tcp_try_recv once regardless of rd_enabled. And nvme_tcp_poll has no checks against rd_enabled.

After receiving an unsupported PDU, the header has been read but the payload remains in the socket queue. And the nvme_tcp queue state looks like it's ready to receive the payload of a C2H data PDU. Any additional calls to nvme_tcp_try_recv can incorerectly interpret the next bits as a command ID, lookup a request from the tagset using this bogus ID, and start copying the payload data from the unsupported PDU to an invalid destination address.

This has been seen with a buggy target that sent extranious bytes in the TCP stream, but also I believe with a properly functioning target that sent a Controller to Host Terminate Connection (C2HTermReq). The Fatal Error Status field was used as a Command ID and brought the host system down.

An additonal check against queue->rd_enabled at the start of nvme_tcp_recv_skb should protect against both additonal io_work scheduling and nvme_tcp_poll use after a receive transport error.

- Chris

Chris Leech (1):
nvme-tcp: fence TCP socket on receive error

 drivers/nvme/host/tcp.c | 7 +++++++
 1 file changed, 7 insertions(+)
Signature · 1 line
2.39.2
#1 lore raw

PATCH 1/1 nvme-tcp: fence TCP socket on transport error

Ensure that no further socket reads occur after a receive processing error, either from io_work being re-scheduled or nvme_tcp_poll.

Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.

---
 drivers/nvme/host/tcp.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/nvme/host/tcp.c +7 -0
index 42c0598c31f2..49e8eb576527 100644
@@ -888,6 +888,13 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,
 	size_t consumed = len;
 	int result;
 
+	if (!queue->rd_enabled) {
+		/* io_work or polling happening after receive error
+		 * waiting on error recovery
+		 */
+		return -EFAULT;
+	}
+
 	while (len) {
 		switch (nvme_tcp_recv_state(queue)) {
 		case NVME_TCP_RECV_PDU:
Signature · 1 line
2.39.2
maintainer
#2 lore raw

Hey Chris,

Quoted 7 lines

Ensure that no further socket reads occur after a receive processing error, either from io_work being re-scheduled or nvme_tcp_poll.

Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.

I agree with your analysis.

Quoted 11 lines
---
  drivers/nvme/host/tcp.c | 7 +++++++
  1 file changed, 7 insertions(+)
drivers/nvme/host/tcp.c +0 -0
index 42c0598c31f2..49e8eb576527 100644

drivers/nvme/host/tcp.c static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, · line 896

Quoted 9 lines
drivers/nvme/host/tcp.c +6 -0
@@ -888,6 +888,13 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,
  	size_t consumed = len;
  	int result;
 
+	if (!queue->rd_enabled) {
+		/* io_work or polling happening after receive error
+		 * waiting on error recovery
+		 */
+		return -EFAULT;
+	}

I think we can drop the comment, the code is somewhat self-explanatory, if read is not enabled, we shouldn't try and read from the socket.

	if (!queue->rd_enabled)
		return -EFAULT;
#3 lore raw
Quoted 7 lines · Sagi Grimberg

drivers/nvme/host/tcp.c static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, · line 896

Quoted 6 lines
drivers/nvme/host/tcp.c +6 -0
+     if (!queue->rd_enabled) {
+             /* io_work or polling happening after receive error
+              * waiting on error recovery
+              */
+             return -EFAULT;
+     }
Quoted 5 lines

I think we can drop the comment, the code is somewhat self-explanatory, if read is not enabled, we shouldn't try and read from the socket.

        if (!queue->rd_enabled)
                return -EFAULT;

Sure, and as it's an error check in the data path I'll add an unlikely hint.

- Chris

#4 lore raw

PATCH v2 1/1 nvme-tcp: fence TCP socket on receive error

In reply to Chris Leech

Ensure that no further socket reads occur after a receive processing error, either from io_work being re-scheduled or nvme_tcp_poll.

Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.

---
 drivers/nvme/host/tcp.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/nvme/host/tcp.c +3 -0
index 42c0598c31f2..99ad715210af 100644
@@ -888,6 +888,9 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,
 	size_t consumed = len;
 	int result;
 
+	if (unlikely(!queue->rd_enabled))
+		return -EFAULT;
+
 	while (len) {
 		switch (nvme_tcp_recv_state(queue)) {
 		case NVME_TCP_RECV_PDU:
Signature · 1 line
2.39.2
Reviewed-by on v2 1/1
#5 lore raw

Re: [PATCH v2 1/1] nvme-tcp: fence TCP socket on receive error

Quoted 27 lines · Chris Leech

Ensure that no further socket reads occur after a receive processing error, either from io_work being re-scheduled or nvme_tcp_poll.

Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.

---
  drivers/nvme/host/tcp.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/nvme/host/tcp.c +3 -0
index 42c0598c31f2..99ad715210af 100644
@@ -888,6 +888,9 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,
  	size_t consumed = len;
  	int result;
 
+	if (unlikely(!queue->rd_enabled))
+		return -EFAULT;
+
  	while (len) {
  		switch (nvme_tcp_recv_state(queue)) {
  		case NVME_TCP_RECV_PDU:
#7 lore raw

Re: [PATCH v2 1/1] nvme-tcp: fence TCP socket on receive error

In reply to Chris Leech

Thanks,

applied to nvme-6.4.

Let me know if you really need this in 6.3, but it seems like we're pretty late in the cycle and this a rather old bug.

Nit: a Fіxes tag would have been nice.