Skip to contentSkip to conversation

Fix some kernel panics due to the host sending invalid PDUs

PATCH Superseded Maurizio Lombardi 2 participants 6 messages Dec 21, 2023 – Dec 22, 2023

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Fix some kernel panics due to the host sending invalid PDUs

I used the reproducer that was published here:
https://lore.kernel.org/linux-nvme/[email protected]/T/#m57f25abc944dc706125fed02cb99d86b56c5d36e

I didn't get the same call trace, however.

Maurizio Lombardi (2):
nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU
length
nvmet-tcp: fix a crash in nvmet_req_complete()

 drivers/nvme/target/tcp.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
Signature · 1 line
2.39.3
#1 lore raw

PATCH 1/2 nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU length

If the host sends an H2CData command with a DATAL field equal to 0, the kernel crashes in nvmet_tcp_build_pdu_iovec().

Unable to handle kernel NULL pointer dereference at
virtual address 0000000000000000
lr : nvmet_tcp_io_work+0x6ac/0x718 [nvmet_tcp]
Call trace:
  process_one_work+0x174/0x3c8
  worker_thread+0x2d0/0x3e8
  kthread+0x104/0x110

Getting an H2CData Packet with DATAL=0 is unexpected, treat it like a fatal transport error.
Also, the PDU length should never exceed the MAXH2CDATA parameter which has been communicated to the host in nvmet_tcp_handle_icreq().

---
 drivers/nvme/target/tcp.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/nvme/target/tcp.c +9 -1
index 4cc27856aa8f..44eda6081e99 100644
@@ -24,6 +24,7 @@
 #include "nvmet.h"
 
 #define NVMET_TCP_DEF_INLINE_DATA_SIZE	(4 * PAGE_SIZE)
+#define NVMET_TCP_MAXH2CDATA		0x400000 /* 16M arbitrary limit */
 
 static int param_store_val(const char *str, int *val, int min, int max)
 {
@@ -923,7 +924,7 @@ static int nvmet_tcp_handle_icreq(struct nvmet_tcp_queue *queue)
 	icresp->hdr.pdo = 0;
 	icresp->hdr.plen = cpu_to_le32(icresp->hdr.hlen);
 	icresp->pfv = cpu_to_le16(NVME_TCP_PFV_1_0);
-	icresp->maxdata = cpu_to_le32(0x400000); /* 16M arbitrary limit */
+	icresp->maxdata = cpu_to_le32(NVMET_TCP_MAXH2CDATA);
 	icresp->cpda = 0;
 	if (queue->hdr_digest)
 		icresp->digest |= NVME_TCP_HDR_DIGEST_ENABLE;
@@ -1002,6 +1003,13 @@ static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue)
 	}
 
 	cmd->pdu_len = le32_to_cpu(data->data_length);
+	if (unlikely(cmd->pdu_len == 0 ||
+		     cmd->pdu_len > NVMET_TCP_MAXH2CDATA)) {
+		pr_err("H2CData PDU len out of range\n");
+		/* FIXME: use proper transport errors */
+		nvmet_tcp_fatal_error(queue);
+		return -EPROTO;
+	}
 	cmd->pdu_recv = 0;
 	nvmet_tcp_build_pdu_iovec(cmd);
 	queue->cmd = cmd;
Signature · 1 line
2.39.3
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU length

drivers/nvme/target/tcp.c static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue) · line 1008

Quoted 7 lines · Maurizio Lombardi
drivers/nvme/target/tcp.c +3 -0
@@ -1002,6 +1003,13 @@ static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue)
 	}
 
 	cmd->pdu_len = le32_to_cpu(data->data_length);
+	if (unlikely(cmd->pdu_len == 0 ||
+		     cmd->pdu_len > NVMET_TCP_MAXH2CDATA)) {
+		pr_err("H2CData PDU len out of range\n");

Please add the 'pdu_len' to the pr_err since that info sounds useful to knowing which type of target error we're dealing with.

Otherwise looks good!

Quoted 7 lines
drivers/nvme/target/tcp.c +4 -0
+		/* FIXME: use proper transport errors */
+		nvmet_tcp_fatal_error(queue);
+		return -EPROTO;
+	}
 	cmd->pdu_recv = 0;
 	nvmet_tcp_build_pdu_iovec(cmd);
 	queue->cmd = cmd;
#3 lore raw

Re: [PATCH 1/2] nvmet-tcp: Fix a kernel panic when host sends an invalid H2C PDU length

čt 21. 12. 2023 v 18:42 odesílatel Keith Busch <[email protected]> napsal:

Quoted 9 lines · Maurizio Lombardi

Please add the 'pdu_len' to the pr_err since that info sounds useful to knowing which type of target error we're dealing with.

Otherwise looks good!

Thanks,

I realized that there is still a possible problem, the data_length sent by the host
could be inside the allowed range, but still higher than the real size of the packet.
This could make the kernel crash.

Today I will send a V2 and add more fixes to this patchset.

Maurizio

#4 lore raw

PATCH 2/2 nvmet-tcp: fix a crash in nvmet_req_complete()

In reply to Maurizio Lombardi

in nvmet_tcp_handle_h2c_data_pdu(), if the host sends a data_offset different from rbytes_done, the driver ends up calling nvmet_req_complete() passing a status error.
The problem is that at this point cmd->req is not yet initialized, the kernel will crash after dereferencing a NULL pointer.

Fix the bug by replacing the call to nvmet_req_complete() with nvmet_tcp_fatal_error().

---
 drivers/nvme/target/tcp.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/target/tcp.c +1 -2
index 44eda6081e99..0e719460199f 100644
@@ -997,8 +997,7 @@ static int nvmet_tcp_handle_h2c_data_pdu(struct nvmet_tcp_queue *queue)
 			data->ttag, le32_to_cpu(data->data_offset),
 			cmd->rbytes_done);
 		/* FIXME: use path and transport errors */
-		nvmet_req_complete(&cmd->req,
-			NVME_SC_INVALID_FIELD | NVME_SC_DNR);
+		nvmet_tcp_fatal_error(queue);
 		return -EPROTO;
 	}
 
Signature · 1 line
2.39.3
maintainer Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] nvmet-tcp: fix a crash in nvmet_req_complete()

Quoted 8 lines · Maurizio Lombardi

in nvmet_tcp_handle_h2c_data_pdu(), if the host sends a data_offset different from rbytes_done, the driver ends up calling nvmet_req_complete() passing a status error.
The problem is that at this point cmd->req is not yet initialized, the kernel will crash after dereferencing a NULL pointer.

Fix the bug by replacing the call to nvmet_req_complete() with nvmet_tcp_fatal_error().

Looks good. The bug this fixes goes back to the beginning:

Fixes 872d26a391da92 ("nvmet-tcp: add NVMe over TCP target driver")