nvme-tcp: fence TCP socket on transport error
PATCH 1/1 nvme-tcp: fence TCP socket on transport error
In reply to Chris LeechEnsure that no further socket reads occur after a receive processing error, either from io_work being re-scheduled or nvme_tcp_poll.
Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.
--- drivers/nvme/host/tcp.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/nvme/host/tcp.c +7 -0
@@ -888,6 +888,13 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,size_t consumed = len;int result; +if (!queue->rd_enabled) { +/* io_work or polling happening after receive error +* waiting on error recovery +*/ +return -EFAULT ; + } +while (len) {switch (nvme_tcp_recv_state (queue)) {case NVME_TCP_RECV_PDU :
Signature · 1 line
2.39.2
cover nvme-tcp: fence TCP socket on transport error
Even after "160f3549a907 nvme-tcp: fix UAF when detecting digest errors", I don't think the queue->rd_enabled flag is fencing the TCP socket from further receive processing.
io_work can be re-queued while running, and if it's pending when a receive error occurs it will bypass the queue->rd_enabled checks that should prevent queueing in nvme_tcp_data_ready. Actually, it looks like nvme_tcp_write_space and nvme_tcp_queue_request would schedule io_work anyway, which will always call nvme_tcp_try_recv once regardless of rd_enabled. And nvme_tcp_poll has no checks against rd_enabled.
After receiving an unsupported PDU, the header has been read but the payload remains in the socket queue. And the nvme_tcp queue state looks like it's ready to receive the payload of a C2H data PDU. Any additional calls to nvme_tcp_try_recv can incorerectly interpret the next bits as a command ID, lookup a request from the tagset using this bogus ID, and start copying the payload data from the unsupported PDU to an invalid destination address.
This has been seen with a buggy target that sent extranious bytes in the TCP stream, but also I believe with a properly functioning target that sent a Controller to Host Terminate Connection (C2HTermReq). The Fatal Error Status field was used as a Command ID and brought the host system down.
An additonal check against queue->rd_enabled at the start of nvme_tcp_recv_skb should protect against both additonal io_work scheduling and nvme_tcp_poll use after a receive transport error.
- Chris
Chris Leech (1):
nvme-tcp: fence TCP socket on receive error
drivers/nvme/host/tcp.c | 7 +++++++ 1 file changed, 7 insertions(+)
Signature · 1 line
2.39.2
Sagi Grimberg
maintainerHey Chris,
Quoted 7 lines
Ensure that no further socket reads occur after a receive processing error, either from
io_workbeing re-scheduled ornvme_tcp_poll.Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.
I agree with your analysis.
Quoted 11 lines
Signed-off-by Chris Leech <[email protected]>--- drivers/nvme/host/tcp.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, · line 896
Quoted 9 lines
drivers/nvme/host/tcp.c +6
-0@@ -888,6 +888,13 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,size_t consumed = len;int result; +if (!queue->rd_enabled) { +/* io_work or polling happening after receive error +* waiting on error recovery +*/ +return -EFAULT ; + }
I think we can drop the comment, the code is somewhat self-explanatory, if read is not enabled, we shouldn't try and read from the socket.
if (!queue->rd_enabled) return -EFAULT;
Quoted 7 lines · Sagi Grimberg
drivers/nvme/host/tcp.c static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, · line 896
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+if (!queue->rd_enabled) { +/* io_work or polling happening after receive error +* waiting on error recovery +*/ +return -EFAULT ; + }
Quoted 5 lines
I think we can drop the comment, the code is somewhat self-explanatory, if read is not enabled, we shouldn't try and read from the socket.
if (!queue->rd_enabled) return -EFAULT;
Sure, and as it's an error check in the data path I'll add an unlikely hint.
- Chris
PATCH v2 1/1 nvme-tcp: fence TCP socket on receive error
In reply to Chris LeechEnsure that no further socket reads occur after a receive processing error, either from io_work being re-scheduled or nvme_tcp_poll.
Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.
--- drivers/nvme/host/tcp.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/nvme/host/tcp.c +3 -0
@@ -888,6 +888,9 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,size_t consumed = len;int result; +if (unlikely (!queue->rd_enabled)) +return -EFAULT ; +while (len) {switch (nvme_tcp_recv_state (queue)) {case NVME_TCP_RECV_PDU :
Signature · 1 line
2.39.2
Re: [PATCH v2 1/1] nvme-tcp: fence TCP socket on receive error
Quoted 27 lines · Chris Leech
Ensure that no further socket reads occur after a receive processing error, either from
io_workbeing re-scheduled ornvme_tcp_poll.Failing to do so can result in unrecognised PDU payloads or TCP stream garbage being processed as a C2H data PDU, and potentially start copying the payload to an invalid destination after looking up a request using a bogus command id.
Signed-off-by Chris Leech <[email protected]>--- drivers/nvme/host/tcp.c | 3 +++ 1 file changed, 3 insertions(+)drivers/nvme/host/tcp.c +3
-0@@ -888,6 +888,9 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,size_t consumed = len;int result; +if (unlikely (!queue->rd_enabled)) +return -EFAULT ; +while (len) {switch (nvme_tcp_recv_state (queue)) {case NVME_TCP_RECV_PDU :
Re: [PATCH v2 1/1] nvme-tcp: fence TCP socket on receive error
In reply to Chris LeechThanks Chris,
Christoph Hellwig
AppliedRe: [PATCH v2 1/1] nvme-tcp: fence TCP socket on receive error
In reply to Chris LeechThanks,
applied to nvme-6.4.
Let me know if you really need this in 6.3, but it seems like we're pretty late in the cycle and this a rather old bug.
Nit: a Fіxes tag would have been nice.