nvme-tcp: fixup I/O stall on congested sockets
Subsystems (3)
v3 cover nvme-tcp: fixup I/O stall on congested sockets
Hi all,
I have been chasing keep-alive timeouts with TLS enabled in the last few days (weeks, even :-( ). On larger setups (eg with 32 queues) the connection never got established properly as I've been hitting keep-alive timeouts before the last queue got connected.
Turns out that occasionally we simply do not send the keep-alive request; it's been added to the request list but the io_work workqueue function is never restarted as it bails out after nvme_tcp_try_recv() returns -EAGAIN. During debugging I also found that we're quite lazy with the list handling of requests, so I've added two preliminary patches to ensure that all list elements are properly terminated.
As usual, comments and reviews are welcome.
Changes to v2:
- Removed AEN patches again
Changes to the original submission:
- Include reviews from Chris Leech
- Add patch to requeue namespace scan
- Add patch to re-read ANA log page
Hannes Reinecke (3):
nvme-tcp: open-code nvme_tcp_queue_request() for R2T
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets
drivers/nvme/host/tcp.c | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-)
Signature · 1 line
2.35.3
PATCH v3 1/3 nvme-tcp: open-code nvme_tcp_queue_request() for R2T
When handling an R2T PDU we short-circuit nvme_tcp_queue_request() as we should not attempt to send consecutive PDUs. So open-code nvme_tcp_queue_request() for R2T and drop the last argument.
--- drivers/nvme/host/tcp.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/nvme/host/tcp.c +7 -5
@@ -404,7 +404,7 @@ static inline bool nvme_tcp_queue_more(struct nvme_tcp_queue *queue) }static inline void nvme_tcp_queue_request (struct nvme_tcp_request *req, -bool sync,bool last) +bool last) {struct nvme_tcp_queue *queue = req->queue;bool empty;
@@ -418,7 +418,7 @@ static inline void nvme_tcp_queue_request(struct nvme_tcp_request *req,* are on the same cpu, so we don't introduce contention. */ if (queue->io_cpu ==raw_smp_processor_id () && - sync && empty &&mutex_trylock (&queue->send_mutex)) { + empty &&mutex_trylock (&queue->send_mutex)) {nvme_tcp_send_all (queue);mutex_unlock (&queue->send_mutex); }
@@ -771,7 +771,9 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, req->ttag = pdu->ttag;nvme_tcp_setup_h2c_data_pdu (req); -nvme_tcp_queue_request (req, false, true); + +llist_add (&req->lentry, &queue->req_list); +queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);return 0 ; }
@@ -2557,7 +2559,7 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg) ctrl->async_req.curr_bio =NULL ; ctrl->async_req.data_len =0 ; -nvme_tcp_queue_request (&ctrl->async_req, true, true); +nvme_tcp_queue_request (&ctrl->async_req, true); }static void nvme_tcp_complete_timed_out (struct request *rq)
@@ -2709,7 +2711,7 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx,nvme_start_request (rq); -nvme_tcp_queue_request (req, true, bd->last); +nvme_tcp_queue_request (req, bd->last);return BLK_STS_OK ; }
Signature · 1 line
2.35.3
PATCH v3 3/3 nvme-tcp: fix I/O stalls on congested sockets
In reply to Hannes ReineckeWhen the socket is busy processing nvme_tcp_try_recv() might return -EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests once nvme_tcp_try_recv() returns -EAGAIN and continue with the sending loop to avoid I/O stalls.
--- drivers/nvme/host/tcp.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +4 -1
@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result =nvme_tcp_try_recv (queue);if (result >0 ) pending = true; -else if (unlikely (result <0 )) +else if (unlikely (result <0 ) && result != -EAGAIN )return ; +if (nvme_tcp_queue_has_pending (queue)) + pending = true; +if (!pending || !queue->rd_enabled)return ;
Signature · 1 line
2.35.3
PATCH v3 2/3 nvme-tcp: sanitize request list handling
In reply to Hannes ReineckeValidate the request in nvme_tcp_handle_r2t() to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.
--- drivers/nvme/host/tcp.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +16 -1
@@ -454,6 +454,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) }list_del (&req->entry); +init_llist_node (&req->lentry);return req; }
@@ -561,6 +562,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue;nvme_req (rq)->ctrl = &ctrl->ctrl;nvme_req (rq)->cmd = &pdu->cmd; +init_llist_node (&req->lentry); +INIT_LIST_HEAD (&req->entry);return 0 ; }
@@ -765,6 +768,15 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,return -EPROTO ; } +if (queue->request == req || +llist_on_list (&req->lentry) || + !list_empty (&req->entry)) { +dev_err (queue->ctrl->ctrl.device, +"req %d unexpected r2t while processing request\n" , + rq->tag); +return -EPROTO ; + } + req->pdu_len =0 ; req->h2cdata_left = r2t_length; req->h2cdata_offset = r2t_offset;
@@ -773,7 +785,8 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,nvme_tcp_setup_h2c_data_pdu (req);llist_add (&req->lentry, &queue->req_list); -queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work); +if (list_empty (&queue->send_list)) +queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);return 0 ; }
@@ -2558,6 +2571,8 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg) ctrl->async_req.offset =0 ; ctrl->async_req.curr_bio =NULL ; ctrl->async_req.data_len =0 ; +init_llist_node (&ctrl->async_req.lentry); +INIT_LIST_HEAD (&ctrl->async_req.entry);nvme_tcp_queue_request (&ctrl->async_req, true); }
Signature · 1 line
2.35.3
Re: [PATCH 1/3] nvme-tcp: open-code nvme_tcp_queue_request() for R2T
In reply to Hannes ReineckeThis is better indeed,
Sagi Grimberg
maintainerRe: [PATCH 2/3] nvme-tcp: sanitize request list handling
In reply to Hannes ReineckeQuoted 3 lines · Hannes Reinecke
Validate the request in
nvme_tcp_handle_r2t()to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.
Not clear what do you mean by "malicious R2T PDU".
Can you please clarify what you have seen/observed in the commit msg (i.e. what led you to this patch)?
Quoted 44 lines
Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/tcp.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +12
-0@@ -454,6 +454,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) }list_del (&req->entry); +init_llist_node (&req->lentry);return req; }@@ -561,6 +562,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue;nvme_req (rq)->ctrl = &ctrl->ctrl;nvme_req (rq)->cmd = &pdu->cmd; +init_llist_node (&req->lentry); +INIT_LIST_HEAD (&req->entry);return 0 ; }@@ -765,6 +768,15 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,return -EPROTO ; } +if (queue->request == req || +llist_on_list (&req->lentry) || + !list_empty (&req->entry)) { +dev_err (queue->ctrl->ctrl.device, +"req %d unexpected r2t while processing request\n" , + rq->tag); +return -EPROTO ; + } + req->pdu_len =0 ; req->h2cdata_left = r2t_length; req->h2cdata_offset = r2t_offset;
drivers/nvme/host/tcp.c static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, · line 789
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-1@@ -773,7 +785,8 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,nvme_tcp_setup_h2c_data_pdu (req);llist_add (&req->lentry, &queue->req_list); -queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work); +if (list_empty (&queue->send_list)) +queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);
Is this change mandatory? looks out of place.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Hannes ReineckeQuoted 18 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/tcp.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1393
Quoted 6 lines
drivers/nvme/host/tcp.c +1
-1@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result =nvme_tcp_try_recv (queue);if (result >0 ) pending = true; -else if (unlikely (result <0 )) +else if (unlikely (result <0 ) && result != -EAGAIN )return ;
The way that the send path was done - is that EAGAIN returns 0 (success returns >0, failure returns <0) Perhaps we can make recv do the same?
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1397
Quoted 4 lines
drivers/nvme/host/tcp.c +3
-0+if (nvme_tcp_queue_has_pending (queue)) + pending = true; +
Something is not clear to me, this suggest that try_send was not able to send data on the socket, shouldn't a .write_space() callback wake you when the socket send buffer gets some space? Why do you immediately try more even if you're sendmsg is returning EAGAIN?
This is specific to TLS I assume here?
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 16 lines · Sagi Grimberg
Quoted 13 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)
Quoted 16 lines
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct · line 1394
Quoted 12 lines
drivers/nvme/host/tcp.c +1
-1@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue); if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return;The way that the send path was done - is that EAGAIN returns 0 (success returns >0, failure returns <0) Perhaps we can make recv do the same?
I guess we can.
Quoted 7 lines
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1397
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ if (nvme_tcp_queue_has_pending(queue)) + pending = true; +Something is not clear to me, this suggest that
try_sendwas not able to send data on the socket, shouldn't a .write_space() callback wake you when the socket send buffer gets some space? Why> do you immediately try more even if you're sendmsg is returning EAGAIN?
But that's precisely the point: sendmsg() did _not_ return -EAGAIN. recvmsg() did.
We just imply that nvme_tcp_try_send() will return -EAGAIN once nvme_tcp_try_recv() did.
Which is wrong; -EAGAIN on reception can be due to a number of factors, and does _not_ imply in any shape or form that sending will exhibit the same error.
Quoted 1 line
This is specific to TLS I assume here?
Not necessarily; I've seen it with my performance testing on 10GigE with normal TCP, too. TLS is just an easier way to reproduce it.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 2/3] nvme-tcp: sanitize request list handling
In reply to Sagi GrimbergQuoted 10 lines · Sagi Grimberg
Quoted 3 lines · Hannes Reinecke
Validate the request in
nvme_tcp_handle_r2t()to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.Not clear what do you mean by "malicious R2T PDU".
Can you please clarify what you have seen/observed in the commit msg (i.e. what led you to this patch)?
This is coming from code inspection only.
In nvme_tcp_handle_r2t() we are looking up a request by the 'command_id' value in the pdu, and then add it to 'queue->req_list' without further checking.
So if a malicious R2T packet is received containing the command_id of a command currently on 'queue->req_list' we'll end up with duplicate entries on the list.
[ .. ]
Quoted 10 lines
drivers/nvme/host/tcp.c static int nvme_tcp_handle_r2t(struct · line 789
Quoted 7 lines
drivers/nvme/host/tcp.c +2
-1@@ -773,7 +785,8 @@ static int nvme_tcp_handle_r2t(structnvme_tcp_queue *queue, nvme_tcp_setup_h2c_data_pdu(req); llist_add(&req->lentry, &queue->req_list); - queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); + if (list_empty(&queue->send_list)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);Is this change mandatory? looks out of place.
Arguably an optimisation. I can leave it out.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Hannes ReineckeQuoted 39 lines · Hannes Reinecke
Quoted 16 lines · Sagi Grimberg
Quoted 13 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)Quoted 16 lines
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1394
Quoted 12 lines
drivers/nvme/host/tcp.c +1
-1@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue); if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return;The way that the send path was done - is that EAGAIN returns 0 (success returns >0, failure returns <0) Perhaps we can make recv do the same?
I guess we can.
Quoted 4 lines
Quoted 3 lines
drivers/nvme/host/tcp.c +3
-0+ if (nvme_tcp_queue_has_pending(queue)) + pending = true; +
Quoted 6 lines
Quoted 2 lines
Something is not clear to me, this suggest that
try_sendwas not able to send data on the socket, shouldn't a .write_space() callback wake you when the socket send buffer gets some space? Why> do you immediately trymore even if you're sendmsg is returning EAGAIN?
Quoted 1 line
But that's precisely the point:
sendmsg()did _not_ return-EAGAIN.recvmsg()did.
What did try_send return? Why didn't it set pending?
Quoted 7 lines
We just imply that
nvme_tcp_try_send()will return-EAGAINoncenvme_tcp_try_recv()did.Which is wrong;
-EAGAINon reception can be due to a number of factors, and does _not_ imply in any shape or form that sending will exhibit the same error.
I get that recv returning EAGAIN, we should not return from io_work(). The second check is not clear to me. If try_recv got stuff, we set pending=1, if try_send was able to send, we set pending=1. They should be independent. Why is this check in the end of the loop correct.
It can be that the queue has pending requests, but there is no room in the socket send buffer, otherwise, it would have set pending=1 no?
I am missing something.
Sagi Grimberg
maintainerRe: [PATCH 2/3] nvme-tcp: sanitize request list handling
In reply to Hannes ReineckeQuoted 11 lines · Hannes Reinecke
Quoted 10 lines · Sagi Grimberg
Quoted 3 lines · Hannes Reinecke
Validate the request in
nvme_tcp_handle_r2t()to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.Not clear what do you mean by "malicious R2T PDU".
Can you please clarify what you have seen/observed in the commit msg (i.e. what led you to this patch)?
Quoted 7 lines
This is coming from code inspection only.
Innvme_tcp_handle_r2t()we are looking up a request by the 'command_id' value in the pdu, and then add it to 'queue->req_list' without further checking.
So if a malicious R2T packet is received containing thecommand_idof a command currently on 'queue->req_list' we'll end up with duplicate entries on the list.
I think this falls into a bucket of a bunch checks that the host does not do. But ok.
Quoted 13 lines
[ .. ]
Quoted 10 lines
drivers/nvme/host/tcp.c static int nvme_tcp_handle_r2t(struct · line 789
Quoted 7 lines
drivers/nvme/host/tcp.c +2
-1@@ -773,7 +785,8 @@ static int nvme_tcp_handle_r2t(structnvme_tcp_queue *queue, nvme_tcp_setup_h2c_data_pdu(req); llist_add(&req->lentry, &queue->req_list); - queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); + if (list_empty(&queue->send_list)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);Is this change mandatory? looks out of place.
Quoted 1 line
Arguably an optimisation. I can leave it out.
When it comes in this context it is not clear what it is designed to do. Don't mind having it, but lets separate it from this patch.
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Hannes ReineckeQuoted 21 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>--- drivers/nvme/host/tcp.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result =nvme_tcp_try_recv (queue);if (result >0 ) pending = true;
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1400
Quoted 10 lines
drivers/nvme/host/tcp.c +4
-1-else if (unlikely (result <0 )) +else if (unlikely (result <0 ) && result != -EAGAIN )return ; +if (nvme_tcp_queue_has_pending (queue)) + pending = true; +if (!pending || !queue->rd_enabled)return ;
The various 'try_send' function will return -EAGAIN for a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it. It would still be an optimisation as we're saving the round-trip via socket callbacks.
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 36 lines · Hannes Reinecke
Quoted 20 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue);drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1399
Quoted 10 lines
drivers/nvme/host/tcp.c +3
-1if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return; + if (nvme_tcp_queue_has_pending(queue)) + pending = true; + if (!pending || !queue->rd_enabled) return;The various '
try_send' function will return-EAGAINfor a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.
Quoted 2 lines
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it.
Umm, that is my understanding. If you tried to send X and were able to send Y where Y < X, you shouldn't have to keep trying in a busy loop, the stack should tell you when you can send again.
Quoted 3 lines
It would
still be an optimisation as we're saving the round-trip via socket callbacks.
But you are doing a busy loop on a socket that cannot accept new data, there are other sockets that the kthread can be working on.
Quoted 4 lines
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.
I still don't understand why a partial send warrants a busy loop call to sock_sendmsg...
My assumption is that the call right after the partial send, will see EAGAIN error. But I may be missing something here... I just never expected that a partial write means that we must busy loop sending to the socket.
What does a blocking sendmsg do under the hood? does it also follow this practice?
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 39 lines · Sagi Grimberg
Quoted 36 lines · Hannes Reinecke
Quoted 20 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue);drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1399
Quoted 10 lines
drivers/nvme/host/tcp.c +3
-1if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return; + if (nvme_tcp_queue_has_pending(queue)) + pending = true; + if (!pending || !queue->rd_enabled) return;The various '
try_send' function will return-EAGAINfor a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.
Quoted 6 lines
Quoted 2 lines
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it.
Umm, that is my understanding. If you tried to send X and were able to
send Y where Y < X, you shouldn't have to keep trying in a busy loop, the stack should tell you when you can send again.
Okay, I could try that.
Quoted 6 lines
Quoted 3 lines
It would
still be an optimisation as we're saving the round-trip via socket callbacks.But you are doing a busy loop on a socket that cannot accept new data, there are other sockets that the kthread can be working on.
But we might be _sending_ data, right?
Quoted 7 lines
Quoted 4 lines
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.I still don't understand why a partial send warrants a busy loop call to
sock_sendmsg...
This is, it's not just sendmsg. It's the combination of send and recv. In my tests I have seen sendmsg return with a partial/incomplete status, consequently recvmsg has nothing to receive, and io_work stops. And that is what the patch fixes.
I haven't checked the ->write_space() callback (which should've been triggered), but my feeling is that the ->write_space() callback hit when we were still busy processing, so the queue_work() went
nowhere.
Maybe we can fix it with setting a flag when ->write_space() triggers ('hey, more data to process'), to be evaluated during the io_work() loop. But that would be pretty close to the check 'nvme_tcp_queue_has_pending', so I'm not sure if we gain anything.
And I really haven't seen any detrimental performance effects with this patch; in the contrary, performance was on par, and if anything standard deviation went down.
Quoted 3 lines
My assumption is that the call right after the partial send, will see EAGAIN error. But I may
be missing something here... I just never expected that a partial write means that we must busy loop sending to the socket.
Well ... we do it in nvme_tcp_try_send_data().
Quoted 1 line
What does a blocking sendmsg do under the hood? does it also follow this practice?
How would I know. But I haven't seen any negative effects during performance testing with this patch.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 48 lines · Hannes Reinecke
Quoted 39 lines · Sagi Grimberg
Quoted 36 lines · Hannes Reinecke
Quoted 20 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue);drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1399
Quoted 10 lines
drivers/nvme/host/tcp.c +3
-1if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return; + if (nvme_tcp_queue_has_pending(queue)) + pending = true; + if (!pending || !queue->rd_enabled) return;The various '
try_send' function will return-EAGAINfor a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.Quoted 6 lines
Quoted 2 lines
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it.
Umm, that is my understanding. If you tried to send X and were able to
send Y where Y < X, you shouldn't have to keep trying in a busy loop, the stack should tell you when you can send again.Okay, I could try that.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It would
still be an optimisation as we're saving the round-trip via socket callbacks.But you are doing a busy loop on a socket that cannot accept new data, there are other sockets that the kthread can be working on.
But we might be _sending_ data, right?
I'd say that odds are you're not in the next attempt...
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.I still don't understand why a partial send warrants a busy loop call to
sock_sendmsg...This is, it's not just sendmsg. It's the combination of send and recv.
I agree with you that the recv fix is correct.
Quoted 3 lines
In my tests I have seen sendmsg return with a partial/incomplete status, consequently recvmsg has nothing to receive, and
io_workstops. And that is what the patch fixes.
The question is if the recv fix is sufficient?
Quoted 4 lines
I haven't checked the ->write_space() callback (which should've been triggered), but my feeling is that the ->write_space() callback hit when we were still busy processing, so the
queue_work()went
nowhere.
That makes sense.
Quoted 6 lines
Maybe we can fix it with setting a flag when ->write_space() triggers ('hey, more data to process'), to be evaluated during the
io_work()loop. But that would be pretty close to the check 'nvme_tcp_queue_has_pending', so I'm not sure if we gain anything.
How about checking sk_stream_is_writeable() instead?
I think we also need to flag the queue for the write_space during IO work...
Quoted 4 lines
And I really haven't seen any detrimental performance effects with this patch; in the contrary, performance was on par, and if anything standard deviation went down.
Still you agree that busy looping a socket that may not have space is less efficient? When there are other queues waiting to execute io_work?
I agree there is a problem here, but I am trying to figure out what we want to do here.
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 72d260201d8c..4eb9a2dec07e 100644
drivers/nvme/host/tcp.c +0 -0
@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
queue->nr_cqe = 0;
consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk);
- return consumed;
+ return consumed == -EAGAIN ? 0 : consumed;
}
static void nvme_tcp_io_work(struct work_struct *w) --
[2 based on your partial write fix]:
drivers/nvme/host/tcp.c +0 -0
@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags {
NVME_TCP_Q_LIVE = 1,
NVME_TCP_Q_POLLING = 2,
NVME_TCP_Q_IO_CPU_SET = 3,
+ NVME_TCP_Q_WAKE_SENDER = 4,
};
enum nvme_tcp_recv_state {
@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data;
if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); }
read_unlock_bh(&sk->sk_callback_lock);
@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1);
+ clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags);
do {
bool pending = false;
int result;
@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0))
return;
- if (!pending || !queue->rd_enabled)
+ /* did we get some space after spending time in recv ? */ + if (nvme_tcp_queue_has_pending(queue) &&
+ sk_stream_is_writeable(queue->sock->sk))
+ pending = true;
+
+ if (!queue->rd_enabled)
+ return;
+
+ if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags)) return;
} while (!time_after(jiffies, deadline)); /* quota is exhausted */ --
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Sagi,
I tried both of these patches but looks like #1 causes an infinite loop. dmesg was full of panics. I had tried just #1 and later with #1+#2. Both failed the same way.
Quoted 64 lines
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index72d260201d8c..4eb9a2dec07e100644drivers/nvme/host/tcp.c +1
-1@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe = 0; consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk); - return consumed; + return consumed == -EAGAIN ? 0 : consumed; }static void
nvme_tcp_io_work(struct work_struct *w)--[2 based on your partial write fix]:
drivers/nvme/host/tcp.c +12
-1@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_IO_CPU_SET = 3, + NVME_TCP_Q_WAKE_SENDER = 4, }; enum nvme_tcp_recv_state {@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data; if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); } read_unlock_bh(&sk->sk_callback_lock);@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1); + clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); do { bool pending = false; int result;@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0)) return; - if (!pending || !queue->rd_enabled) + /* did we get some space after spending time in recv ? */ + if (nvme_tcp_queue_has_pending(queue) && + sk_stream_is_writeable(queue->sock->sk)) + pending = true; + + if (!queue->rd_enabled) + return; + + if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags)) return;
Quoted 3 lines
} while (!time_after(jiffies, deadline)); /* quota is exhausted */ --
-Kamaljit
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Sagi,
I tried both of these patches but looks like #1 causes an infinite loop. dmesg was full of panics. I had tried just #1 and later with #1+#2. Both failed the same way.
Quoted 64 lines
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --gita/drivers/nvme/host/tcp.cb/drivers/nvme/host/tcp.cindex72d260201d8c..4eb9a2dec07e100644drivers/nvme/host/tcp.c +1
-1@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe = 0; consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk); -return consumed; +return consumed == -EAGAIN ?0 : consumed; }static void
nvme_tcp_io_work(struct work_struct *w)--[2 based on your partial write fix]:
drivers/nvme/host/tcp.c +12
-1@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_IO_CPU_SET = 3, +NVME_TCP_Q_WAKE_SENDER =4 , }; enum nvme_tcp_recv_state {@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data; if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); +set_bit (NVME_TCP_Q_WAKE_SENDER , &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); } read_unlock_bh(&sk->sk_callback_lock);@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1); +clear_bit (NVME_TCP_Q_WAKE_SENDER , &queue->flags); do { bool pending = false; int result;@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0)) return; -if (!pending || !queue->rd_enabled) +/* did we get some space after spending time in recv ? */ +if (nvme_tcp_queue_has_pending (queue) && +sk_stream_is_writeable (queue->sock->sk)) + pending = true; + +if (!queue->rd_enabled) +return ; + +if (!pending && !test_bit (NVME_TCP_Q_WAKE_SENDER , &queue->flags)) return;
Quoted 3 lines
} while (!time_after(jiffies, deadline)); /* quota is exhausted */ --
-Kamaljit
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Kamaljit SinghQuoted 3 lines · Kamaljit Singh
Sagi,
I tried both of these patches but looks like #1 causes an infinite loop. dmesg was full of panics. I had tried just #1 and later with #1+#2. Both failed the same way.
That's no good :)
Can you share the panics? This version should be identical to what Hannes introduced in:
@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w)
result = nvme_tcp_try_recv(queue);
if (result > 0)
pending = true;
- else if (unlikely(result < 0))
+ else if (unlikely(result < 0) && result != -EAGAIN)
return;
Quoted 69 lines
Quoted 65 lines
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index72d260201d8c..4eb9a2dec07e100644drivers/nvme/host/tcp.c +1
-1@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe = 0; consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk); - return consumed; + return consumed == -EAGAIN ? 0 : consumed; }static void
nvme_tcp_io_work(struct work_struct *w)
--[2 based on your partial write fix]:
drivers/nvme/host/tcp.c +12
-1@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_IO_CPU_SET = 3, + NVME_TCP_Q_WAKE_SENDER = 4, }; enum nvme_tcp_recv_state {@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data; if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); } read_unlock_bh(&sk->sk_callback_lock);@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1); + clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); do { bool pending = false; int result;@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0)) return; - if (!pending || !queue->rd_enabled) + /* did we get some space after spending time in recv ? */ + if (nvme_tcp_queue_has_pending(queue) && + sk_stream_is_writeable(queue->sock->sk)) + pending = true; + + if (!queue->rd_enabled) + return; + + if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags)) return;Quoted 3 lines
} while (!time_after(jiffies, deadline)); /* quota is exhausted */
---Kamaljit
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Sagi,
From: Sagi Grimberg <[email protected]>
Date: Friday, April 18, 2025 at 03:51
Quoted 8 lines
Quoted 3 lines
Sagi,
I tried both of these patches but looks like #1 causes an infinite loop. dmesg was full of panics. I had tried just #1 and later with #1+#2. Both failed the same way.That's no good :)
Can you share the panics? This version should be identical to what Hannes introduced in:
Sorry, I'm posting ~200 lines at the bottom, starting from the beginning of the panic until the next "---cut here---". I can send the whole dmesg as attachment as well. Please let me know.
The patches I applied are these:
1: nvme-tcp: partial write tcp-q-wake-sender untested by Sagi
2: nvme-tcp: recv-side EAGAIN untested fix by Sagi
3: nvme-tcp: fix I/O stalls on congested sockets
4: nvme-tcp: sanitize request list handling
5: nvme-tcp: avoid inline sending when handling R2T PDUs
6: tls: use independent record sz for data sends <--- this is my internal/temp fix for a TLS send issue we found You can probably ignore #6.
[1] nvme-tcp: partial write tcp-q-wake-sender untested by Sagi
drivers/nvme/host/tcp.c +10 -2
@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags {NVME_TCP_Q_LIVE =1 ,NVME_TCP_Q_POLLING =2 ,NVME_TCP_Q_IO_CPU_SET =3 , + NVME_TCP_Q_WAKE_SENDER = 4, }; enum nvme_tcp_recv_state {
@@ -1074,6 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data;if (likely (queue &&sk_stream_is_writeable (sk))) {clear_bit (SOCK_NOSPACE , &sk->sk_socket->flags); +set_bit (NVME_TCP_Q_WAKE_SENDER , &queue->flags);queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work); }read_unlock_bh (&sk->sk_callback_lock);
@@ -1368,6 +1370,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work);unsigned long deadline = jiffies +msecs_to_jiffies (1 ); +clear_bit (NVME_TCP_Q_WAKE_SENDER , &queue->flags); do { bool pending = false; int result;
@@ -1387,10 +1390,15 @@ static void nvme_tcp_io_work(struct work_struct *w)else if (unlikely (result <0 ) && result != -EAGAIN )return ; -if (nvme_tcp_queue_has_pending (queue)) +/* did we get some space after spending time in recv ? */ +if (nvme_tcp_queue_has_pending (queue) && +sk_stream_is_writeable (queue->sock->sk)) pending = true; -if (!pending || !queue->rd_enabled) +if (!queue->rd_enabled) +return ; + +if (!pending && !test_bit (NVME_TCP_Q_WAKE_SENDER , &queue->flags))return ; }while (!time_after (jiffies, deadline));/* quota is exhausted */
[2] nvme-tcp: recv-side EAGAIN untested fix by Sagi
drivers/nvme/host/tcp.c +1 -1
@@ -1359,7 +1359,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe =0 ; consumed = sock->ops->read_sock (sk, &rd_desc, nvme_tcp_recv_skb);release_sock (sk); -return consumed; +return consumed == -EAGAIN ?0 : consumed; }static void nvme_tcp_io_work (struct work_struct *w)
[3] nvme-tcp: fix I/O stalls on congested sockets (by Hannes R.)
When the socket is busy processing nvme_tcp_try_recv() might return -EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests once nvme_tcp_try_recv() returns -EAGAIN and continue with the sending loop to avoid I/O stalls.
drivers/nvme/host/tcp.c +4 -1
@@ -1384,9 +1384,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result =nvme_tcp_try_recv (queue);if (result >0 ) pending = true; -else if (unlikely (result <0 )) +else if (unlikely (result <0 ) && result != -EAGAIN )return ; +if (nvme_tcp_queue_has_pending (queue)) + pending = true; +if (!pending || !queue->rd_enabled)return ;
[4] nvme-tcp: sanitize request list handling
Validate the request in nvme_tcp_handle_r2t() to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.
drivers/nvme/host/tcp.c +10 -1
@@ -455,6 +455,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) }list_del (&req->entry); +init_llist_node (&req->lentry);return req; }
@@ -562,6 +563,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue;nvme_req (rq)->ctrl = &ctrl->ctrl;nvme_req (rq)->cmd = &pdu->cmd; +init_llist_node (&req->lentry); +INIT_LIST_HEAD (&req->entry);return 0 ; }
@@ -773,8 +776,12 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,nvme_tcp_setup_h2c_data_pdu (req); +WARN_ON (queue->request == req); +WARN_ON (llist_on_list (&req->lentry)); +WARN_ON (!list_empty (&req->entry));llist_add (&req->lentry, &queue->req_list); -queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work); +if (list_empty (&queue->send_list)) +queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);return 0 ; }
@@ -2632,6 +2639,8 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg) ctrl->async_req.offset =0 ; ctrl->async_req.curr_bio =NULL ; ctrl->async_req.data_len =0 ; +init_llist_node (&ctrl->async_req.lentry); +INIT_LIST_HEAD (&ctrl->async_req.entry);nvme_tcp_queue_request (&ctrl->async_req, true); }
[5] nvme-tcp: avoid inline sending when handling R2T PDUs (by Hannes R.)
When handling an R2T PDU we should not attempt to send consecutive PDUs as we are running from an softirq context, and sending PDUs from the receive context will mess up latencies.
So just queue it and let the io_work workqueue function do the work.
drivers/nvme/host/tcp.c +7 -5
@@ -405,7 +405,7 @@ static inline bool nvme_tcp_queue_more(struct nvme_tcp_queue *queue) }static inline void nvme_tcp_queue_request (struct nvme_tcp_request *req, -bool sync,bool last) +bool last) {struct nvme_tcp_queue *queue = req->queue;bool empty;
@@ -419,7 +419,7 @@ static inline void nvme_tcp_queue_request(struct nvme_tcp_request *req,* are on the same cpu, so we don't introduce contention. */ if (queue->io_cpu ==raw_smp_processor_id () && - sync && empty &&mutex_trylock (&queue->send_mutex)) { + empty &&mutex_trylock (&queue->send_mutex)) {nvme_tcp_send_all (queue);mutex_unlock (&queue->send_mutex); }
@@ -772,7 +772,9 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, req->ttag = pdu->ttag;nvme_tcp_setup_h2c_data_pdu (req); -nvme_tcp_queue_request (req, false, true); + +llist_add (&req->lentry, &queue->req_list); +queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);return 0 ; }
@@ -2631,7 +2633,7 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg) ctrl->async_req.curr_bio =NULL ; ctrl->async_req.data_len =0 ; -nvme_tcp_queue_request (&ctrl->async_req, true, true); +nvme_tcp_queue_request (&ctrl->async_req, true); }static void nvme_tcp_complete_timed_out (struct request *rq)
@@ -2783,7 +2785,7 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx,nvme_start_request (rq); -nvme_tcp_queue_request (req, true, bd->last); +nvme_tcp_queue_request (req, bd->last);return BLK_STS_OK ; }
[6] tls: use independent record sz for data sends (by Kamaljit)
Author: Kamaljit Singh <[email protected]> Date: Wed Apr 9 15:11:54 2025 -0700
tls: use independent record sz for data sends
[Hack] send data to use a smaller default value of 4096 instead of the common macro TLS_MAX_PAYLOAD_SIZE, which applies to both directions.
Add a new user configurable parameter max_send_record_bytes to allow testing with different values other than 4096 bytes.
net/tls/tls_device.c +3 -1
@@ -55,6 +55,8 @@ static DEFINE_SPINLOCK(tls_device_lock);static struct page *dummy_page; +extern unsigned int max_send_record_bytes; +static void tls_device_free_ctx (struct tls_context *ctx) {if (ctx->tx_conf ==TLS_HW )
@@ -459,7 +461,7 @@ static int tls_push_data(struct sock *sk,/* TLS_HEADER_SIZE is not counted as part of the TLS record, and * we need to leave room for an authentication tag. */ - max_open_record_len =TLS_MAX_PAYLOAD_SIZE + + max_open_record_len = max_send_record_bytes + prot->prepend_size; do { rc = tls_do_allocation(sk, ctx, pfrag, prot->prepend_size);
net/tls/tls_main.c +5 -0
@@ -52,6 +52,11 @@ MODULE_DESCRIPTION("Transport Layer Security Support");MODULE_LICENSE ("Dual BSD/GPL" );MODULE_ALIAS_TCP_ULP ("tls" ); +unsigned int max_send_record_bytes =4096 ; +module_param (max_send_record_bytes, uint,0644 ); +MODULE_PARM_DESC (max_send_record_bytes ," max send record size in bytes (default 4096)" ); +EXPORT_SYMBOL_GPL (max_send_record_bytes); +enum {TLSV4 ,TLSV6 ,
net/tls/tls_sw.c +3 -1
@@ -48,6 +48,8 @@#include "tls.h" +extern unsigned int max_send_record_bytes; + struct tls_decrypt_arg { struct_group(inargs, bool zc;
@@ -1059,7 +1061,7 @@ static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, orig_size = msg_pl->sg.size; full_record = false; try_to_copy =msg_data_left (msg); - record_room =TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; + record_room = max_send_record_bytes - msg_pl->sg.size;if (try_to_copy >= record_room) { try_to_copy = record_room; full_record = true;
================================
Snapshot of panic messages:
...
[2025-04-16 22:55:32.992] [Wed Apr 16 22:55:34 2025] nvme nvme0: QID 0x0: traddr=10.10.10.223,trsvcid=4420,sport=0x8ac6,dport=0x1144
[2025-04-16 22:55:33.445] [Wed Apr 16 22:55:34 2025] nvme nvme0: D3 entry latency set to 16 seconds
[2025-04-16 22:55:33.898] [Wed Apr 16 22:55:35 2025] nvme nvme0: queue_size 128 > ctrl sqsize 17, clamping down
[2025-04-16 22:55:33.929] [Wed Apr 16 22:55:35 2025] nvme nvme0: creating 3 I/O queues.
[2025-04-16 22:55:35.398] [Wed Apr 16 22:55:36 2025] nvme nvme0: mapped 3/0/0 default/read/poll queues.
[2025-04-16 22:55:35.445] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID 0x1: traddr=10.10.10.223,trsvcid=4420,sport=0x8ac8,dport=0x1144
[2025-04-16 22:55:35.508] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID 0x2: traddr=10.10.10.223,trsvcid=4420,sport=0x8ad4,dport=0x1144
[2025-04-16 22:55:35.554] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID 0x3: traddr=10.10.10.223,trsvcid=4420,sport=0x8ad8,dport=0x1144
[2025-04-16 22:55:35.632] [Wed Apr 16 22:55:37 2025] nvme nvme0: new ctrl: NQN "nqn.2015-09.com.wdc:nvme.1", addr 10.10.10.223:4420, hostnqn: nqn.2014-08.org.nvmexpress:host.0
[2025-04-16 22:55:36.195] [Wed Apr 16 22:55:37 2025] nvme nvme0: Failed to get ANA log: 16386
[2025-04-16 22:55:43.023]
[2025-04-16 22:55:43.492]
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 22 PID: 134878 at tcp.c:782 nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] Modules linked in: nvme_tcp(OE) nvme_rdma(E) nvme_fabrics(E) nvme(E) nvme_core(E) nvme_keyring(E) nvme_auth(E) xt_tcpudp(E) nft_compat(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) cmac(E) nls_utf8(E) nfs(E) cifs(E) lockd(E) cifs_arc4(E) nls_ucs2_utils(E) grace(E) cifs_md4(E) nf_tables(E) netfs(E) cfg80211(E) ipmi_ssif(E) amd_atl(E) intel_rapl_msr(E) intel_rapl_common(E) rpcrdma(E) amd64_edac(E) rdma_ucm(E) edac_mce_amd(E) scsi_transport_iscsi(E) rdma_cm(E) ib_umad(E) ib_ipoib(E) iw_cm(E) kvm_amd(E) ib_cm(E) kvm(E) ftdi_sio(E) ast(E) [2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] usbserial(E)
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] rapl(E) i2c_algo_bit(E) acpi_ipmi(E) ccp(E) ipmi_si(E) i2c_piix4(E) ipmi_devintf(E) k10temp(E) ptdma(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 44 PID: 1609 at tcp.c:782 nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] i2c_smbus(E) joydev(E) input_leds(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] Modules linked in:
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ipmi_msghandler(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_tcp(OE) nvme_rdma(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] mac_hid(E) bnxt_re(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_fabrics(E) nvme(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sunrpc(E) binfmt_misc(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_core(E) nvme_keyring(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sch_fq_codel(E) efi_pstore(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_auth(E) xt_tcpudp(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nfnetlink(E) dmi_sysfs(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nft_compat(E) rpcsec_gss_krb5(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ip_tables(E) x_tables(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] auth_rpcgss(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] autofs4(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nfsv4(E) cmac(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] btrfs(E) blake2b_generic(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nls_utf8(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] zstd_compress(E) raid10(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nfs(E) cifs(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] raid456(E) async_raid6_recov(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] lockd(E) cifs_arc4(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] async_memcpy(E) async_pq(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nls_ucs2_utils(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] async_xor(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] grace(E) cifs_md4(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] async_tx(E) xor(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nf_tables(E) netfs(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] raid6_pq(E) raid1(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] cfg80211(E) ipmi_ssif(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] raid0(E) mlx5_ib(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] amd_atl(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_uverbs(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] intel_rapl_msr(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] macsec(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] intel_rapl_common(E) rpcrdma(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_core(E) mlx5_core(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] amd64_edac(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------ [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] mlxfw(E) psample(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] rdma_ucm(E) edac_mce_amd(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] polyval_clmulni(E) polyval_generic(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] scsi_transport_iscsi(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ghash_clmulni_intel(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 0 PID: 832 at tcp.c:782 nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp] [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sha256_ssse3(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] rdma_cm(E)
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] Modules linked in:
[2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_umad(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] sha1_ssse3(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_tcp(OE) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ahci(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_rdma(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] ib_ipoib(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] bnxt_en(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme_fabrics(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] iw_cm(E) [2025-04-16 22:56:24.773] [Wed Apr 16 22:56:26 2025] nvme(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] tls(E) pci_hyperv_intf(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] nvme_core(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] kvm_amd(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] ib_cm(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] nvme_keyring(E) nvme_auth(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] kvm(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] libahci(E) hid_generic(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] ftdi_sio(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] xt_tcpudp(E) nft_compat(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] ast(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] usbhid(E) hid(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] usbserial(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] rpcsec_gss_krb5(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] rapl(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] aesni_intel(E) [2025-04-16 22:56:24.805] [Wed Apr 16 22:56:26 2025] auth_rpcgss(E) nfsv4(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] crypto_simd(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cmac(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] i2c_algo_bit(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] acpi_ipmi(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] nls_utf8(E) nfs(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cryptd(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cifs(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] [last unloaded: nvme_tcp(E)] [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] ccp(E) ipmi_si(E) [2025-04-16 22:56:24.820] [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] i2c_piix4(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] lockd(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] ipmi_devintf(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] cifs_arc4(E) nls_ucs2_utils(E) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] CPU: 22 UID: 0 PID: 134878 Comm: kworker/22:0H Tainted: G OE 6.14.0-qp-sdprt-adc-49-50-51-52-ktls+ #1 PREEMPT(voluntary) [2025-04-16 22:56:24.820] [Wed Apr 16 22:56:26 2025] k10temp(E) ptdma(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] grace(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] i2c_smbus(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] cifs_md4(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] Hardware name: Supermicro AS -1114S-WTRT/H12SSW-NT, BIOS 1.0b 11/15/2019 [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] joydev(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] nf_tables(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] input_leds(E) [2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] ipmi_msghandler(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] Workqueue: nvme_tcp_wq nvme_tcp_io_work [nvme_tcp]
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] netfs(E) cfg80211(E)
[2025-04-16 22:56:24.835]
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] mac_hid(E) bnxt_re(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] ipmi_ssif(E)
[2025-04-16 22:56:24.835] [Wed Apr 16 22:56:26 2025] RIP: 0010:nvme_tcp_recv_skb+0x115e/0x11c0 [nvme_tcp]
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] amd_atl(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] sunrpc(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] Code: 00 41 8b 57 20 44 89 d1 4d 89 c8 48 c7 c6 88 d1 7a c1 e8 a5 b6 2f c9 e9 cc fb ff ff 0f 0b e9 28 fb ff ff 0f 0b e9 35 fb ff ff <0f> 0b e9 45 fb ff ff 49 8b 87 28 01 00 00 89 d1 89 f2 48 c7 c6 10 [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] binfmt_misc(E) sch_fq_codel(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] intel_rapl_msr(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RSP: 0018:ffffcdccf87afc80 EFLAGS: 00010293
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] intel_rapl_common(E)
[2025-04-16 22:56:24.851]
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] efi_pstore(E) nfnetlink(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] rpcrdma(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RAX: dead000000000100 RBX: ffff8dbda9193680 RCX: 0000000000000000
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] amd64_edac(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8dbd986b14a8
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] dmi_sysfs(E) [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] rdma_ucm(E) [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] ip_tables(E) x_tables(E) [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] edac_mce_amd(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] RBP: ffffcdccf87afd08 R08: 0000000000008000 R09: 0000000000000000 [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] scsi_transport_iscsi(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] R10: 0000000000000000 R11: 000000000000001c R12: ffff8dbe027c8290 [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] autofs4(E) btrfs(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] R13: 0000000000000005 R14: 000000000000001c R15: ffff8dbd986b1498 [2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] rdma_cm(E) ib_umad(E)
[2025-04-16 22:56:24.851] [Wed Apr 16 22:56:26 2025] FS: 0000000000000000(0000) GS:ffff8e3b3f1b4000(0000) knlGS:0000000000000000
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] blake2b_generic(E) [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] ib_ipoib(E) [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] zstd_compress(E) [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] raid10(E) [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] iw_cm(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] kvm_amd(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] CR2: 00005e9c7424be98 CR3: 00000002357ec000 CR4: 0000000000350ef0 [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] raid456(E) async_raid6_recov(E)
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] Call Trace:
[2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] ib_cm(E) [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] async_memcpy(E) [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] <TASK> [2025-04-16 22:56:24.867] [Wed Apr 16 22:56:26 2025] async_pq(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] kvm(E) ftdi_sio(E) ast(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] async_xor(E) async_tx(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] usbserial(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] xor(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] tls_sw_read_sock+0x12c/0x4a0 [tls] [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] rapl(E) i2c_algo_bit(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] raid6_pq(E) raid1(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] acpi_ipmi(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] ? __pfx_nvme_tcp_recv_skb+0x10/0x10 [nvme_tcp] [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] ccp(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] raid0(E) mlx5_ib(E) ib_uverbs(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] ipmi_si(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] nvme_tcp_try_recv+0x78/0xc0 [nvme_tcp] [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] i2c_piix4(E) [2025-04-16 22:56:24.882] [Wed Apr 16 22:56:26 2025] macsec(E) ib_core(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ipmi_devintf(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] nvme_tcp_io_work+0xc1/0x1e0 [nvme_tcp] [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] k10temp(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] mlx5_core(E) mlxfw(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ptdma(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] process_one_work+0x191/0x3e0 [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] i2c_smbus(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] psample(E) polyval_clmulni(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] joydev(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] worker_thread+0x2e3/0x420 [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] polyval_generic(E) ghash_clmulni_intel(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] input_leds(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] sha256_ssse3(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ? srso_return_thunk+0x5/0x5f [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] sha1_ssse3(E) [2025-04-16 22:56:24.898] [Wed Apr 16 22:56:26 2025] ipmi_msghandler(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ? __pfx_worker_thread+0x10/0x10 [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] mac_hid(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ahci(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] kthread+0x10d/0x230 [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] bnxt_re(E) sunrpc(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] bnxt_en(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ? __pfx_kthread+0x10/0x10 [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] tls(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] binfmt_misc(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] pci_hyperv_intf(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ret_from_fork+0x47/0x70 [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] sch_fq_codel(E) efi_pstore(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] libahci(E) [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] ? __pfx_kthread+0x10/0x10 [2025-04-16 22:56:24.914] [Wed Apr 16 22:56:26 2025] hid_generic(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] nfnetlink(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] ret_from_fork_asm+0x1a/0x30 [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] usbhid(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] dmi_sysfs(E) ip_tables(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] hid(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] </TASK> [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] aesni_intel(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] x_tables(E) autofs4(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] crypto_simd(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] btrfs(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] blake2b_generic(E) [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] cryptd(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] ---[ end trace 0000000000000000 ]--- [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] [last unloaded: nvme_tcp(E)] [2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] zstd_compress(E)
[2025-04-16 22:56:24.929]
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] CPU: 44 UID: 0 PID: 1609 Comm: kworker/44:1H Tainted: G W OE 6.14.0-qp-sdprt-adc-49-50-51-52-ktls+ #1 PREEMPT(voluntary)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] async_pq(E) async_xor(E)
[2025-04-16 22:56:24.929] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
...
...
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 8 lines · Kamaljit Singh
Sagi,
From: Sagi Grimberg <[email protected]>
Date: Friday, April 18, 2025 at 03:51Quoted 3 lines
Quoted 3 lines
Sagi,
I tried both of these patches but looks like #1 causes an infinite loop. dmesg was full of panics. I had tried just #1 and later with #1+#2. Both failed the same way.
Quoted 5 lines
Quoted 4 lines
That's no good :)
Can you share the panics? This version should be identical to what Hannes introduced in:
Sorry, I'm posting ~200 lines at the bottom, starting from the beginning of the panic until the next "---cut here---". I can send the whole dmesg as attachment as well. Please let me know.
No need.
However the log prints are WARN messages that does not exist in any of the patches.
Quoted 5 lines
The patches I applied are these:
1: nvme-tcp: partial write tcp-q-wake-sender untested by Sagi 2: nvme-tcp: recv-side EAGAIN untested fix by Sagi 3: nvme-tcp: fix I/O stalls on congested sockets
The patch I suggested was a replacement for patch (3). Didn't you get conflicts?
Quoted 4 lines
4: nvme-tcp: sanitize request list handling
5: nvme-tcp: avoid inline sending when handling R2T PDUs
6: tls: use independent record sz for data sends <--- this is my internal/temp fix for a TLS send issue we found You can probably ignore #6.
Ignoring.
Quoted 28 lines
[4] nvme-tcp: sanitize request list handling
Validate the request in
nvme_tcp_handle_r2t()to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.drivers/nvme/host/tcp.c +3
-0@@ -455,6 +455,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) }list_del (&req->entry); +init_llist_node (&req->lentry);return req; }@@ -562,6 +563,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue;nvme_req (rq)->ctrl = &ctrl->ctrl;nvme_req (rq)->cmd = &pdu->cmd; +init_llist_node (&req->lentry); +INIT_LIST_HEAD (&req->entry);return 0 ; }
drivers/nvme/host/tcp.c static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, · line 781
Quoted 6 lines
drivers/nvme/host/tcp.c +3
-0@@ -773,8 +776,12 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,nvme_tcp_setup_h2c_data_pdu (req); +WARN_ON (queue->request == req); +WARN_ON (llist_on_list (&req->lentry)); +WARN_ON (!list_empty (&req->entry));
These are not included in Hannes patch afaict.
I'm assuming that these are the stack traces that are firing...
Can you please share the output of gdb:
l *nvme_tcp_recv_skb+0x115e
My assumption is that in your case, queue->request is still referenced with the current request when the r2t for it arrives, which is very much possible...
So this means that the patch "nvme-tcp: sanitize request list handling" is wrong, queue->request may still be referenced when its r2t arrives.
I think this patch needs the following:
--
drivers/nvme/host/tcp.c +1 -2
@@ -767,8 +767,7 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,return -EPROTO ; } -if (queue->request == req || -llist_on_list (&req->lentry) || +if (llist_on_list (&req->lentry) || !list_empty (&req->entry)) { dev_err(queue->ctrl->ctrl.device, "req %d unexpected r2t while processing request\n",
--
Or in your case - remove the above:
--
WARN_ON(queue->request == req);
--
Quoted 18 lines
================================
Snapshot of panic messages:...
[2025-04-16 22:55:32.992] [Wed Apr 16 22:55:34 2025] nvme nvme0: QID0x0: traddr=10.10.10.223,trsvcid=4420,sport=0x8ac6,dport=0x1144
[2025-04-16 22:55:33.445] [Wed Apr 16 22:55:34 2025] nvme nvme0: D3 entry latency set to 16 seconds
[2025-04-16 22:55:33.898] [Wed Apr 16 22:55:35 2025] nvme nvme0:queue_size128 > ctrl sqsize 17, clamping down
[2025-04-16 22:55:33.929] [Wed Apr 16 22:55:35 2025] nvme nvme0: creating 3 I/O queues.
[2025-04-16 22:55:35.398] [Wed Apr 16 22:55:36 2025] nvme nvme0: mapped 3/0/0 default/read/poll queues.
[2025-04-16 22:55:35.445] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID0x1: traddr=10.10.10.223,trsvcid=4420,sport=0x8ac8,dport=0x1144
[2025-04-16 22:55:35.508] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID0x2: traddr=10.10.10.223,trsvcid=4420,sport=0x8ad4,dport=0x1144
[2025-04-16 22:55:35.554] [Wed Apr 16 22:55:36 2025] nvme nvme0: QID0x3: traddr=10.10.10.223,trsvcid=4420,sport=0x8ad8,dport=0x1144
[2025-04-16 22:55:35.632] [Wed Apr 16 22:55:37 2025] nvme nvme0: new ctrl: NQN "nqn.2015-09.com.wdc:nvme.1", addr 10.10.10.223:4420, hostnqn: nqn.2014-08.org.nvmexpress:host.0 [2025-04-16 22:55:36.195] [Wed Apr 16 22:55:37 2025] nvme nvme0: Failed to get ANA log: 16386
[2025-04-16 22:55:43.023]
[2025-04-16 22:55:43.492]
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] ------------[ cut here ]------------
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 22 PID: 134878 at tcp.c:782nvme_tcp_recv_skb+0x115e/0x11c0[nvme_tcp]
My assumption is that this is a result ofWARN_ON(queue->request == req);
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Sagi,
Quoted 2 lines
However the log prints are WARN messages that does not exist in any of the patches.
Just found the issue with my patch integration. I had picked up Hannes' patch in email titled "[PATCH 2/3] nvme-tcp: sanitize request list handling" and dated Fri Match 7th. This had the 3 WARN_ON msgs.
I just found another patch-set with almost the same title dated March 27th. I'll integrate this one "PATCH 2/5] nvme-tcp: sanitize request list handling". This replaced those WARN_ON msgs w/ dev_err() after if-checks for the same conditions.
Quoted 7 lines
Quoted 4 lines
The patches I applied are these:
1: nvme-tcp: partial write tcp-q-wake-sender untested by Sagi 2: nvme-tcp: recv-side EAGAIN untested fix by Sagi 3: nvme-tcp: fix I/O stalls on congested socketsThe patch I suggested was a replacement for patch (3). Didn't you get conflicts?
No. But I reviewed it again and have undone this part of patch-3 in my setup.
- else if (unlikely(result < 0))
+ else if (unlikely(result < 0) && result != -EAGAIN)
Quoted 33 lines
Quoted 33 lines
4: nvme-tcp: sanitize request list handling
5: nvme-tcp: avoid inline sending when handling R2T PDUs[4] nvme-tcp: sanitize request list handling
Validate the request in
nvme_tcp_handle_r2t()to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.drivers/nvme/host/tcp.c +3
-0@@ -455,6 +455,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) } list_del(&req->entry); + init_llist_node(&req->lentry); return req; }@@ -562,6 +563,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue; nvme_req(rq)->ctrl = &ctrl->ctrl; nvme_req(rq)->cmd = &pdu->cmd; + init_llist_node(&req->lentry); + INIT_LIST_HEAD(&req->entry); return 0; }@@ -773,8 +776,12 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
Quoted 7 lines
drivers/nvme/host/tcp.c static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, · line 781
Quoted 5 lines
drivers/nvme/host/tcp.c +3
-0nvme_tcp_setup_h2c_data_pdu(req); + WARN_ON(queue->request == req); + WARN_ON(llist_on_list(&req->lentry)); + WARN_ON(!list_empty(&req->entry));These are not included in Hannes patch afaict.
They were, as I noted on above.
Quoted 4 lines
I'm assuming that these are the stack traces that are firing...
Can you please share the output of gdb:
l *nvme_tcp_recv_skb+0x115e
It points to the 3rd WARN_ON msg. I've replaced those warnings with the patch you suggested below.
(gdb) l *(nvme_tcp_recv_skb+0x115e)0x628e is in nvme_tcp_recv_skb (tcp.c:782).
777
778 nvme_tcp_setup_h2c_data_pdu(req);
779
780 WARN_ON(queue->request == req); 781 WARN_ON(llist_on_list(&req->lentry)); 782 WARN_ON(!list_empty(&req->entry)); 783 llist_add(&req->lentry, &queue->req_list); 784 if (list_empty(&queue->send_list))
785 queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
Quoted 27 lines
My assumption is that in your case,
queue->requestis still referenced with the current request when the r2t for it arrives, which is very much possible...So this means that the patch "nvme-tcp: sanitize request list handling" is wrong,
queue->requestmay still be referenced when its r2t arrives.I think this patch needs the following:
--drivers/nvme/host/tcp.c +1
-2@@ -767,8 +767,7 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, return -EPROTO; } - if (queue->request == req || - llist_on_list(&req->lentry) || + if (llist_on_list(&req->lentry) || !list_empty(&req->entry)) { dev_err(queue->ctrl->ctrl.device,"req %d unexpected r2t while processing request\n", --
Quoted 4 lines
Or in your case - remove the above:
--
WARN_ON(queue->request == req);
--
Made that change. Tested and seeing this. -71 is -EPROTO that patch-2 returns . Trying to narrow it down further. If you have any suggestion, let me know.
[Mon Apr 21 11:18:58 2025] nvme nvme0: req 7 unexpected r2t while processing request [Mon Apr 21 11:18:58 2025] nvme nvme0: receive failed: -71 [Mon Apr 21 11:18:58 2025] nvme nvme0: starting error recovery [Mon Apr 21 11:18:58 2025] nvme nvme0: req 11 unexpected r2t while processing request [Mon Apr 21 11:18:58 2025] block nvme0n1: no usable path - requeuing I/O [Mon Apr 21 11:18:58 2025] nvme nvme0: receive failed: -71
Quoted 4 lines
Quoted 1 line
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 22 PID: 134878 at tcp.c:782
nvme_tcp_recv_skb+0x115e/0x11c0[nvme_tcp]My assumption is that this is a result of
WARN_ON(queue->request == req);
From the gdb decode it points to WARN_ON(!list_empty(&req->entry))
Christoph Hellwig
maintainerRe: [PATCH 1/3] nvme-tcp: open-code nvme_tcp_queue_request() for R2T
In reply to Hannes ReineckeQuoted 3 lines · Hannes Reinecke
When handling an R2T PDU we short-circuit
nvme_tcp_queue_request()as we should not attempt to send consecutive PDUs. So open-codenvme_tcp_queue_request()for R2T and drop the last argument.
I've added this patch to nvme-6.16. I'm waiting for the discussion to conclude or a resend on the others.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Kamaljit SinghQuoted 75 lines · Kamaljit Singh
Sagi,
Quoted 2 lines
However the log prints are WARN messages that does not exist in any of the patches.
Just found the issue with my patch integration. I had picked up Hannes' patch in email titled "[PATCH 2/3] nvme-tcp: sanitize request list handling" and dated Fri Match 7th. This had the 3
WARN_ONmsgs.I just found another patch-set with almost the same title dated March 27th. I'll integrate this one "PATCH 2/5] nvme-tcp: sanitize request list handling". This replaced those
WARN_ONmsgs w/dev_err()after if-checks for the same conditions.Quoted 6 lines
Quoted 4 lines
The patches I applied are these:
1: nvme-tcp: partial write tcp-q-wake-sender untested by Sagi 2: nvme-tcp: recv-side EAGAIN untested fix by Sagi 3: nvme-tcp: fix I/O stalls on congested socketsThe patch I suggested was a replacement for patch (3). Didn't you get conflicts?
No. But I reviewed it again and have undone this part of patch-3 in my setup.
(quoted diff) +1
-1- else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN)Quoted 33 lines
Quoted 33 lines
4: nvme-tcp: sanitize request list handling
5: nvme-tcp: avoid inline sending when handling R2T PDUs
[4] nvme-tcp: sanitize request list handlingValidate the request in
nvme_tcp_handle_r2t()to ensure it's not part of any list, otherwise a malicious R2T PDU might inject a loop in request list processing.drivers/nvme/host/tcp.c +3
-0@@ -455,6 +455,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) } list_del(&req->entry); + init_llist_node(&req->lentry); return req; }@@ -562,6 +563,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue; nvme_req(rq)->ctrl = &ctrl->ctrl; nvme_req(rq)->cmd = &pdu->cmd; + init_llist_node(&req->lentry); + INIT_LIST_HEAD(&req->entry); return 0; }@@ -773,8 +776,12 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,Quoted 6 lines
drivers/nvme/host/tcp.c static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, · line 781
Quoted 5 lines
drivers/nvme/host/tcp.c +3
-0nvme_tcp_setup_h2c_data_pdu(req); + WARN_ON(queue->request == req); + WARN_ON(llist_on_list(&req->lentry)); + WARN_ON(!list_empty(&req->entry));These are not included in Hannes patch afaict.
They were, as I noted on above.
Quoted 4 lines
I'm assuming that these are the stack traces that are firing...
Can you please share the output of gdb:
l *nvme_tcp_recv_skb+0x115eIt points to the 3rd
WARN_ONmsg. I've replaced those warnings with the patch you suggested below.(gdb) l *(nvme_tcp_recv_skb+0x115e)
0x628eis innvme_tcp_recv_skb(tcp.c:782).
777
778nvme_tcp_setup_h2c_data_pdu(req);
779
Quoted 6 lines
780 WARN_ON(queue->request == req); 781 WARN_ON(llist_on_list(&req->lentry)); 782 WARN_ON(!list_empty(&req->entry)); 783 llist_add(&req->lentry, &queue->req_list); 784 if (list_empty(&queue->send_list))785
queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
That is strange - should not happen afaict. You did not change the code and recompile before checking this correct?
Quoted 41 lines
Quoted 26 lines
My assumption is that in your case,
queue->requestis still referenced with the current request when the r2t for it arrives, which is very much possible...So this means that the patch "nvme-tcp: sanitize request list handling" is wrong,
queue->requestmay still be referenced when its r2t arrives.I think this patch needs the following:
--drivers/nvme/host/tcp.c +1
-2@@ -767,8 +767,7 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, return -EPROTO; } - if (queue->request == req || - llist_on_list(&req->lentry) || + if (llist_on_list(&req->lentry) || !list_empty(&req->entry)) { dev_err(queue->ctrl->ctrl.device,"req %d unexpected r2t while processing request\n",
--Quoted 4 lines
Or in your case - remove the above:
--
WARN_ON(queue->request == req);
--Made that change. Tested and seeing this. -71 is
-EPROTOthat patch-2 returns . Trying to narrow it down further. If you have any suggestion, let me know.[Mon Apr 21 11:18:58 2025] nvme nvme0: req 7 unexpected r2t while processing request [Mon Apr 21 11:18:58 2025] nvme nvme0: receive failed: -71 [Mon Apr 21 11:18:58 2025] nvme nvme0: starting error recovery [Mon Apr 21 11:18:58 2025] nvme nvme0: req 11 unexpected r2t while processing request [Mon Apr 21 11:18:58 2025] block nvme0n1: no usable path - requeuing I/O [Mon Apr 21 11:18:58 2025] nvme nvme0: receive failed: -71
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 22 PID: 134878 at tcp.c:782
nvme_tcp_recv_skb+0x115e/0x11c0[nvme_tcp]My assumption is that this is a result of
WARN_ON(queue->request == req);From the gdb decode it points to WARN_ON(!list_empty(&req->entry))
Can you please apply this on top:
--
drivers/nvme/host/tcp.c +0 -0
@@ -453,7 +453,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)
return NULL;
}
- list_del(&req->entry);
+ list_del_init(&req->entry);
init_llist_node(&req->lentry);
return req;
}
--
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Sagi, Hannes,
Quoted 16 lines
Quoted 9 lines
Quoted 2 lines
Can you please share the output of gdb:
l *nvme_tcp_recv_skb+0x115eIt points to the 3rd
WARN_ONmsg. I've replaced those warnings with the patch you suggested below.(gdb) l *(nvme_tcp_recv_skb+0x115e)
0x628eis innvme_tcp_recv_skb(tcp.c:782).
777
778nvme_tcp_setup_h2c_data_pdu(req);
779Quoted 6 lines
780 WARN_ON(queue->request == req); 781 WARN_ON(llist_on_list(&req->lentry)); 782 WARN_ON(!list_empty(&req->entry)); 783 llist_add(&req->lentry, &queue->req_list); 784 if (list_empty(&queue->send_list))785
queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
Quoted 3 lines
That is strange - should not happen afaict. You did not change the code and recompile
before checking this correct?
No, no other code changes were done before checking above.
Quoted 26 lines
Quoted 26 lines
Quoted 19 lines
I think this patch needs the following:
--drivers/nvme/host/tcp.c +1
-2@@ -767,8 +767,7 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, return -EPROTO ; } -if (queue->request == req || -llist_on_list (&req->lentry) || +if (llist_on_list (&req->lentry) || !list_empty (&req->entry)) { dev_err(queue->ctrl->ctrl.device,"req %d unexpected r2t while processing request\n",
--Quoted 4 lines
Or in your case - remove the above:
--WARN_ON(queue->request == req);--
Made that change. Tested and seeing this. -71 is
-EPROTOthat patch-2 returns . Trying to narrow it down further. If you have any suggestion, let me know.
Quoted 6 lines
Quoted 6 lines
[Mon Apr 21 11:18:58 2025] nvme nvme0: req 7 unexpected r2t while processing request [Mon Apr 21 11:18:58 2025] nvme nvme0: receive failed: -71 [Mon Apr 21 11:18:58 2025] nvme nvme0: starting error recovery [Mon Apr 21 11:18:58 2025] nvme nvme0: req 11 unexpected r2t while processing request [Mon Apr 21 11:18:58 2025] block nvme0n1: no usable path - requeuing I/O [Mon Apr 21 11:18:58 2025] nvme nvme0: receive failed: -71
The last patch had failed in under a minute. I had removed all WARN_ONs by this time.
Quoted 12 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
[2025-04-16 22:56:24.758] [Wed Apr 16 22:56:26 2025] WARNING: CPU: 22 PID: 134878 at tcp.c:782
nvme_tcp_recv_skb+0x115e/0x11c0[nvme_tcp]My assumption is that this is a result of
WARN_ON(queue->request == req);From the gdb decode it points to WARN_ON(!list_empty(&req->entry))
Can you please apply this on top:
--drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)
Quoted 9 lines
drivers/nvme/host/tcp.c +1
-2@@ -453,7 +453,7 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)return NULL ; } -list_del (&req->entry); +list_del_init (&req->entry);init_llist_node (&req->lentry);return req; } --
list_del_init did the trick. Test ran successfully for >18 hours, while the original failure took <6 hours. Can one of you please merge this in? Which branch will this be targeted for?
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Sagi GrimbergQuoted 171 lines · Sagi Grimberg
Quoted 48 lines · Hannes Reinecke
Quoted 39 lines · Sagi Grimberg
Quoted 36 lines · Hannes Reinecke
Quoted 20 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue);drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1399
Quoted 10 lines
drivers/nvme/host/tcp.c +3
-1if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return; + if (nvme_tcp_queue_has_pending(queue)) + pending = true; + if (!pending || !queue->rd_enabled) return;The various '
try_send' function will return-EAGAINfor a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.Quoted 6 lines
Quoted 2 lines
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it.
Umm, that is my understanding. If you tried to send X and were able to
send Y where Y < X, you shouldn't have to keep trying in a busy loop, the stack should tell you when you can send again.Okay, I could try that.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It would
still be an optimisation as we're saving the round-trip via socket callbacks.But you are doing a busy loop on a socket that cannot accept new data, there are other sockets that the kthread can be working on.
But we might be _sending_ data, right?
I'd say that odds are you're not in the next attempt...
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.I still don't understand why a partial send warrants a busy loop call to
sock_sendmsg...This is, it's not just sendmsg. It's the combination of send and recv.
I agree with you that the recv fix is correct.
Quoted 3 lines
In my tests I have seen sendmsg return with a partial/incomplete status, consequently recvmsg has nothing to receive, and
io_workstops. And that is what the patch fixes.The question is if the recv fix is sufficient?
Quoted 4 lines
I haven't checked the ->write_space() callback (which should've been triggered), but my feeling is that the ->write_space() callback hit when we were still busy processing, so the
queue_work()went
nowhere.That makes sense.
Quoted 6 lines
Maybe we can fix it with setting a flag when ->write_space() triggers ('hey, more data to process'), to be evaluated during the
io_work()loop. But that would be pretty close to the check 'nvme_tcp_queue_has_pending', so I'm not sure if we gain anything.How about checking
sk_stream_is_writeable()instead?
I think we also need to flag the queue for thewrite_spaceduring IO work...Quoted 4 lines
And I really haven't seen any detrimental performance effects with this patch; in the contrary, performance was on par, and if anything standard deviation went down.
Still you agree that busy looping a socket that may not have space is less efficient? When there are other queues waiting to execute
io_work?I agree there is a problem here, but I am trying to figure out what we want to do here.
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --gita/drivers/nvme/host/tcp.cb/drivers/nvme/host/tcp.cindex72d260201d8c..4eb9a2dec07e100644drivers/nvme/host/tcp.c +1
-1@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe = 0; consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk); - return consumed; + return consumed == -EAGAIN ? 0 : consumed; }static void
nvme_tcp_io_work(struct work_struct *w)Signature · 1 line
[2 based on your partial write fix]:drivers/nvme/host/tcp.c +2
-0@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_IO_CPU_SET = 3, + NVME_TCP_Q_WAKE_SENDER = 4, }; enum nvme_tcp_recv_state {@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data; if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue- >io_work); }
read_unlock_bh(&sk->sk_callback_lock);drivers/nvme/host/tcp.c +10
-1@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1); + clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); do { bool pending = false; int result;@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0)) return; - if (!pending || !queue->rd_enabled) + /* did we get some space after spending time in recv ? */ + if (nvme_tcp_queue_has_pending(queue) && + sk_stream_is_writeable(queue->sock->sk)) + pending = true; + + if (!queue->rd_enabled) + return; + + if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags)) return;
Quoted 3 lines
} while (!time_after(jiffies, deadline)); /* quota is exhausted */
While trying to sift through the patches trying to come up with
a next version:
Why do you check for Q_WAKE_SENDER after checking for sk_stream_is_writable?
Surely sk_stream_is_writable() is a necessary condition for nvme_tcp_try_send(), no?
Maybe it's a better idea to check sk_stream_is_writable() just before nvme_tcp_try_send()...
Let me see how things pan out.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 174 lines · Hannes Reinecke
Quoted 173 lines · Sagi Grimberg
Quoted 48 lines · Hannes Reinecke
Quoted 39 lines · Sagi Grimberg
Quoted 36 lines · Hannes Reinecke
Quoted 20 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue);drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1399
Quoted 10 lines
drivers/nvme/host/tcp.c +3
-1if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return; + if (nvme_tcp_queue_has_pending(queue)) + pending = true; + if (!pending || !queue->rd_enabled) return;The various '
try_send' function will return-EAGAINfor a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.Quoted 6 lines
Quoted 2 lines
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it.
Umm, that is my understanding. If you tried to send X and were able to
send Y where Y < X, you shouldn't have to keep trying in a busy loop, the stack should tell you when you can send again.Okay, I could try that.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It would
still be an optimisation as we're saving the round-trip via socket callbacks.But you are doing a busy loop on a socket that cannot accept new data, there are other sockets that the kthread can be working on.
But we might be _sending_ data, right?
I'd say that odds are you're not in the next attempt...
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.I still don't understand why a partial send warrants a busy loop call to
sock_sendmsg...This is, it's not just sendmsg. It's the combination of send and recv.
I agree with you that the recv fix is correct.
Quoted 3 lines
In my tests I have seen sendmsg return with a partial/incomplete status, consequently recvmsg has nothing to receive, and
io_workstops. And that is what the patch fixes.The question is if the recv fix is sufficient?
Quoted 4 lines
I haven't checked the ->write_space() callback (which should've been triggered), but my feeling is that the ->write_space() callback hit when we were still busy processing, so the
queue_work()went
nowhere.That makes sense.
Quoted 6 lines
Maybe we can fix it with setting a flag when ->write_space() triggers ('hey, more data to process'), to be evaluated during the
io_work()loop. But that would be pretty close to the check 'nvme_tcp_queue_has_pending', so I'm not sure if we gain anything.How about checking
sk_stream_is_writeable()instead?
I think we also need to flag the queue for thewrite_spaceduring IO work...Quoted 4 lines
And I really haven't seen any detrimental performance effects with this patch; in the contrary, performance was on par, and if anything standard deviation went down.
Still you agree that busy looping a socket that may not have space is less efficient? When there are other queues waiting to execute
io_work?I agree there is a problem here, but I am trying to figure out what we want to do here.
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --gita/drivers/nvme/host/tcp.cb/drivers/nvme/host/tcp.cindex72d260201d8c..4eb9a2dec07e100644drivers/nvme/host/tcp.c +1
-1@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe = 0; consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk); - return consumed; + return consumed == -EAGAIN ? 0 : consumed; }static void
nvme_tcp_io_work(struct work_struct *w)Signature · 1 line
[2 based on your partial write fix]:drivers/nvme/host/tcp.c +3
-0@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_IO_CPU_SET = 3, + NVME_TCP_Q_WAKE_SENDER = 4, }; enum nvme_tcp_recv_state {@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data; if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue- >io_work); } read_unlock_bh(&sk->sk_callback_lock);@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1); + clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); do { bool pending = false;int result;
drivers/nvme/host/tcp.c +1
-1@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0)) return; - if (!pending || !queue->rd_enabled) + /* did we get some space after spending time in recv? */
+ if (nvme_tcp_queue_has_pending(queue)&&
+sk_stream_is_writeable(queue->sock->sk))
+ pending = true;drivers/nvme/host/tcp.c +5
-0+ + if (!queue->rd_enabled) + return; + + if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER,
&queue->flags))
return;
Quoted 8 lines
Quoted 3 lines
} while (!time_after(jiffies, deadline)); /* quota is exhausted */
While trying to sift through the patches trying to come up with
a next version:
Why do you check forQ_WAKE_SENDERafter checking forsk_stream_is_writable?
Surelysk_stream_is_writable()is a necessary condition fornvme_tcp_try_send(), no?
Right after the check sk_stream_is_writeable(), there could have new room in the socket send buffer, state_change() fired, and queued io_work, before we get to the pending check. This is the "lost event" problem.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 186 lines · Sagi Grimberg
Quoted 174 lines · Hannes Reinecke
Quoted 173 lines · Sagi Grimberg
Quoted 48 lines · Hannes Reinecke
Quoted 39 lines · Sagi Grimberg
Quoted 36 lines · Hannes Reinecke
Quoted 20 lines · Hannes Reinecke
When the socket is busy processing
nvme_tcp_try_recv()might return-EAGAIN, but this doesn't automatically imply that the sending side is blocked, too.
So check if there are pending requests oncenvme_tcp_try_recv()returns-EAGAINand continue with the sending loop to avoid I/O stalls.Acked-by Chris Leech <[email protected]>Signed-off-by Hannes Reinecke <[email protected]>---
drivers/nvme/host/tcp.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0@@ -1389,9 +1389,12 @@ static void nvme_tcp_io_work(struct work_struct *w) result = nvme_tcp_try_recv(queue);drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct · line 1399
Quoted 10 lines
drivers/nvme/host/tcp.c +3
-1if (result > 0) pending = true; - else if (unlikely(result < 0)) + else if (unlikely(result < 0) && result != -EAGAIN) return; + if (nvme_tcp_queue_has_pending(queue)) + pending = true; + if (!pending || !queue->rd_enabled) return;The various '
try_send' function will return-EAGAINfor a partial send. But it doesn't indicate a blocked Tx, rather we should retry directly. Hence this check.Quoted 6 lines
Quoted 2 lines
Unless you tell me differently and even a partial send will cause ->write_space() to be invoked, then we wouldn't _need_ it.
Umm, that is my understanding. If you tried to send X and were able to
send Y where Y < X, you shouldn't have to keep trying in a busy loop, the stack should tell you when you can send again.Okay, I could try that.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
It would
still be an optimisation as we're saving the round-trip via socket callbacks.But you are doing a busy loop on a socket that cannot accept new data, there are other sockets that the kthread can be working on.
But we might be _sending_ data, right?
I'd say that odds are you're not in the next attempt...
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
We could aim for a different error here, to differentiate between a 'real' EAGAIN and a partial send.
Whatever you prefer.I still don't understand why a partial send warrants a busy loop call to
sock_sendmsg...This is, it's not just sendmsg. It's the combination of send and recv.
I agree with you that the recv fix is correct.
Quoted 3 lines
In my tests I have seen sendmsg return with a partial/incomplete status, consequently recvmsg has nothing to receive, and
io_workstops. And that is what the patch fixes.The question is if the recv fix is sufficient?
Quoted 4 lines
I haven't checked the ->write_space() callback (which should've been triggered), but my feeling is that the ->write_space() callback hit when we were still busy processing, so the
queue_work()went
nowhere.That makes sense.
Quoted 6 lines
Maybe we can fix it with setting a flag when ->write_space() triggers ('hey, more data to process'), to be evaluated during the
io_work()loop. But that would be pretty close to the check 'nvme_tcp_queue_has_pending', so I'm not sure if we gain anything.How about checking
sk_stream_is_writeable()instead?
I think we also need to flag the queue for thewrite_spaceduring IO work...Quoted 4 lines
And I really haven't seen any detrimental performance effects with this patch; in the contrary, performance was on par, and if anything standard deviation went down.
Still you agree that busy looping a socket that may not have space is less efficient? When there are other queues waiting to execute
io_work?I agree there is a problem here, but I am trying to figure out what we want to do here.
How about these two (untested) patches:
[1 based on your recv-side fix]:
--diff --gita/drivers/nvme/host/tcp.cb/drivers/nvme/host/tcp.cindex72d260201d8c..4eb9a2dec07e100644drivers/nvme/host/tcp.c +1
-1@@ -1348,7 +1348,7 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue) queue->nr_cqe = 0; consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb); release_sock(sk); - return consumed; + return consumed == -EAGAIN ? 0 : consumed; }static void
nvme_tcp_io_work(struct work_struct *w)Signature · 1 line
[2 based on your partial write fix]:drivers/nvme/host/tcp.c +3
-0@@ -129,6 +129,7 @@ enum nvme_tcp_queue_flags { NVME_TCP_Q_LIVE = 1, NVME_TCP_Q_POLLING = 2, NVME_TCP_Q_IO_CPU_SET = 3, + NVME_TCP_Q_WAKE_SENDER = 4, }; enum nvme_tcp_recv_state {@@ -1063,6 +1064,7 @@ static void nvme_tcp_write_space(struct sock *sk) queue = sk->sk_user_data; if (likely(queue && sk_stream_is_writeable(sk))) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); + set_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue- >io_work); } read_unlock_bh(&sk->sk_callback_lock);@@ -1357,6 +1359,7 @@ static void nvme_tcp_io_work(struct work_struct *w) container_of(w, struct nvme_tcp_queue, io_work); unsigned long deadline = jiffies + msecs_to_jiffies(1); + clear_bit(NVME_TCP_Q_WAKE_SENDER, &queue->flags); do { bool pending = false;int result;
drivers/nvme/host/tcp.c +1
-1@@ -1376,7 +1379,15 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0)) return; - if (!pending || !queue->rd_enabled) + /* did we get some space after spending time in recv? */
+ if (nvme_tcp_queue_has_pending(queue)&&
+sk_stream_is_writeable(queue->sock->sk))
+ pending = true;drivers/nvme/host/tcp.c +5
-0+ + if (!queue->rd_enabled) + return; + + if (!pending && !test_bit(NVME_TCP_Q_WAKE_SENDER,
&queue->flags))
return;Quoted 8 lines
Quoted 3 lines
} while (!time_after(jiffies, deadline)); /* quota is exhausted */
While trying to sift through the patches trying to come up with
a next version:
Why do you check forQ_WAKE_SENDERafter checking forsk_stream_is_writable?
Surelysk_stream_is_writable()is a necessary condition fornvme_tcp_try_send(), no?
Quoted 3 lines
Right after the check
sk_stream_is_writeable(), there could have new room in the socket send buffer,state_change()fired, and queuedio_work, before we get to the pending check. This is the "lost event" problem.
Actually, this is incorrect. write_space() -> queue_work(io_work) will be re-queued if io_work is currently running. Hence I'm not sure that this flag is needed.
Kamaljit, can you run with the below and report if it fixes your problem? --
drivers/nvme/host/tcp.c +0 -0
@@ -1390,6 +1390,11 @@ static void nvme_tcp_io_work(struct work_struct *w)
else if (unlikely(result < 0))
return;
+ /* did we get some space after spending time in recv? */ + if (nvme_tcp_queue_has_pending(queue) &&
+ sk_stream_is_writeable(queue->sock->sk)) + pending = true;
+
if (!pending || !queue->rd_enabled)
return;
--
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Sagi,
Quoted 7 lines
Quoted 6 lines
Right after the check
sk_stream_is_writeable(), there could have new room in the socket
send buffer,state_change()fired, and queuedio_work, before we get to the pending
check. This is the "lost event" problem.
Quoted 5 lines
Actually, this is incorrect.
write_space()->queue_work(io_work)will be re-queued ifio_work
is currently running. Hence I'm not sure that this flag is needed.Kamaljit, can you run with the below and report if it fixes your problem?
With the last set of fixes that included list_del_init() you suggested, all tests have been stable. I'm not seeing any more issues. Are you asking to test this patch regardless? If so, does this need to be in addition to previous patches?
Quoted 7 lines
--
drivers/nvme/host/tcp.c +0
-0@@ -1390,6 +1390,11 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0))
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1395
Quoted 10 lines
drivers/nvme/host/tcp.c +5
-1return; + /* did we get some space after spending time in recv? */ + if (nvme_tcp_queue_has_pending(queue) && + sk_stream_is_writeable(queue->sock->sk)) + pending = true; + if (!pending || !queue->rd_enabled) return; --
Thanks,
Kamaljit
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 11 lines · Kamaljit Singh
Sagi,
Quoted 9 lines
Quoted 5 lines
Right after the check
sk_stream_is_writeable(), there could have new room in the socket
send buffer,state_change()fired, and queuedio_work, before we get to the pending
check. This is the "lost event" problem.Actually, this is incorrect.
write_space()->queue_work(io_work)will be re-queued ifio_work
is currently running. Hence I'm not sure that this flag is needed.
Quoted 5 lines
Quoted 1 line
Kamaljit, can you run with the below and report if it fixes your problem?
With the last set of fixes that included
list_del_init()you suggested, all tests have been stable. I'm not seeing any more issues. Are you asking to test this patch regardless? If so, does this need to be in addition to previous patches?
Instead of my patch that introduces WAKE_SENDER - apply just the change below
that looks at sk_stream_is_writeable() only. I no longer think that the other changes in that patch solve anything.
Quoted 21 lines
Quoted 9 lines
--
drivers/nvme/host/tcp.c +0
-0@@ -1390,6 +1390,11 @@ static void nvme_tcp_io_work(struct work_struct *w) else if (unlikely(result < 0))drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1395
Quoted 10 lines
drivers/nvme/host/tcp.c +5
-1return; + /* did we get some space after spending time in recv? */ + if (nvme_tcp_queue_has_pending(queue) && + sk_stream_is_writeable(queue->sock->sk)) + pending = true; + if (!pending || !queue->rd_enabled) return; --Thanks,
Kamaljit
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to a message not in the archiveHi Kamaljit,
Quoted 20 lines · Kamaljit Singh
Hi Sagi,
Quoted 8 lines · Kamaljit Singh
Quoted 7 lines
Sagi,
Quoted 5 lines
Quoted 5 lines
Right after the check
sk_stream_is_writeable(), there could have new room in the socket
send buffer,state_change()fired, and queuedio_work, before we get to the pending
check. This is the "lost event" problem.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Actually, this is incorrect.
write_space()->queue_work(io_work)will be re-queued ifio_work
is currently running. Hence I'm not sure that this flag is needed.Kamaljit, can you run with the below and report if it fixes your
problem?
Quoted 1 line
Quoted 1 line
With the last set of fixes that included
list_del_init()yousuggested, all
Quoted 1 line
Quoted 1 line
tests have been stable. I'm not seeing any more issues. Are you
asking to
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
test this patch regardless? If so, does this need to be in addition to previous patches?
Instead of my patch that introduces
WAKE_SENDER- apply just the change below
that looks atsk_stream_is_writeable()only. I no longer think that the other changes in that patch solve anything.Sorry, I was away on PTO. Working now to get the below patch tested.
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ? (on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here,
but we need to have confirmation.
Thanks a bunch.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Hi Hannes,
Quoted 5 lines · Hannes Reinecke
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ?
(on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here, but we need to have confirmation.
I've built the kernel against the latest of nvme-6.16 branch along with these patches. Its with test now. Will let you know when we get some results back.
0. [PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets - Apr 29, 01:18 Hannes Reinecke (2):
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets
1. [PATCH 1/2] nvme-tcp: sanitize request list handling - Apr 29, 02:31
2. [PATCH 2/2] nvme-tcp: fix I/O stalls on congested sockets - Apr 29, 01:18
Meanwhile, quick question, is there a known issue in cloning from this repo? I'm seeing super slow response, which eventually timesout. Had to try multiple times over several hours.
$ git clone git://git.infradead.org/nvme.git linux-nvme-6.16 Cloning into 'linux-nvme-6.16'... remote: Enumerating objects: 10795230, done. remote: Counting objects: 100% (333577/333577), done. remote: Compressing objects: 100% (99866/99866), done. fatal: read error: Operation timed out 76.01 KiB | 1024 bytes/s fatal: early EOF fatal: fetch-pack: invalid index-pack output
Even a git pull --rebase into an older clone was failing to refresh remotes & show the nvme-6.16 branch.
Thanks,
Kamaljit Singh
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Hi Hannes, Sagi,
Quoted 10 lines · Kamaljit Singh
Quoted 5 lines
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ?
(on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here, but we need to have confirmation.I've built the kernel against the latest of nvme-6.16 branch along with these patches. Its with test now. Will let you know when we get some results back.
Quoted 7 lines
0. [PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets - Apr 29, 01:18 Hannes Reinecke (2):
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets1. [PATCH 1/2] nvme-tcp: sanitize request list handling - Apr 29, 02:31
2. [PATCH 2/2] nvme-tcp: fix I/O stalls on congested sockets - Apr 29, 01:18
IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode 0x2 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bd nvme-tcp: sanitize request list handling 9260acd6c230 nvme-tcp: fix I/O stalls on congested sockets
Thanks,
Kamaljit Singh
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 57 lines · Kamaljit Singh
Hi Hannes, Sagi,
Quoted 10 lines · Kamaljit Singh
Quoted 5 lines
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ?
(on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here, but we need to have confirmation.I've built the kernel against the latest of nvme-6.16 branch along with these patches. Its with test now. Will let you know when we get some results back.
Quoted 7 lines
0. [PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets - Apr 29, 01:18 Hannes Reinecke (2):
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets1. [PATCH 1/2] nvme-tcp: sanitize request list handling - Apr 29, 02:31
2. [PATCH 2/2] nvme-tcp: fix I/O stalls on congested sockets - Apr 29, 01:18IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode 0x2 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...
Quoted 4 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested sockets
Extremely wild guess: Can you try with this patch on top?
drivers/nvme/host/tcp.c +1 -4
@@ -1277,10 +1277,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req) .iov_len = NVME_TCP_DIGEST_LENGTH - req->offset }; -if (nvme_tcp_queue_more (queue)) - msg.msg_flags |=MSG_MORE ; -else - msg.msg_flags |=MSG_EOR ; + msg.msg_flags |=MSG_EOR ; ret =kernel_sendmsg (queue->sock, &msg, &iov,1 , iov.iov_len);if (unlikely (ret <=0 ))
It _could_ be that we're waiting in sendmsg() due to MSG_MORE, causing these I/O timeouts as processing doesn't continue.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 82 lines · Hannes Reinecke
Quoted 57 lines · Kamaljit Singh
Hi Hannes, Sagi,
Quoted 10 lines · Kamaljit Singh
Quoted 5 lines
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ?
(on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here, but we need to have confirmation.I've built the kernel against the latest of nvme-6.16 branch along with these patches. Its with test now. Will let you know when we get some results back.
Quoted 7 lines
0. [PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets - Apr 29, 01:18 Hannes Reinecke (2):
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets1. [PATCH 1/2] nvme-tcp: sanitize request list handling - Apr 29, 02:31
2. [PATCH 2/2] nvme-tcp: fix I/O stalls on congested sockets - Apr 29, 01:18IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode
0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode0x2(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...Quoted 4 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested socketsExtremely wild guess: Can you try with this patch on top?
drivers/nvme/host/tcp.c +1
-4@@ -1277,10 +1277,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req) .iov_len = NVME_TCP_DIGEST_LENGTH - req->offset }; - if (nvme_tcp_queue_more(queue)) - msg.msg_flags |= MSG_MORE; - else - msg.msg_flags |= MSG_EOR; + msg.msg_flags |= MSG_EOR; ret = kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len);if (unlikely(ret <= 0))
Quoted 3 lines
It _could_ be that we're waiting in
sendmsg()due toMSG_MORE, causing these I/O timeouts as processing doesn't continue.
After a bit more thought, I really do think this is a different issue. The original issue I've encountered was an I/O stall where the io_work thread would never be restarted even though I/O was pending.
But we still have the problem that we will encounter I/O timeouts on oversubscribed links, where we shove too much work onto the same link. In that case we also will be seeing I/O timeouts (as the PDUs take too long to be sent), but this is not something my patchset does address.
And seeing that your error depends on ECN (which _will_ cause a delay on packet transmission) I tend to think it's actually a problem with oversubscription, not with io_work not being restarted.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Kamaljit SinghQuoted 56 lines · Kamaljit Singh
Hi Hannes, Sagi,
Quoted 9 lines · Kamaljit Singh
Quoted 5 lines
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ?
(on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here, but we need to have confirmation.I've built the kernel against the latest of nvme-6.16 branch along with these patches. Its with test now. Will let you know when we get some results back.
Quoted 7 lines
0. [PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets - Apr 29, 01:18 Hannes Reinecke (2):
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets1. [PATCH 1/2] nvme-tcp: sanitize request list handling - Apr 29, 02:31
2. [PATCH 2/2] nvme-tcp: fix I/O stalls on congested sockets - Apr 29, 01:18IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode 0x2 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode 0x1 (I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...
Quoted 3 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested sockets
Kamaljit, with the prior version of the patchset (the proposal with the wake_sender flag) did this not reproduce regardless of ECN?
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Hannes ReineckeQuoted 79 lines · Hannes Reinecke
Quoted 57 lines · Kamaljit Singh
Hi Hannes, Sagi,
Quoted 10 lines · Kamaljit Singh
Quoted 5 lines
Can you please retest with the patchset '[PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets' _only_ ?
(on top of nvme-6.16 latest, of course).
I think I _should_ have included all the suggestions floating here, but we need to have confirmation.I've built the kernel against the latest of nvme-6.16 branch along with these patches. Its with test now. Will let you know when we get some results back.
Quoted 7 lines
0. [PATCHv5 0/2] nvme-tcp: fixup I/O stall on congested sockets - Apr 29, 01:18 Hannes Reinecke (2):
nvme-tcp: sanitize request list handling
nvme-tcp: fix I/O stalls on congested sockets1. [PATCH 1/2] nvme-tcp: sanitize request list handling - Apr 29, 02:31
2. [PATCH 2/2] nvme-tcp: fix I/O stalls on congested sockets - Apr 29, 01:18IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode
0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode0x2(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...Quoted 4 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested socketsExtremely wild guess: Can you try with this patch on top?
drivers/nvme/host/tcp.c +0
-0@@ -1277,10 +1277,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req).iov_len =
NVME_TCP_DIGEST_LENGTH-req->offset};- if (
nvme_tcp_queue_more(queue))
- msg.msg_flags |=MSG_MORE;
- else
- msg.msg_flags |=MSG_EOR;
+ msg.msg_flags |=MSG_EOR;
Quoted 5 lines
ret =
kernel_sendmsg(queue->sock, &msg, &iov, 1, iov.iov_len); if (unlikely(ret <= 0))It _could_ be that we're waiting in
sendmsg()due toMSG_MORE, causing these I/O timeouts as processing doesn't continue.
This change doesn't make sense to me...
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
In reply to Sagi GrimbergHi Sagi, Hannes,
Quoted 37 lines · Sagi Grimberg
Quoted 37 lines
IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode
0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode0x2(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...
Quoted 6 lines
Quoted 3 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested socketsKamaljit, with the prior version of the patchset (the proposal with the
wake_senderflag) did this not reproduce regardless of ECN?
With the last patchset, when ECN=off, we did not see any IO timeouts even with a weekend long test. This was true for both cases, i.e. with Inband Auth and with SecureConcat.
With ECN=on & HD+DD=on IO timeout still persists for both Inband Auth & SC. I’m currently debugging a possible target side issue with ECN. I’ll let you know once I have some resolution.
I don't have any clear indications of the original kernel issue to be able to differentiate against the current target-side issue. So, if you want to go ahead and merge those two patchsets that may be fine for now.
Thanks,
Kamaljit
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 51 lines · Kamaljit Singh
Hi Sagi, Hannes,
Quoted 37 lines · Sagi Grimberg
Quoted 37 lines
IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode
0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode0x2(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...Quoted 6 lines
Quoted 3 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested socketsKamaljit, with the prior version of the patchset (the proposal with the
wake_senderflag) did this not reproduce regardless of ECN?With the last patchset, when ECN=off, we did not see any IO timeouts even with a weekend long test. This was true for both cases, i.e. with Inband Auth and with SecureConcat.
Quoted 8 lines
With ECN=on & HD+DD=on IO timeout still persists for both Inband Auth & SC. I’m currently debugging a possible target side issue with ECN. I’ll let you know once I have some resolution.
I don't have any clear indications of the original kernel issue to be able to differentiate against the current target-side issue. So, if you want to go ahead and merge those two patchsets that may be fine for now.
Thanks a lot for your confirmation.
We continue to have issues with high load or oversubscribed fabrics. But this patchset is addressing the problem of I/O timeouts during _connect_, which I would argue is a different story.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 55 lines · Hannes Reinecke
Quoted 54 lines · Kamaljit Singh
Hi Sagi, Hannes,
Quoted 37 lines · Sagi Grimberg
Quoted 37 lines
IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode
0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode0x2(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...Quoted 6 lines
Quoted 3 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested socketsKamaljit, with the prior version of the patchset (the proposal with the
wake_senderflag) did this not reproduce regardless of ECN?With the last patchset, when ECN=off, we did not see any IO timeouts even with a weekend long test. This was true for both cases, i.e. with Inband Auth and with SecureConcat.
With ECN=on & HD+DD=on IO timeout still persists for both Inband Auth & SC. I’m currently debugging a possible target side issue with ECN. I’ll let you know once I have some resolution.
Quoted 8 lines
Quoted 4 lines
I don't have any clear indications of the original kernel issue to be able to differentiate against the current target-side issue. So, if you want to go ahead and merge those two patchsets that may be fine for now.
Thanks a lot for your confirmation.
We continue to have issues with high load or oversubscribed fabrics. But this patchset is addressing the problem of I/O timeouts during _connect_, which I would argue is a different story.
We still need to hunt these down. I'm still puzzled why adding the WAKE_SENDER flag was able to make this issue disappear? I'll have another look at this patch.
For now, I think we can go with this patchset, and then incrementally fix the remains.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 67 lines · Sagi Grimberg
Quoted 55 lines · Hannes Reinecke
Quoted 54 lines · Kamaljit Singh
Hi Sagi, Hannes,
Quoted 37 lines · Sagi Grimberg
Quoted 37 lines
IO timeouts are still occurring with Writes. The only Read that timed out was most likely due to the path error. It takes ~4.5 hours to fail.
However, this test does not fail if either ECN is off or if digests are not enabled. These passing combinations were run for 16+ hours without any issues. Both ECN and Header+Data Digests need to be turned on for it to fail.
Do you have a failing test as well? If so, is it quicker to cause the failure? Would you mind sharing any details?
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 2 (f002) type 4 opcode
0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 1 (2001) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 4 (c004) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: starting error recovery
[2025-05-07 19:57:13.295] nvme nvme1: I/O tag 15 (000f) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 6 (5006) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 3 (2003) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n3: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 8 (0008) type 4 opcode0x2(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 14 (400e) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 13 (100d) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n4: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n2: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 5 (5005) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 7 (0007) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 11 (a00b) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] nvme nvme1: I/O tag 12 (f00c) type 4 opcode0x1(I/O Cmd) QID 4 timeout [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] block nvme1n1: no usable path - requeuing I/O [2025-05-07 19:57:13.295] nvme nvme1: Reconnecting in 10 seconds...Quoted 6 lines
Quoted 3 lines
In the current build I had these patches on top of the "nvme-6.16" branch:
41b2c90a51bdnvme-tcp: sanitize request list handling9260acd6c230nvme-tcp: fix I/O stalls on congested socketsKamaljit, with the prior version of the patchset (the proposal with the
wake_senderflag) did this not reproduce regardless of ECN?With the last patchset, when ECN=off, we did not see any IO timeouts even with a weekend long test. This was true for both cases, i.e. with Inband Auth and with SecureConcat.
With ECN=on & HD+DD=on IO timeout still persists for both Inband Auth & SC. I’m currently debugging a possible target side issue with ECN. I’ll let you know once I have some resolution.
Quoted 8 lines
Quoted 4 lines
I don't have any clear indications of the original kernel issue to be able to differentiate against the current target-side issue. So, if you want to go ahead and merge those two patchsets that may be fine for now.
Thanks a lot for your confirmation.
We continue to have issues with high load or oversubscribed fabrics. But this patchset is addressing the problem of I/O timeouts during _connect_, which I would argue is a different story.
Quoted 4 lines
We still need to hunt these down. I'm still puzzled why adding the
WAKE_SENDERflag was able to make this issue disappear? I'll have another look at this patch.For now, I think we can go with this patchset, and then incrementally fix the remains.
Kamaljit, can you check the following patch on top of the patchset from Hannes that gets a reproduction?
--
drivers/nvme/host/tcp.c +0 -0
@@ -1075,7 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk)
read_lock_bh(&sk->sk_callback_lock);
queue = sk->sk_user_data;
- if (likely(queue && sk_stream_is_writeable(sk))) {
+ if (likely(queue)) {
clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); }
--
This think this may be preventing the scheduling of io_work. But now that io_work is also ceasing based on sk_stream_is_writeable, we should probably still schedule it.
Sagi Grimberg
maintainerRe: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 27 lines
Quoted 5 lines
We still need to hunt these down. I'm still puzzled why adding the
WAKE_SENDERflag was able to make this issue disappear? I'll have another look at this patch.For now, I think we can go with this patchset, and then incrementally fix the remains.
Kamaljit, can you check the following patch on top of the patchset from Hannes that gets a reproduction?
drivers/nvme/host/tcp.c +1
-1@@ -1075,7 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk) read_lock_bh(&sk->sk_callback_lock); queue = sk->sk_user_data; - if (likely(queue && sk_stream_is_writeable(sk))) { + if (likely(queue)) { clear_bit(SOCK_NOSPACE, &sk->sk_socket->flags); queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); }Signature · 2 lines
This think this may be preventing the scheduling of io_work. But now that io_work is also ceasing based on sk_stream_is_writeable, we should probably still schedule it.
Kamaljit? Any info on this?
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Hi Sagi,
Quoted 31 lines · Sagi Grimberg
Quoted 30 lines
Quoted 8 lines
We still need to hunt these down. I'm still puzzled why adding the
WAKE_SENDER
flag was able to make this issue disappear? I'll have another look at this patch.For now, I think we can go with this patchset, and then incrementally fix the remains.
Kamaljit, can you check the following patch on top of the patchset from Hannes that
gets a reproduction?--
drivers/nvme/host/tcp.c +1
-1@@ -1075,7 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk)read_lock_bh (&sk->sk_callback_lock); queue = sk->sk_user_data; -if (likely (queue &&sk_stream_is_writeable (sk))) { +if (likely (queue)) {clear_bit (SOCK_NOSPACE , &sk->sk_socket->flags);queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);}--
Quoted 6 lines
Quoted 4 lines
This think this may be preventing the scheduling of
io_work. But now thatio_workis
also ceasing based onsk_stream_is_writeable, we should probably still schedule it.Kamaljit? Any info on this?
Sorry for the delayed reply. We've been hitting one issue after another, hence the delays. Hoping that by tomorrow I should have clean results with a clear a-b comparison of only the changes you suggested.
Thanks,
Kamaljit
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Quoted 40 lines · Kamaljit Singh
Hi Sagi,
Quoted 31 lines · Sagi Grimberg
Quoted 30 lines
Quoted 8 lines
We still need to hunt these down. I'm still puzzled why adding the
WAKE_SENDER
flag was able to make this issue disappear? I'll have another look at this patch.For now, I think we can go with this patchset, and then incrementally fix the remains.
Kamaljit, can you check the following patch on top of the patchset from Hannes that
gets a reproduction?--
drivers/nvme/host/tcp.c +1
-1@@ -1075,7 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk)read_lock_bh (&sk->sk_callback_lock); queue = sk->sk_user_data; -if (likely (queue &&sk_stream_is_writeable (sk))) { +if (likely (queue)) {clear_bit (SOCK_NOSPACE , &sk->sk_socket->flags);queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);}--
Quoted 6 lines
Quoted 4 lines
This think this may be preventing the scheduling of
io_work. But now thatio_workis
also ceasing based onsk_stream_is_writeable, we should probably still schedule it.Kamaljit? Any info on this?
Quoted 4 lines
Sorry for the delayed reply. We've been hitting one issue after another, hence the delays. Hoping that by tomorrow I should have clean results with a clear a-b comparison of only the changes you suggested.
Just to make you aware, I've found another issue wrt sk_stream_is_writeable(); we are checking it from the data_ready callback, and do not schedule io_work if no write space is available.
Problem here is that the data_ready callback indicates that there
_should_ be space available, so by checking sk_stream_is_writeable()
we already assume that the actual write space might change, and
the callback might not be a reliable indicator.
But from that follows that also the opposite might be true, namely
that write space _might_ be available by the time io_work is scheduled.
I'll repost the series.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 3/3] nvme-tcp: fix I/O stalls on congested sockets
Hi Hannes & Sagi,
Quoted 38 lines · Hannes Reinecke
Quoted 38 lines
Quoted 37 lines · Sagi Grimberg
Quoted 28 lines
Quoted 8 lines
We still need to hunt these down. I'm still puzzled why adding the
WAKE_SENDER
flag was able to make this issue disappear? I'll have another look at this patch.For now, I think we can go with this patchset, and then incrementally fix the remains.
Kamaljit, can you check the following patch on top of the patchset from Hannes that
gets a reproduction?--
drivers/nvme/host/tcp.c +1
-1@@ -1075,7 +1075,7 @@ static void nvme_tcp_write_space(struct sock *sk)read_lock_bh (&sk->sk_callback_lock); queue = sk->sk_user_data; -if (likely (queue &&sk_stream_is_writeable (sk))) { +if (likely (queue)) {clear_bit (SOCK_NOSPACE , &sk->sk_socket->flags);queue_work_on (queue->io_cpu, nvme_tcp_wq, &queue->io_work);}Quoted 6 lines
--
This think this may be preventing the scheduling of
io_work. But now thatio_workis
also ceasing based onsk_stream_is_writeable, we should probably still schedule it.Kamaljit? Any info on this?
Quoted 3 lines
Quoted 3 lines
Sorry for the delayed reply. We've been hitting one issue after another, hence the delays. Hoping that by tomorrow I should have clean results with a clear a-b comparison of only the changes you suggested.
Sagi,
After several more runs, I still don't have any definitive results for the driver change you suggested. The tests did fail with IO timeouts but the rootcauses were clearly unrelated to the Kernel/driver.
Quoted 10 lines
Just to make you aware, I've found another issue wrt
sk_stream_is_writeable(); we are checking it from thedata_readycallback, and do not scheduleio_workif no write space is available. Problem here is that thedata_readycallback indicates that there _should_ be space available, so by checkingsk_stream_is_writeable()we already assume that the actual write space might change, and the callback might not be a reliable indicator.
But from that follows that also the opposite might be true, namely that write space _might_ be available by the timeio_workis scheduled. I'll repost the series.
Hannes, Thank you for the V4 patchset. Looks like V4 includes the last change that Sagi suggested, i.e. remove sk_stream_is_writeable(sk) from nvme_tcp_write_space. We will continue testing w/ your V4 patchset and provided any feedback.
Thanks,
Kamaljit