nvme-tcp: rerun io_work if req_list is not empty
Keith Busch
maintainerRFC PATCH 1/1 nvme-tcp: rerun io_work if req_list is not empty
A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by nvme_tcp_send_all() if it happens to be running. The driver relies on io_work to send the enqueued request when it is runs again, but the concurrently running nvme_tcp_send_all() may not have released the send_mutex at that time. If no future commands are enqueued to re-kick the io_work, the request will timeout in the SEND_H2C state, resulting in a timeout error like:
nvme nvme0: queue 1: timeout request 0x3 type 6
Ensure the io_work continues to run as long as the req_list is not empty.
---
Marking this RFC because the timeout is difficult to recreate, so difficult to verify the patch. The patch was created purely from code inspection, so I'm just hoping for feedback on my analysis right now.
drivers/nvme/host/tcp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +2 -1
@@ -1140,7 +1140,8 @@ static void nvme_tcp_io_work(struct work_struct *w) pending = true; else if (unlikely(result < 0)) break; - } + }else + pending = !llist_empty (&queue->req_list); result =nvme_tcp_try_recv (queue);if (result >0 )
Signature · 7 lines
2.25.4 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 10 lines
A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by
nvme_tcp_send_all()if it happens to be running. The driver relies onio_workto send the enqueued request when it is runs again, but the concurrently runningnvme_tcp_send_all()may not have released thesend_mutexat that time. If no future commands are enqueued to re-kick theio_work, the request will timeout in theSEND_H2Cstate, resulting in a timeout error like:
Quoted 4 lines
nvme nvme0: queue 1: timeout request 0x3 type 6Ensure the
io_workcontinues to run as long as thereq_listis not empty.
There is a version of this patch that I personally suggested before, however I couldn't explain why that should happen...
nvme_tcp_send_all tries to send everything it has queues, it means should either be able to send everything, or it should see a full socket buffer. But in case the socket buffer is full, there should be a .write_space() sk callback triggering when the socket buffer evacuates space... Maybe there is a chance that write_space triggered, started execution, and that the send_mutex is still taken?
Can we maybe try to catch if that is the case?
Quoted 15 lines
Signed-off-by Keith Busch <[email protected]>---
Marking this RFC because the timeout is difficult to recreate, so difficult to verify the patch. The patch was created purely from code inspection, so I'm just hoping for feedback on my analysis right now.drivers/nvme/host/tcp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1144
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-1@@ -1140,7 +1140,8 @@ static void nvme_tcp_io_work(struct work_struct *w) pending = true; else if (unlikely(result < 0)) break; - } + }else + pending = !llist_empty (&queue->req_list);
In my version, pending was unconditionally set to true...
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 17 lines · Sagi Grimberg
Quoted 9 lines
A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by
nvme_tcp_send_all()if it happens to be running. The driver relies onio_workto send the enqueued request when it is runs again, but the concurrently runningnvme_tcp_send_all()may not have released thesend_mutexat that time. If no future commands are enqueued to re-kick theio_work, the request will timeout in theSEND_H2Cstate, resulting in a timeout error like:Quoted 4 lines
nvme nvme0: queue 1: timeout request 0x3 type 6Ensure the
io_workcontinues to run as long as thereq_listis not empty.There is a version of this patch that I personally suggested before, however I couldn't explain why that should happen...
Quoted 6 lines
nvme_tcp_send_alltries to send everything it has queues, it means should either be able to send everything, or it should see a full socket buffer. But in case the socket buffer is full, there should be a .write_space() sk callback triggering when the socket buffer evacuates space... Maybe there is a chance thatwrite_spacetriggered, started execution, and that thesend_mutexis still taken?
nvme_tcp_send_all() breaks out of the loop if nvme_tcp_fetch_request() returns NULL. If that happens just before io_work calls nvme_tcp_handle_r2t() to enqueue the H2C request, nvme_tcp_send_all() will not see that request, but send_mutex is still held. We're counting on io_work to run again to handle sending the H2C data in that case. Unlikely as it sounds, if the same nvme_tcp_send_all() context is still holding the send_mutex when io_work gets back to trying to take it, how will the data get sent?
Quoted 1 line
Can we maybe try to catch if that is the case?
Do you have a better idea on how we can catch this? I think there was only one occurance of this sighting so far, and it looks like it took a long time to encounter it, but we will try again if you have a proposal.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 25 lines
Quoted 18 lines
Quoted 10 lines
A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by
nvme_tcp_send_all()if it happens to be running. The driver relies onio_workto send the enqueued request when it is runs again, but the concurrently runningnvme_tcp_send_all()may not have released thesend_mutexat that time. If no future commands are enqueued to re-kick theio_work, the request will timeout in theSEND_H2Cstate, resulting in a timeout error like:Quoted 4 lines
nvme nvme0: queue 1: timeout request 0x3 type 6Ensure the
io_workcontinues to run as long as thereq_listis not empty.There is a version of this patch that I personally suggested before, however I couldn't explain why that should happen...
Quoted 6 lines
nvme_tcp_send_alltries to send everything it has queues, it means should either be able to send everything, or it should see a full socket buffer. But in case the socket buffer is full, there should be a .write_space() sk callback triggering when the socket buffer evacuates space... Maybe there is a chance thatwrite_spacetriggered, started execution, and that thesend_mutexis still taken?
Quoted 8 lines
nvme_tcp_send_all()breaks out of the loop ifnvme_tcp_fetch_request()returns NULL. If that happens just beforeio_workcallsnvme_tcp_handle_r2t()to enqueue the H2C request,nvme_tcp_send_all()will not see that request, butsend_mutexis still held. We're counting onio_workto run again to handle sending the H2C data in that case. Unlikely as it sounds, if the samenvme_tcp_send_all()context is still holding thesend_mutexwhenio_workgets back to trying to take it, how will the data get sent?
Yes you are correct, overlooked this race. I guess this is all coming from having less queues than cores (where rx really competes with tx) which is not as common as a non default.
This is enough to convince me that this is needed:
Quoted 5 lines
Quoted 1 line
Can we maybe try to catch if that is the case?
Do you have a better idea on how we can catch this? I think there was only one occurance of this sighting so far, and it looks like it took a long time to encounter it, but we will try again if you have a proposal.
We can continue to test with the patch and hunt for another occurance, given the argument above, this patch is needed regardless...
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 9 lines · Sagi Grimberg
Quoted 8 lines
nvme_tcp_send_all()breaks out of the loop ifnvme_tcp_fetch_request()returns NULL. If that happens just beforeio_workcallsnvme_tcp_handle_r2t()to enqueue the H2C request,nvme_tcp_send_all()will not see that request, butsend_mutexis still held. We're counting onio_workto run again to handle sending the H2C data in that case. Unlikely as it sounds, if the samenvme_tcp_send_all()context is still holding thesend_mutexwhenio_workgets back to trying to take it, how will the data get sent?
Quoted 6 lines
Yes you are correct, overlooked this race. I guess this is all coming from having less queues than cores (where rx really competes with tx) which is not as common as a non default.
This is enough to convince me that this is needed:
Reviewed-by Sagi Grimberg <[email protected]>
Great! I thought the scenario seemed possible, but wasn't completely sure, so thank you for confirming.
Christoph, can we pick this up for the next rc? For stable, we can add
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Can we maybe try to catch if that is the case?
Do you have a better idea on how we can catch this? I think there was only one occurance of this sighting so far, and it looks like it took a long time to encounter it, but we will try again if you have a proposal.
We can continue to test with the patch and hunt for another occurance, given the argument above, this patch is needed regardless...
Sounds good, we'll run with the patch and see what happens. If the tests are successful, I'm not sure if we can conclude this definitely fixes the timeout or if we just got lucky. If a timeout is observed, though, I will try to work in a debug patch.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Thanks,
applied to nvme-5.13 with the fixes tag added.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme