Skip to contentSkip to conversation

nvme-tcp: rerun io_work if req_list is not empty

RFC PATCH Applied Keith Busch 3 participants 6 messages May 17, 2021 – May 19, 2021 NVMe

lore
Share & tools
lore

Keith Busch

author maintainer
#0 lore raw

RFC PATCH 1/1 nvme-tcp: rerun io_work if req_list is not empty

A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by nvme_tcp_send_all() if it happens to be running. The driver relies on io_work to send the enqueued request when it is runs again, but the concurrently running nvme_tcp_send_all() may not have released the send_mutex at that time. If no future commands are enqueued to re-kick the io_work, the request will timeout in the SEND_H2C state, resulting in a timeout error like:

nvme nvme0: queue 1: timeout request 0x3 type 6

Ensure the io_work continues to run as long as the req_list is not empty.

---
Marking this RFC because the timeout is difficult to recreate, so difficult to verify the patch. The patch was created purely from code inspection, so I'm just hoping for feedback on my analysis right now.

 drivers/nvme/host/tcp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +2 -1
index 0222e23f5936..d07eb13d8713 100644
@@ -1140,7 +1140,8 @@ static void nvme_tcp_io_work(struct work_struct *w)
 				pending = true;
 			else if (unlikely(result < 0))
 				break;
-		}
+		} else
+			pending = !llist_empty(&queue->req_list);
 
 		result = nvme_tcp_try_recv(queue);
 		if (result > 0)
Signature · 7 lines
2.25.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#1 lore raw
Quoted 10 lines

A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by nvme_tcp_send_all() if it happens to be running. The driver relies on io_work to send the enqueued request when it is runs again, but the concurrently running nvme_tcp_send_all() may not have released the send_mutex at that time. If no future commands are enqueued to re-kick the io_work, the request will timeout in the SEND_H2C state, resulting in a timeout error like:

Quoted 4 lines
   nvme nvme0: queue 1: timeout request 0x3 type 6

Ensure the io_work continues to run as long as the req_list is not empty.

There is a version of this patch that I personally suggested before, however I couldn't explain why that should happen...

nvme_tcp_send_all tries to send everything it has queues, it means should either be able to send everything, or it should see a full socket buffer. But in case the socket buffer is full, there should be a .write_space() sk callback triggering when the socket buffer evacuates space... Maybe there is a chance that write_space triggered, started execution, and that the send_mutex is still taken?

Can we maybe try to catch if that is the case?

Quoted 15 lines

---
Marking this RFC because the timeout is difficult to recreate, so difficult to verify the patch. The patch was created purely from code inspection, so I'm just hoping for feedback on my analysis right now.

  drivers/nvme/host/tcp.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/host/tcp.c +0 -0
index 0222e23f5936..d07eb13d8713 100644

drivers/nvme/host/tcp.c static void nvme_tcp_io_work(struct work_struct *w) · line 1144

Quoted 6 lines
drivers/nvme/host/tcp.c +2 -1
@@ -1140,7 +1140,8 @@ static void nvme_tcp_io_work(struct work_struct *w)
  				pending = true;
  			else if (unlikely(result < 0))
  				break;
-		}
+		} else
+			pending = !llist_empty(&queue->req_list);

In my version, pending was unconditionally set to true...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#2 lore raw
Quoted 17 lines · Sagi Grimberg
Quoted 9 lines

A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by nvme_tcp_send_all() if it happens to be running. The driver relies on io_work to send the enqueued request when it is runs again, but the concurrently running nvme_tcp_send_all() may not have released the send_mutex at that time. If no future commands are enqueued to re-kick the io_work, the request will timeout in the SEND_H2C state, resulting in a timeout error like:

Quoted 4 lines
   nvme nvme0: queue 1: timeout request 0x3 type 6

Ensure the io_work continues to run as long as the req_list is not empty.

There is a version of this patch that I personally suggested before, however I couldn't explain why that should happen...

Quoted 6 lines

nvme_tcp_send_all tries to send everything it has queues, it means should either be able to send everything, or it should see a full socket buffer. But in case the socket buffer is full, there should be a .write_space() sk callback triggering when the socket buffer evacuates space... Maybe there is a chance that write_space triggered, started execution, and that the send_mutex is still taken?

nvme_tcp_send_all() breaks out of the loop if nvme_tcp_fetch_request() returns NULL. If that happens just before io_work calls nvme_tcp_handle_r2t() to enqueue the H2C request, nvme_tcp_send_all() will not see that request, but send_mutex is still held. We're counting on io_work to run again to handle sending the H2C data in that case. Unlikely as it sounds, if the same nvme_tcp_send_all() context is still holding the send_mutex when io_work gets back to trying to take it, how will the data get sent?

Quoted 1 line

Can we maybe try to catch if that is the case?

Do you have a better idea on how we can catch this? I think there was only one occurance of this sighting so far, and it looks like it took a long time to encounter it, but we will try again if you have a proposal.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer Reviewed-by
#3 lore raw
Quoted 25 lines
Quoted 18 lines
Quoted 10 lines

A possible race condition exists where the request to send data is enqueued from nvme_tcp_handle_r2t()'s will not be observed by nvme_tcp_send_all() if it happens to be running. The driver relies on io_work to send the enqueued request when it is runs again, but the concurrently running nvme_tcp_send_all() may not have released the send_mutex at that time. If no future commands are enqueued to re-kick the io_work, the request will timeout in the SEND_H2C state, resulting in a timeout error like:

Quoted 4 lines
    nvme nvme0: queue 1: timeout request 0x3 type 6

Ensure the io_work continues to run as long as the req_list is not empty.

There is a version of this patch that I personally suggested before, however I couldn't explain why that should happen...

Quoted 6 lines

nvme_tcp_send_all tries to send everything it has queues, it means should either be able to send everything, or it should see a full socket buffer. But in case the socket buffer is full, there should be a .write_space() sk callback triggering when the socket buffer evacuates space... Maybe there is a chance that write_space triggered, started execution, and that the send_mutex is still taken?

Quoted 8 lines

nvme_tcp_send_all() breaks out of the loop if nvme_tcp_fetch_request() returns NULL. If that happens just before io_work calls nvme_tcp_handle_r2t() to enqueue the H2C request, nvme_tcp_send_all() will not see that request, but send_mutex is still held. We're counting on io_work to run again to handle sending the H2C data in that case. Unlikely as it sounds, if the same nvme_tcp_send_all() context is still holding the send_mutex when io_work gets back to trying to take it, how will the data get sent?

Yes you are correct, overlooked this race. I guess this is all coming from having less queues than cores (where rx really competes with tx) which is not as common as a non default.

This is enough to convince me that this is needed:

Quoted 5 lines
Quoted 1 line

Can we maybe try to catch if that is the case?

Do you have a better idea on how we can catch this? I think there was only one occurance of this sighting so far, and it looks like it took a long time to encounter it, but we will try again if you have a proposal.

We can continue to test with the patch and hunt for another occurance, given the argument above, this patch is needed regardless...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#4 lore raw
Quoted 9 lines · Sagi Grimberg
Quoted 8 lines

nvme_tcp_send_all() breaks out of the loop if nvme_tcp_fetch_request() returns NULL. If that happens just before io_work calls nvme_tcp_handle_r2t() to enqueue the H2C request, nvme_tcp_send_all() will not see that request, but send_mutex is still held. We're counting on io_work to run again to handle sending the H2C data in that case. Unlikely as it sounds, if the same nvme_tcp_send_all() context is still holding the send_mutex when io_work gets back to trying to take it, how will the data get sent?

Quoted 6 lines

Yes you are correct, overlooked this race. I guess this is all coming from having less queues than cores (where rx really competes with tx) which is not as common as a non default.

This is enough to convince me that this is needed:

Great! I thought the scenario seemed possible, but wasn't completely sure, so thank you for confirming.

Christoph, can we pick this up for the next rc? For stable, we can add

Fixes db5ad6b7f8cdd ("nvme-tcp: try to send request in queue_rq context")
Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Can we maybe try to catch if that is the case?

Do you have a better idea on how we can catch this? I think there was only one occurance of this sighting so far, and it looks like it took a long time to encounter it, but we will try again if you have a proposal.

We can continue to test with the patch and hunt for another occurance, given the argument above, this patch is needed regardless...

Sounds good, we'll run with the patch and see what happens. If the tests are successful, I'm not sure if we can conclude this definitely fixes the timeout or if we just got lucky. If a timeout is observed, though, I will try to work in a debug patch.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme