Skip to contentSkip to conversation

nvme: cancel requests for real

PATCH In review Keith Busch 5 participants 8 messages May 27, 2020 – May 28, 2020

Subsystems (2)
lore
Share & tools
lore

Keith Busch

author maintainer
#0 lore raw

PATCH 1/1 nvme: cancel requests for real

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +1 -1
index ba860efd250d..72e5973dda3a 100644
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
 		return true;
 
 	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-	blk_mq_complete_request(req);
+	while (!blk_mq_complete_request(req));
 	return true;
 }
 EXPORT_SYMBOL_GPL(nvme_cancel_request);
Signature · 7 lines
2.24.1


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw

Hi Keith,

Quoted 15 lines · Keith Busch

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index ba860efd250d..72e5973dda3a 100644

drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 313

Quoted 5 lines
drivers/nvme/host/core.c +1 -1
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
 		return true;
 
 	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-	blk_mq_complete_request(req);
+	while (!blk_mq_complete_request(req));

If the probability is configured to fail every time with 100% probability, we would run into endless loop?

Dongli Zhang

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#2 lore raw
Quoted 18 lines · Dongli Zhang
Quoted 17 lines · Keith Busch

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index ba860efd250d..72e5973dda3a 100644
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
 		return true;
 
Quoted 5 lines

drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 313

Quoted 3 lines
drivers/nvme/host/core.c +1 -1
 	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-	blk_mq_complete_request(req);
+	while (!blk_mq_complete_request(req));

If the probability is configured to fail every time with 100% probability, we would run into endless loop?

If completing requests is not possible, it's already endless. This is just a tighter loop in that case.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
In reply to Keith Busch
Quoted 16 lines

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
  drivers/nvme/host/core.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index ba860efd250d..72e5973dda3a 100644

drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 313

Quoted 5 lines
drivers/nvme/host/core.c +1 -1
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
  		return true;
 
  	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-	blk_mq_complete_request(req);
+	while (!blk_mq_complete_request(req));

Maybe add cond_resched in the loop? just to play nice in case something is wrong?

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#4 lore raw
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
  drivers/nvme/host/core.c | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines

drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, · line 313

Quoted 10 lines
drivers/nvme/host/core.c +1 -1
index ba860efd250d..72e5973dda3a 100644
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, 
 void *data, bool reserved)
           return true;
         nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-    blk_mq_complete_request(req);
+    while (!blk_mq_complete_request(req));

Maybe add cond_resched in the loop? just to play nice in case something is wrong?

Is this a nvme issue or a block issue?  My original fix was in the block layer.

block/blk-timeout.c +0 -0
@@ -22,10 +22,10 @@ static int __init setup_fail_io_timeout(char *str)
 

int blk_should_fake_timeout(struct request_queue *q) {
-       if (!test_bit(QUEUE_FLAG_FAIL_IO, &q->queue_flags)) -               return 0;
-
-       return should_fail(&fail_io_timeout, 1);
+       if (test_bit(QUEUE_FLAG_FAIL_IO, &q->queue_flags) && +           !blk_queue_quiesced(q))
+               return should_fail(&fail_io_timeout, 1); +       return 0;
}

static int __init fail_io_timeout_debugfs(void)

Signature · 10 lines
Alan





_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#6 lore raw
In reply to Keith Busch
Quoted 15 lines · Keith Busch

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index ba860efd250d..72e5973dda3a 100644

drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 314

Quoted 6 lines
drivers/nvme/host/core.c +1 -1
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
 		return true;
 
 	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-	blk_mq_complete_request(req);
+	while (!blk_mq_complete_request(req));
 	return true;

This is just disgusting. Take a look at how blk_mq_complete_request is implemented, and what might be a better function to export and call here, or any other error handling context. Bonus points for finding a saner name for it.

I'm also thinking that whole idea of randomly not doing anything in blk_mq_complete_request is just a bad idea vs doing targeted hooks in the actual drivers.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw
In reply to Alan Adamson
Quoted 5 lines · Alan Adamson
block/blk-timeout.c +0 -0
@@ -22,10 +22,10 @@ static int __init setup_fail_io_timeout(char *str)
 
 int blk_should_fake_timeout(struct request_queue *q)

block/blk-timeout.c static int __init setup_fail_io_timeout(char *str) · line 29

Quoted 10 lines
block/blk-timeout.c +4 -4
 {
-       if (!test_bit(QUEUE_FLAG_FAIL_IO, &q->queue_flags))
-               return 0;
-
-       return should_fail(&fail_io_timeout, 1);
+       if (test_bit(QUEUE_FLAG_FAIL_IO, &q->queue_flags) &&
+           !blk_queue_quiesced(q))
+               return should_fail(&fail_io_timeout, 1);
+       return 0;
 }

While that is way better than the loop I think this encodes too way too many assumptions.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author maintainer
#7 lore raw
In reply to Christoph Hellwig
Quoted 18 lines · Christoph Hellwig
Quoted 17 lines · Keith Busch

Once the driver decides to cancel requests, the concept of those requests timing out should no longer exist. Since we can't stop fake timeouts from preventing a forced reclaim, continue completing the same request until the block layer isn't told to pretend that didn't happen.

---
 drivers/nvme/host/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/core.c +0 -0
index ba860efd250d..72e5973dda3a 100644
@@ -310,7 +310,7 @@ bool nvme_cancel_request(struct request *req, void *data, bool reserved)
 		return true;
 
Quoted 6 lines

drivers/nvme/host/core.c bool nvme_cancel_request(struct request *req, void *data, bool reserved) · line 314

Quoted 4 lines
drivers/nvme/host/core.c +1 -1
 	nvme_req(req)->status = NVME_SC_HOST_ABORTED_CMD;
-	blk_mq_complete_request(req);
+	while (!blk_mq_complete_request(req));
 	return true;

This is just disgusting.

Thanks for noticing. :)

Quoted 3 lines

I'm also thinking that whole idea of randomly not doing anything in blk_mq_complete_request is just a bad idea vs doing targeted hooks in the actual drivers.

Yeah, I don't like fake test code in the middle of production code. Testing code shouldn't modify the code being testing. IMO, leave real stuff alone and create a thunk for error injection, preferably one specific to the low-level driver. I know we can do that on x86, but not so sure about other archs.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme