Skip to contentSkip to conversation

scsi: core: only re-run queue in scsi_end_request() if device queue is busy

PATCH Superseded Ming Lei 7 participants 8 messages Aug 17, 2020 – Sep 8, 2020

Subsystems (2)

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH v4 1/1 scsi: core: only re-run queue in scsi_end_request() if device queue is busy

Now the request queue is run in scsi_end_request() unconditionally if both target queue and host queue is ready. We should have re-run request queue only after this device queue becomes busy for restarting this LUN only.

Recently Long Li reported that cost of run queue may be very heavy in case of high queue depth. So improve this situation by only running the request queue when this LUN is busy.

Cc Ewan D. Milne <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Long Li <[email protected]>
Cc John Garry <[email protected]>

---
V4:
- fix one race reported by Kashyap, and simplify the implementation a bit; also pass Kashyap's both function and performance test
V3:
- add one smp_mb() in scsi_mq_get_budget() and comment

V2:
- commit log change, no any code change
- add reported-by tag

 drivers/scsi/scsi_lib.c    | 51 +++++++++++++++++++++++++++++++++++---
 include/scsi/scsi_device.h |  1 +
 2 files changed, 49 insertions(+), 3 deletions(-)
drivers/scsi/scsi_lib.c +48 -3
index 7c6dd6f75190..a62c29058d26 100644
@@ -551,8 +551,27 @@ static void scsi_run_queue_async(struct scsi_device *sdev)
 	if (scsi_target(sdev)->single_lun ||
 	    !list_empty(&sdev->host->starved_list))
 		kblockd_schedule_work(&sdev->requeue_work);
-	else
-		blk_mq_run_hw_queues(sdev->request_queue, true);
+	else {
+		/*
+		 * smp_mb() implied in either rq->end_io or blk_mq_free_request
+		 * is for ordering writing .device_busy in scsi_device_unbusy()
+		 * and reading sdev->restarts.
+		 */
+		int old = atomic_read(&sdev->restarts);
+
+		if (old) {
+			/*
+			 * ->restarts has to be kept as non-zero if there is
+			 *  new budget contention comes.
+			 *
+			 *  No need to run queue when either another re-run
+			 *  queue wins in updating ->restarts or one new budget
+			 *  contention comes.
+			 */
+			if (atomic_cmpxchg(&sdev->restarts, old, 0) == old)
+				blk_mq_run_hw_queues(sdev->request_queue, true);
+		}
+	}
 }
 
 /* Returns false when no more bytes to process, true if there are more */
@@ -1611,8 +1630,34 @@ static void scsi_mq_put_budget(struct request_queue *q)
 static bool scsi_mq_get_budget(struct request_queue *q)
 {
 	struct scsi_device *sdev = q->queuedata;
+	int ret = scsi_dev_queue_ready(q, sdev);
+
+	if (ret)
+		return true;
+
+	atomic_inc(&sdev->restarts);
 
-	return scsi_dev_queue_ready(q, sdev);
+	/*
+	 * Order writing .restarts and reading .device_busy, and make sure
+	 * .restarts is visible to scsi_end_request(). Its pair is implied by
+	 * __blk_mq_end_request() in scsi_end_request() for ordering
+	 * writing .device_busy in scsi_device_unbusy() and reading .restarts.
+	 *
+	 */
+	smp_mb__after_atomic();
+
+	/*
+	 * If all in-flight requests originated from this LUN are completed
+	 * before setting .restarts, sdev->device_busy will be observed as
+	 * zero, then blk_mq_delay_run_hw_queues() will dispatch this request
+	 * soon. Otherwise, completion of one of these request will observe
+	 * the .restarts flag, and the request queue will be run for handling
+	 * this request, see scsi_end_request().
+	 */
+	if (unlikely(atomic_read(&sdev->device_busy) == 0 &&
+				!scsi_device_blocked(sdev)))
+		blk_mq_delay_run_hw_queues(sdev->request_queue, SCSI_QUEUE_DELAY);
+	return false;
 }
 
 static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
include/scsi/scsi_device.h +1 -0
index bc5909033d13..1a5c9a3df6d6 100644
@@ -109,6 +109,7 @@ struct scsi_device {
 	atomic_t device_busy;		/* commands actually active on LLDD */
 	atomic_t device_blocked;	/* Device returned QUEUE_FULL. */
 
+	atomic_t restarts;
 	spinlock_t list_lock;
 	struct list_head starved_entry;
 	unsigned short queue_depth;	/* How deep of a queue we want */
Signature · 1 line
2.25.2
Reviewed-by on v4 1/1
#1 lore raw
Quoted 8 lines · Ming Lei

Now the request queue is run in scsi_end_request() unconditionally if both target queue and host queue is ready. We should have re-run request queue only after this device queue becomes busy for restarting this LUN only.

Recently Long Li reported that cost of run queue may be very heavy in case of high queue depth. So improve this situation by only running the request queue when this LUN is busy.

Quoted 10 lines
Cc Ewan D. Milne <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Long Li <[email protected]>
Cc John Garry <[email protected]>

FWIW, this looks ok to me, and tested on scsi_debug showed less time in blk_mq_run_hw_queues() [for reduced queue depth]:

thanks

Tested-by
#2 lore raw
Quoted 24 lines

Subject: Re: [PATCH V4] scsi: core: only re-run queue in scsi_end_request() if device queue is busy

Quoted 3 lines · Ming Lei

Now the request queue is run in scsi_end_request() unconditionally if both target queue and host queue is ready. We should have re-run request queue only after this device queue becomes busy for restarting this

LUN only.

Quoted 5 lines

Recently Long Li reported that cost of run queue may be very heavy in case of high queue depth. So improve this situation by only running the request queue when this LUN is busy.

Quoted 10 lines
Cc Ewan D. Milne <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Long Li <[email protected]>
Cc John Garry <[email protected]>
Quoted 6 lines

FWIW, this looks ok to me, and tested on scsi_debug showed less time in blk_mq_run_hw_queues() [for reduced queue depth]:

thanks

On Azure Standard D64s v3 (64 vcpus, 256 GiB memory), it sees better CPU utilization with the patch.

#3 lore raw
Quoted 5 lines

On Azure Standard D64s v3 (64 vcpus, 256 GiB memory), it sees better CPU utilization with the patch.

Since this is a core change I'd like to see another review tag.

Thanks!

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering
Reviewed-by on v4 1/1
#4 lore raw
In reply to Ming Lei
Quoted 25 lines · Ming Lei

Now the request queue is run in scsi_end_request() unconditionally if both target queue and host queue is ready. We should have re-run request queue only after this device queue becomes busy for restarting this LUN only.

Recently Long Li reported that cost of run queue may be very heavy in case of high queue depth. So improve this situation by only running the request queue when this LUN is busy.

Cc Ewan D. Milne <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Long Li <[email protected]>
Cc John Garry <[email protected]>

---
V4:
- fix one race reported by Kashyap, and simplify the implementation a bit; also pass Kashyap's both function and performance test
V3:
- add one smp_mb() in scsi_mq_get_budget() and comment

Quoted 8 lines

V2:
- commit log change, no any code change
- add reported-by tag

  drivers/scsi/scsi_lib.c    | 51 +++++++++++++++++++++++++++++++++++---
  include/scsi/scsi_device.h |  1 +
  2 files changed, 49 insertions(+), 3 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
#5 lore raw
In reply to Ming Lei

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 554

Quoted 11 lines · Ming Lei
drivers/scsi/scsi_lib.c +1 -2
index 7c6dd6f75190..a62c29058d26 100644
@@ -551,8 +551,27 @@ static void scsi_run_queue_async(struct scsi_device *sdev)
 	if (scsi_target(sdev)->single_lun ||
 	    !list_empty(&sdev->host->starved_list))
 		kblockd_schedule_work(&sdev->requeue_work);
-	else
-		blk_mq_run_hw_queues(sdev->request_queue, true);
+	else {

Has this patch been verified with checkpatch? Checkpatch should have warned about the unbalanced braces.

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 559

Quoted 5 lines
drivers/scsi/scsi_lib.c +5 -0
+		/*
+		 * smp_mb() implied in either rq->end_io or blk_mq_free_request
+		 * is for ordering writing .device_busy in scsi_device_unbusy()
+		 * and reading sdev->restarts.
+		 */

Hmm ... I don't see what orders the atomic_dec(&sdev->device_busy) from scsi_device_unbusy() and the atomic_read() below? I don't think that the block layer guarantees ordering of these two memory accesses since both accesses happen in the request completion path.

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 565

Quoted 6 lines
drivers/scsi/scsi_lib.c +6 -0
+		int old = atomic_read(&sdev->restarts);
+
+		if (old) {
+			/*
+			 * ->restarts has to be kept as non-zero if there is
+			 *  new budget contention comes.

There are two verbs in the above sentence ("is" and "comes"). Please remove "comes" such that the sentence becomes grammatically correct.

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 574

Quoted 9 lines
drivers/scsi/scsi_lib.c +9 -0
+			 *
+			 *  No need to run queue when either another re-run
+			 *  queue wins in updating ->restarts or one new budget
+			 *  contention comes.
+			 */
+			if (atomic_cmpxchg(&sdev->restarts, old, 0) == old)
+				blk_mq_run_hw_queues(sdev->request_queue, true);
+		}
+	}

Please combine the two if-statements into a single if-statement using "&&" to keep the indentation level low.

Quoted 10 lines
drivers/scsi/scsi_lib.c +6 -0
@@ -1611,8 +1630,34 @@ static void scsi_mq_put_budget(struct request_queue *q)
 static bool scsi_mq_get_budget(struct request_queue *q)
 {
 	struct scsi_device *sdev = q->queuedata;
+	int ret = scsi_dev_queue_ready(q, sdev);
+
+	if (ret)
+		return true;
+
+	atomic_inc(&sdev->restarts);

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q) · line 1647

Quoted 10 lines
drivers/scsi/scsi_lib.c +8 -1
 
-	return scsi_dev_queue_ready(q, sdev);
+	/*
+	 * Order writing .restarts and reading .device_busy, and make sure
+	 * .restarts is visible to scsi_end_request(). Its pair is implied by
+	 * __blk_mq_end_request() in scsi_end_request() for ordering
+	 * writing .device_busy in scsi_device_unbusy() and reading .restarts.
+	 *
+	 */
+	smp_mb__after_atomic();

Barriers do not guarantee "is visible to". Barriers enforce ordering of memory accesses performed by a certain CPU core. Did you perhaps mean that sdev->restarts must be incremented before the code below reads sdev->device busy?

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q) · line 1661

Quoted 13 lines
drivers/scsi/scsi_lib.c +12 -0
+	/*
+	 * If all in-flight requests originated from this LUN are completed
+	 * before setting .restarts, sdev->device_busy will be observed as
+	 * zero, then blk_mq_delay_run_hw_queues() will dispatch this request
+	 * soon. Otherwise, completion of one of these request will observe
+	 * the .restarts flag, and the request queue will be run for handling
+	 * this request, see scsi_end_request().
+	 */
+	if (unlikely(atomic_read(&sdev->device_busy) == 0 &&
+				!scsi_device_blocked(sdev)))
+		blk_mq_delay_run_hw_queues(sdev->request_queue, SCSI_QUEUE_DELAY);
+	return false;
 }

Thanks,

Bart.

author
#6 lore raw
Quoted 15 lines · Bart Van Assche

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 554

Quoted 11 lines · Ming Lei
drivers/scsi/scsi_lib.c +1 -2
index 7c6dd6f75190..a62c29058d26 100644
@@ -551,8 +551,27 @@ static void scsi_run_queue_async(struct scsi_device *sdev)
 	if (scsi_target(sdev)->single_lun ||
 	    !list_empty(&sdev->host->starved_list))
 		kblockd_schedule_work(&sdev->requeue_work);
-	else
-		blk_mq_run_hw_queues(sdev->request_queue, true);
+	else {

Has this patch been verified with checkpatch? Checkpatch should have warned about the unbalanced braces.

[linux]$ ./scripts/checkpatch.pl -g HEAD
total: 0 errors, 0 warnings, 71 lines checked

Commit 0cbe51645b54 ("scsi: core: only re-run queue in scsi_end_request() if device queue is busy") has no obvious style problems and is ready for submission.

Quoted 11 lines

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 559

Quoted 6 lines
drivers/scsi/scsi_lib.c +5 -0
+		/*
+		 * smp_mb() implied in either rq->end_io or blk_mq_free_request
+		 * is for ordering writing .device_busy in scsi_device_unbusy()
+		 * and reading sdev->restarts.
+		 */

Hmm ... I don't see what orders the atomic_dec(&sdev->device_busy) from scsi_device_unbusy() and the atomic_read() below? I don't think that the block layer guarantees ordering of these two memory accesses since both accesses happen in the request completion path.

__blk_mq_end_request() is called between scsi_device_unbusy() and scsi_run_queue_async(). When __blk_mq_end_request() is called, this request is actually ended really because SCMD_STATE_COMPLETE is covered race between timeout and normal completion, so:

1) either __blk_mq_free_request() is called, smp_mb__after_atomic() is implied in sbitmap_queue_clear() called from blk_mq_put_tag()

2) or rq->end_io() is called. We don't have too many ->end_io() implemented. Either wake_up_process() or blk_mq_free_request() is called in ->end_io(), so memory barrier is implied.

Quoted 35 lines

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 565

Quoted 7 lines
drivers/scsi/scsi_lib.c +6 -0
+		int old = atomic_read(&sdev->restarts);
+
+		if (old) {
+			/*
+			 * ->restarts has to be kept as non-zero if there is
+			 *  new budget contention comes.

There are two verbs in the above sentence ("is" and "comes"). Please remove "comes" such that the sentence becomes grammatically correct.

drivers/scsi/scsi_lib.c static void scsi_run_queue_async(struct scsi_device *sdev) · line 574

Quoted 9 lines
drivers/scsi/scsi_lib.c +9 -0
+			 *
+			 *  No need to run queue when either another re-run
+			 *  queue wins in updating ->restarts or one new budget
+			 *  contention comes.
+			 */
+			if (atomic_cmpxchg(&sdev->restarts, old, 0) == old)
+				blk_mq_run_hw_queues(sdev->request_queue, true);
+		}
+	}

Please combine the two if-statements into a single if-statement using "&&" to keep the indentation level low.

Quoted 11 lines
drivers/scsi/scsi_lib.c +6 -0
@@ -1611,8 +1630,34 @@ static void scsi_mq_put_budget(struct request_queue *q)
 static bool scsi_mq_get_budget(struct request_queue *q)
 {
 	struct scsi_device *sdev = q->queuedata;
+	int ret = scsi_dev_queue_ready(q, sdev);
+
+	if (ret)
+		return true;
+
+	atomic_inc(&sdev->restarts);
 
Quoted 13 lines

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q) · line 1647

Quoted 9 lines
drivers/scsi/scsi_lib.c +8 -1
-	return scsi_dev_queue_ready(q, sdev);
+	/*
+	 * Order writing .restarts and reading .device_busy, and make sure
+	 * .restarts is visible to scsi_end_request(). Its pair is implied by
+	 * __blk_mq_end_request() in scsi_end_request() for ordering
+	 * writing .device_busy in scsi_device_unbusy() and reading .restarts.
+	 *
+	 */
+	smp_mb__after_atomic();

Barriers do not guarantee "is visible to". Barriers enforce ordering of memory accesses performed by a certain CPU core. Did you perhaps mean that sdev->restarts must be incremented before the code below reads sdev->device busy?

Right, ->restart has to be incremented before reading sdev->device_busy.

Thanks,
Ming

Reviewed-by on v4 1/1
#7 lore raw
In reply to Ming Lei

So we ran a whole bunch of tests with this patch applied, to see if it would cause any problems, and it ran OK. The V5 patch just posted yesterday does not appear to change the logic functionally.

See below for comments.

Quoted 68 lines · Ming Lei

Now the request queue is run in scsi_end_request() unconditionally if both target queue and host queue is ready. We should have re-run request queue only after this device queue becomes busy for restarting this LUN only.

Recently Long Li reported that cost of run queue may be very heavy in case of high queue depth. So improve this situation by only running the request queue when this LUN is busy.

Cc Ewan D. Milne <[email protected]>
Cc Kashyap Desai <[email protected]>
Cc Hannes Reinecke <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Long Li <[email protected]>
Cc John Garry <[email protected]>

---
V4:
- fix one race reported by Kashyap, and simplify the implementation a bit; also pass Kashyap's both function and performance test
V3:
- add one smp_mb() in scsi_mq_get_budget() and comment

V2:
- commit log change, no any code change
- add reported-by tag

 drivers/scsi/scsi_lib.c    | 51 +++++++++++++++++++++++++++++++++++---
 include/scsi/scsi_device.h |  1 +
 2 files changed, 49 insertions(+), 3 deletions(-)
drivers/scsi/scsi_lib.c +21 -2
index 7c6dd6f75190..a62c29058d26 100644
@@ -551,8 +551,27 @@ static void scsi_run_queue_async(struct scsi_device *sdev)
 	if (scsi_target(sdev)->single_lun ||
 	    !list_empty(&sdev->host->starved_list))
 		kblockd_schedule_work(&sdev->requeue_work);
-	else
-		blk_mq_run_hw_queues(sdev->request_queue, true);
+	else {
+		/*
+		 * smp_mb() implied in either rq->end_io or blk_mq_free_request
+		 * is for ordering writing .device_busy in scsi_device_unbusy()
+		 * and reading sdev->restarts.
+		 */
+		int old = atomic_read(&sdev->restarts);
+
+		if (old) {
+			/*
+			 * ->restarts has to be kept as non-zero if there is
+			 *  new budget contention comes.
+			 *
+			 *  No need to run queue when either another re-run
+			 *  queue wins in updating ->restarts or one new budget
+			 *  contention comes.
+			 */
+			if (atomic_cmpxchg(&sdev->restarts, old, 0) == old)
+				blk_mq_run_hw_queues(sdev->request_queue, true);
+		}
+	}
 }
 
 /* Returns false when no more bytes to process, true if there are more */

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q) · line 1636

Quoted 7 lines
drivers/scsi/scsi_lib.c +4 -0
@@ -1611,8 +1630,34 @@ static void scsi_mq_put_budget(struct request_queue *q)
 static bool scsi_mq_get_budget(struct request_queue *q)
 {
 	struct scsi_device *sdev = q->queuedata;
+	int ret = scsi_dev_queue_ready(q, sdev);
+
+	if (ret)
+		return true;

I think this should just be:

	if (scsi_dev_queue_ready(q, sdev))
		return true;

There's no particular reason to call the function in a local variable initializer, this just makes the code less clear to me. And "ret" isn't needed for any other reason.

drivers/scsi/scsi_lib.c static void scsi_mq_put_budget(struct request_queue *q) · line 1647

Quoted 12 lines
drivers/scsi/scsi_lib.c +10 -1
+
+	atomic_inc(&sdev->restarts);
 
-	return scsi_dev_queue_ready(q, sdev);
+	/*
+	 * Order writing .restarts and reading .device_busy, and make sure
+	 * .restarts is visible to scsi_end_request(). Its pair is implied by
+	 * __blk_mq_end_request() in scsi_end_request() for ordering
+	 * writing .device_busy in scsi_device_unbusy() and reading .restarts.
+	 *
+	 */
+	smp_mb__after_atomic();

I'm not sure how this helps, if you're trying to ensure consistency between sdev->device_busy and sdev->restarts. Another thread could be in scsi_dev_queue_ready(), right? And scsi_run_queue_async() in the scsi_end_request() path is doing an atomic_read() of ->restarts anyway.

Quoted 28 lines
drivers/scsi/scsi_lib.c +13 -0
+
+	/*
+	 * If all in-flight requests originated from this LUN are completed
+	 * before setting .restarts, sdev->device_busy will be observed as
+	 * zero, then blk_mq_delay_run_hw_queues() will dispatch this request
+	 * soon. Otherwise, completion of one of these request will observe
+	 * the .restarts flag, and the request queue will be run for handling
+	 * this request, see scsi_end_request().
+	 */
+	if (unlikely(atomic_read(&sdev->device_busy) == 0 &&
+				!scsi_device_blocked(sdev)))
+		blk_mq_delay_run_hw_queues(sdev->request_queue, SCSI_QUEUE_DELAY);
+	return false;
 }
 
 static blk_status_t scsi_queue_rq(struct blk_mq_hw_ctx *hctx,
include/scsi/scsi_device.h +1 -0
index bc5909033d13..1a5c9a3df6d6 100644
@@ -109,6 +109,7 @@ struct scsi_device {
 	atomic_t device_busy;		/* commands actually active on LLDD */
 	atomic_t device_blocked;	/* Device returned QUEUE_FULL. */
 
+	atomic_t restarts;
 	spinlock_t list_lock;
 	struct list_head starved_entry;
 	unsigned short queue_depth;	/* How deep of a queue we want */