Skip to contentSkip to conversation

blk-mq: implement queue quiesce via percpu_ref for BLK_MQ_F_BLOCKING

RFC PATCH In review Ming Lei 5 participants 20 messages Jul 28, 2020 – Jul 31, 2020 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

RFC PATCH 1/1 blk-mq: implement queue quiesce via percpu_ref for BLK_MQ_F_BLOCKING

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Also it becomes much easier to add interface of async queue quiesce.

Cc Lai Jiangshan <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Josh Triplett <[email protected]>
Cc Sagi Grimberg <[email protected]>
Cc Bart Van Assche <[email protected]>
---
 block/blk-mq-sysfs.c   |  2 -
 block/blk-mq.c         | 94 +++++++++++++++++++-----------------------
 block/blk-sysfs.c      |  6 ++-
 include/linux/blk-mq.h |  7 ----
 include/linux/blkdev.h |  4 ++
 5 files changed, 51 insertions(+), 62 deletions(-)
block/blk-mq-sysfs.c +0 -2
index 062229395a50..799db7937105 100644
@@ -38,8 +38,6 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj)
 
 	cancel_delayed_work_sync(&hctx->run_work);
 
-	if (hctx->flags & BLK_MQ_F_BLOCKING)
-		cleanup_srcu_struct(hctx->srcu);
 	blk_free_flush_queue(hctx->fq);
 	sbitmap_free(&hctx->ctx_map);
 	free_cpumask_var(hctx->cpumask);
block/blk-mq.c +42 -52

215 lines not on this page · raw message

block/blk-sysfs.c +5 -1
index be67952e7be2..8f22de582874 100644
@@ -914,9 +914,13 @@ static void blk_release_queue(struct kobject *kobj)
 
 	blk_queue_free_zone_bitmaps(q);
 
-	if (queue_is_mq(q))
+	if (queue_is_mq(q)) {
 		blk_mq_release(q);
 
+		if (q->tag_set->flags & BLK_MQ_F_BLOCKING)
+			percpu_ref_exit(&q->dispatch_counter);
+	}
+
 	blk_trace_shutdown(q);
 	mutex_lock(&q->debugfs_mutex);
 	debugfs_remove_recursive(q->debugfs_dir);
include/linux/blk-mq.h +0 -7
index 23230c1d031e..70e64e87586c 100644
@@ -169,13 +169,6 @@ struct blk_mq_hw_ctx {
 	 * q->unused_hctx_list.
 	 */
 	struct list_head	hctx_list;
-
-	/**
-	 * @srcu: Sleepable RCU. Use as lock when type of the hardware queue is
-	 * blocking (BLK_MQ_F_BLOCKING). Must be the last member - see also
-	 * blk_mq_hw_ctx_size().
-	 */
-	struct srcu_struct	srcu[];
 };
 
 /**
include/linux/blkdev.h +4 -0
index 06995b96e946..32d6e194e6eb 100644
@@ -567,6 +567,10 @@ struct request_queue {
 	struct list_head	tag_set_list;
 	struct bio_set		bio_split;
 
+	/* only used for BLK_MQ_F_BLOCKING */
+	struct percpu_ref	dispatch_counter;
+	wait_queue_head_t	mq_quiesce_wq;
+
 	struct dentry		*debugfs_dir;
 
 #ifdef CONFIG_BLK_DEBUG_FS
Signature · 1 line
2.25.2
author
#1 lore raw
Quoted 11 lines · Ming Lei

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Also it becomes much easier to add interface of async queue quiesce.

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks,
Ming

#2 lore raw
Quoted 11 lines
Quoted 11 lines

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Quoted 4 lines
Quoted 1 line

Also it becomes much easier to add interface of async queue quiesce.

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks Ming, can you test for non-blocking on the same setup?

I can test some reset storms during traffic.

author
#3 lore raw
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
Quoted 11 lines

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Also it becomes much easier to add interface of async queue quiesce.

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks Ming, can you test for non-blocking on the same setup?

OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?

Thanks,
Ming

#4 lore raw
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Also it becomes much easier to add interface of async queue quiesce.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks Ming, can you test for non-blocking on the same setup?

OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

author
#5 lore raw
Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Quoted 17 lines
Quoted 11 lines
Quoted 11 lines

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Quoted 4 lines
Quoted 1 line

Also it becomes much easier to add interface of async queue quiesce.

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks Ming, can you test for non-blocking on the same setup?

Quoted 9 lines
Quoted 2 lines

OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

BTW, in case of blocking, one hctx may dispatch at most one request because there is only single .run_work, even though when .queue_rq() is slept, that said blk_mq_submit_bio() queues bio in sync style. This way won't be very efficient. So percpu_refcount should be good enough for blocking code path, but may not be well enough for non-blocking case.

Thanks,
Ming

#6 lore raw
Quoted 25 lines
Quoted 25 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Also it becomes much easier to add interface of async queue quiesce.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks Ming, can you test for non-blocking on the same setup?

OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

Quoted 6 lines
Quoted 2 lines

If the cost is negligible, I think the upside is worth it.

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

Quoted 5 lines

BTW, in case of blocking, one hctx may dispatch at most one request because there is only single .run_work, even though when .queue_rq() is slept, that said blk_mq_submit_bio() queues bio in sync style. This way won't be very efficient. So percpu_refcount should be good enough for blocking code path, but may not be well enough for non-blocking case.

Not sure what you mean, the percpu_ref is taken exactly where rcu is taken, not sure what is the difference.

#7 lore raw
Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
Quoted 6 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

Quoted 8 lines
Quoted 3 lines

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

https://lore.kernel.org/linux-block/[email protected]/

#8 lore raw
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

Quoted 8 lines
Quoted 3 lines

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

Quoted 4 lines

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

   https://lore.kernel.org/linux-block/[email protected]/

If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.

#9 lore raw
Quoted 17 lines · Sagi Grimberg
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
Quoted 7 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

Quoted 8 lines
Quoted 4 lines

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

   https://lore.kernel.org/linux-block/[email protected]/

If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.

Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.

maintainer
#10 lore raw
Quoted 23 lines · Keith Busch
Quoted 22 lines · Sagi Grimberg
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

Quoted 8 lines
Quoted 3 lines

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

   https://lore.kernel.org/linux-block/[email protected]/
Quoted 7 lines
Quoted 3 lines

If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.

Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.

I fully agree with that point.

Signature · 1 line
Jens Axboe
#11 lore raw
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

Quoted 8 lines
Quoted 3 lines

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

Quoted 4 lines

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

    https://lore.kernel.org/linux-block/[email protected]/

If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.

Quoted 5 lines
Quoted 3 lines

Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.

I fully agree with that point.

I also agree, just said we should use the same mechanisms, IFF its not
expensive. But I'm concerned that if we use completely different mechanisms we are likely to get wrong assumptions and break one at some point.

Hence my suggestion to move back to srcu and place the rcu_head in the hctx.

author
#12 lore raw
Quoted 30 lines · Sagi Grimberg
Quoted 30 lines
Quoted 29 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
Quoted 7 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

Quoted 8 lines
Quoted 4 lines

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

    https://lore.kernel.org/linux-block/[email protected]/

If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.

Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.

Quoted 8 lines
Quoted 1 line

I fully agree with that point.

I also agree, just said we should use the same mechanisms, IFF its not expensive. But I'm concerned that if we use completely different mechanisms we are likely to get wrong assumptions and break one at some point.

Hence my suggestion to move back to srcu and place the rcu_head in the hctx.

SRCU has been different enough compared with RCU, either implementation or API interface.

Then I'd still suggest to replace SRCU with percpu refcount. Then we can have a simpler quiesce implementation.

Thanks,
Ming

author
#13 lore raw
In reply to Keith Busch
Quoted 17 lines · Keith Busch
Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
Quoted 6 lines

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

Quoted 8 lines
Quoted 3 lines

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

Quoted 4 lines

FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.

https://lore.kernel.org/linux-block/[email protected]/

That is why I don't want to switch non-blocking to percpu-refcount.

However, cost of srcu read lock/unlock is basically similar with percpu ref.

Thanks,
Ming

author
#14 lore raw
In reply to Sagi Grimberg
Quoted 28 lines · Sagi Grimberg
Quoted 28 lines
Quoted 27 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.

However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.

Also it becomes much easier to add interface of async queue quiesce.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.

Thanks Ming, can you test for non-blocking on the same setup?

OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?

I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.

If the cost is negligible, I think the upside is worth it.

Quoted 8 lines
Quoted 3 lines

rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.

It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.

RCU and SRCU have been different mechanism already.

Quoted 9 lines
Quoted 6 lines

BTW, in case of blocking, one hctx may dispatch at most one request because there is only single .run_work, even though when .queue_rq() is slept, that said blk_mq_submit_bio() queues bio in sync style. This way won't be very efficient. So percpu_refcount should be good enough for blocking code path, but may not be well enough for non-blocking case.

Not sure what you mean, the percpu_ref is taken exactly where rcu is taken, not sure what is the difference.

My point is that blocking can't be efficient enough, when .queue_rq() is slept, no any request can be queued to this hctx any more.

thanks,
Ming

#15 lore raw
In reply to Ming Lei

On 28/07/2020 15:50, Ming Lei wrote:
[...]

Quoted 4 lines
block/blk-mq.c +1 -2
-static void hctx_lock(struct blk_mq_hw_ctx *hctx, int *srcu_idx)
-	__acquires(hctx->srcu)
+static void hctx_lock(struct blk_mq_hw_ctx *hctx)
 {

block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713

Quoted 10 lines
block/blk-mq.c +3 -5
-	if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
-		/* shut up gcc false positive */
-		*srcu_idx = 0;
+	if (!(hctx->flags & BLK_MQ_F_BLOCKING))
 		rcu_read_lock();
-	} else
-		*srcu_idx = srcu_read_lock(hctx->srcu);
+	else
+		percpu_ref_get(&hctx->queue->dispatch_counter);
 }

I quite like this because it hides the internals of hctx_lock() (rcu vs srcu) from the callers.

Quoted 11 lines

/**

block/blk-mq.c +0 -2
@@ -1486,8 +1479,6 @@ bool blk_mq_dispatch_rq_list(struct blk_mq_hw_ctx *hctx, struct list_head *list,
  */
 static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 {
-	int srcu_idx;
-
 	/*
 	 * We should be running this queue from one of the CPUs that
 	 * are mapped to it.

block/blk-mq.c static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · line 1517

Quoted 8 lines
block/blk-mq.c +2 -2
@@ -1521,9 +1512,9 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)
 
 	might_sleep_if(hctx->flags & BLK_MQ_F_BLOCKING);
 
-	hctx_lock(hctx, &srcu_idx);
+	hctx_lock(hctx);
 	blk_mq_sched_dispatch_requests(hctx);
-	hctx_unlock(hctx, srcu_idx);
+	hctx_unlock(hctx);

blk_mq_sched_dispatch_requests also has this comment at the beginning:

	/* RCU or SRCU read lock is needed before checking quiesced flag */
	if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)))

I think the SRCU part needs to be changed to percpu_ref or the whole comment should probably just mention the hctx_lock().

#16 lore raw
In reply to Ming Lei
Quoted 27 lines · Ming Lei

void blk_mq_quiesce_queue(struct request_queue *q) {

block/blk-mq.c +6 -11
-	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
-	bool rcu = false;
-
 	blk_mq_quiesce_queue_nowait(q);
 
-	queue_for_each_hw_ctx(q, hctx, i) {
-		if (hctx->flags & BLK_MQ_F_BLOCKING)
-			synchronize_srcu(hctx->srcu);
-		else
-			rcu = true;
-	}
-	if (rcu)
+	if (q->tag_set->flags & BLK_MQ_F_BLOCKING) {
+		percpu_ref_kill(&q->dispatch_counter);
+		wait_event(q->mq_quiesce_wq,
+				percpu_ref_is_zero(&q->dispatch_counter));
+	} else
 		synchronize_rcu();
 }
 
 
 
+static void hctx_lock(struct blk_mq_hw_ctx *hctx)
 {

block/blk-mq.c EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_nowait);

Quoted 10 lines
block/blk-mq.c +3 -5
-	if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
-		/* shut up gcc false positive */
-		*srcu_idx = 0;
+	if (!(hctx->flags & BLK_MQ_F_BLOCKING))
 		rcu_read_lock();
-	} else
-		*srcu_idx = srcu_read_lock(hctx->srcu);
+	else
+		percpu_ref_get(&hctx->queue->dispatch_counter);
 }

percpu_ref_get() will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observe blk_queue_quiesced(). That doesn't require that there are no current dispatchers.

author
#17 lore raw
Quoted 36 lines · Keith Busch
Quoted 35 lines · Ming Lei

void blk_mq_quiesce_queue(struct request_queue *q) {

block/blk-mq.c +8 -16
-	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
-	bool rcu = false;
-
 	blk_mq_quiesce_queue_nowait(q);
 
-	queue_for_each_hw_ctx(q, hctx, i) {
-		if (hctx->flags & BLK_MQ_F_BLOCKING)
-			synchronize_srcu(hctx->srcu);
-		else
-			rcu = true;
-	}
-	if (rcu)
+	if (q->tag_set->flags & BLK_MQ_F_BLOCKING) {
+		percpu_ref_kill(&q->dispatch_counter);
+		wait_event(q->mq_quiesce_wq,
+				percpu_ref_is_zero(&q->dispatch_counter));
+	} else
 		synchronize_rcu();
 }
 
 
 
+static void hctx_lock(struct blk_mq_hw_ctx *hctx)
 {
-	if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
-		/* shut up gcc false positive */
-		*srcu_idx = 0;
+	if (!(hctx->flags & BLK_MQ_F_BLOCKING))
 		rcu_read_lock();
-	} else
-		*srcu_idx = srcu_read_lock(hctx->srcu);
+	else
Quoted 8 lines

block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713

Quoted 2 lines
block/blk-mq.c +1 -0
+		percpu_ref_get(&hctx->queue->dispatch_counter);
 }

percpu_ref_get() will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observe blk_queue_quiesced(). That doesn't require that there are no current dispatchers.

IMO it shouldn't be one issue in reality, because:

- when dispatch can't make progress, the submission side will finally stop because we either run queue from submission side or request completion

- submission side stops because we always have very limited requests

- completion side stops because requests queued to device is limited too

We still can handle this case by not dispatch in case that percpu_ref_tryget() returns false, which will change the usage into the following way:

        if (hctx_lock(hctx)) {
        	blk_mq_sched_dispatch_requests(hctx);
        	hctx_unlock(hctx);
		}

And __blk_mq_try_issue_directly() needs a bit special treatment because the request has to be inserted to queue after queue becomes quiesced.

Thanks,
Ming

#18 lore raw
Quoted 47 lines
Quoted 36 lines
Quoted 36 lines
  void blk_mq_quiesce_queue(struct request_queue *q)
  {
block/blk-mq.c +8 -16
-	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
-	bool rcu = false;
-
  	blk_mq_quiesce_queue_nowait(q);
 
-	queue_for_each_hw_ctx(q, hctx, i) {
-		if (hctx->flags & BLK_MQ_F_BLOCKING)
-			synchronize_srcu(hctx->srcu);
-		else
-			rcu = true;
-	}
-	if (rcu)
+	if (q->tag_set->flags & BLK_MQ_F_BLOCKING) {
+		percpu_ref_kill(&q->dispatch_counter);
+		wait_event(q->mq_quiesce_wq,
+				percpu_ref_is_zero(&q->dispatch_counter));
+	} else
  		synchronize_rcu();
  }
 
 
 
+static void hctx_lock(struct blk_mq_hw_ctx *hctx)
  {
-	if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
-		/* shut up gcc false positive */
-		*srcu_idx = 0;
+	if (!(hctx->flags & BLK_MQ_F_BLOCKING))
  		rcu_read_lock();
-	} else
-		*srcu_idx = srcu_read_lock(hctx->srcu);
+	else
Quoted 8 lines

block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713

Quoted 2 lines
block/blk-mq.c +1 -0
+		percpu_ref_get(&hctx->queue->dispatch_counter);
  }

percpu_ref_get() will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observe blk_queue_quiesced(). That doesn't require that there are no current dispatchers.

IMO it shouldn't be one issue in reality, because:

Quoted 8 lines

- when dispatch can't make progress, the submission side will finally stop because we either run queue from submission side or request completion

- submission side stops because we always have very limited requests

- completion side stops because requests queued to device is limited too

I don't think that any requests should pass after the kill was called, otherwise how can we safely quiesce if requests can come in after it?

Quoted 2 lines

We still can handle this case by not dispatch in case that percpu_ref_tryget()

You meant tryget_live right?

Quoted 9 lines

returns false, which will change the usage into the following way:

         if (hctx_lock(hctx)) {
         	blk_mq_sched_dispatch_requests(hctx);
         	hctx_unlock(hctx);
		}

And __blk_mq_try_issue_directly() needs a bit special treatment because the request has to be inserted to queue after queue becomes quiesced.

Agreed.

author
#19 lore raw
Quoted 51 lines · Sagi Grimberg
Quoted 51 lines
Quoted 35 lines
Quoted 35 lines
  void blk_mq_quiesce_queue(struct request_queue *q)
  {
block/blk-mq.c +8 -16
-	struct blk_mq_hw_ctx *hctx;
-	unsigned int i;
-	bool rcu = false;
-
  	blk_mq_quiesce_queue_nowait(q);
-	queue_for_each_hw_ctx(q, hctx, i) {
-		if (hctx->flags & BLK_MQ_F_BLOCKING)
-			synchronize_srcu(hctx->srcu);
-		else
-			rcu = true;
-	}
-	if (rcu)
+	if (q->tag_set->flags & BLK_MQ_F_BLOCKING) {
+		percpu_ref_kill(&q->dispatch_counter);
+		wait_event(q->mq_quiesce_wq,
+				percpu_ref_is_zero(&q->dispatch_counter));
+	} else
  		synchronize_rcu();
  }
 
 
 
+static void hctx_lock(struct blk_mq_hw_ctx *hctx)
  {
-	if (!(hctx->flags & BLK_MQ_F_BLOCKING)) {
-		/* shut up gcc false positive */
-		*srcu_idx = 0;
+	if (!(hctx->flags & BLK_MQ_F_BLOCKING))
  		rcu_read_lock();
-	} else
-		*srcu_idx = srcu_read_lock(hctx->srcu);
+	else
Quoted 8 lines

block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713

Quoted 2 lines
block/blk-mq.c +1 -0
+		percpu_ref_get(&hctx->queue->dispatch_counter);
  }

percpu_ref_get() will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observe blk_queue_quiesced(). That doesn't require that there are no current dispatchers.

IMO it shouldn't be one issue in reality, because:

- when dispatch can't make progress, the submission side will finally stop because we either run queue from submission side or request completion
- submission side stops because we always have very limited requests

Quoted 6 lines
Quoted 2 lines

- completion side stops because requests queued to device is limited too

I don't think that any requests should pass after the kill was called, otherwise how can we safely quiesce if requests can come in after it?

What we guarantee is that no request can be queued to LLD after blk_mq_quiesce_queue returns.

With percpu_refcount, once percpu_ref_is_zero(&q->dispatch_counter) returns true, all code path can observe the QUIESCED flag reliably just like what SRCU does, so no any request can pass to LLD after blk_mq_quiesce_queue returns.

Quoted 5 lines
Quoted 3 lines

We still can handle this case by not dispatch in case that percpu_ref_tryget()

You meant tryget_live right?

Both works, but tryget_live could be better.

Thanks,
Ming