blk-mq: implement queue quiesce via percpu_ref for BLK_MQ_F_BLOCKING
RFC PATCH 1/1 blk-mq: implement queue quiesce via percpu_ref for BLK_MQ_F_BLOCKING
In case of BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.
However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use percpu_ref to implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observe QUEUE_FLAG_QUIESCED once blk_mq_quiesce_queue() returns.
Also it becomes much easier to add interface of async queue quiesce.
--- block/blk-mq-sysfs.c | 2 - block/blk-mq.c | 94 +++++++++++++++++++----------------------- block/blk-sysfs.c | 6 ++- include/linux/blk-mq.h | 7 ---- include/linux/blkdev.h | 4 ++ 5 files changed, 51 insertions(+), 62 deletions(-)
block/blk-mq-sysfs.c +0 -2
@@ -38,8 +38,6 @@ static void blk_mq_hw_sysfs_release(struct kobject *kobj)cancel_delayed_work_sync (&hctx->run_work); -if (hctx->flags &BLK_MQ_F_BLOCKING ) -cleanup_srcu_struct (hctx->srcu);blk_free_flush_queue (hctx->fq);sbitmap_free (&hctx->ctx_map);free_cpumask_var (hctx->cpumask);
block/blk-mq.c +42 -52
215 lines not on this page · raw message
block/blk-sysfs.c +5 -1
@@ -914,9 +914,13 @@ static void blk_release_queue(struct kobject *kobj)blk_queue_free_zone_bitmaps (q); -if (queue_is_mq (q)) +if (queue_is_mq (q)) {blk_mq_release (q); +if (q->tag_set->flags &BLK_MQ_F_BLOCKING ) +percpu_ref_exit (&q->dispatch_counter); + } +blk_trace_shutdown (q);mutex_lock (&q->debugfs_mutex);debugfs_remove_recursive (q->debugfs_dir);
include/linux/blk-mq.h +0 -7
@@ -169,13 +169,6 @@ struct blk_mq_hw_ctx {* q->unused_hctx_list. */ struct list_head hctx_list; - -/** -* @srcu: Sleepable RCU. Use as lock when type of the hardware queue is -* blocking (BLK_MQ_F_BLOCKING). Must be the last member - see also -* blk_mq_hw_ctx_size(). -*/ -struct srcu_struct srcu[]; };/**
include/linux/blkdev.h +4 -0
@@ -567,6 +567,10 @@ struct request_queue {struct list_head tag_set_list;struct bio_set bio_split; +/* only used for BLK_MQ_F_BLOCKING */ +struct percpu_ref dispatch_counter; +wait_queue_head_t mq_quiesce_wq; +struct dentry *debugfs_dir;#ifdef CONFIG_BLK_DEBUG_FS
Signature · 1 line
2.25.2
Quoted 11 lines · Ming Lei
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.Also it becomes much easier to add interface of async queue quiesce.
BTW, no obvious IOPS difference is observed with this patch applied when running io on null_blk(blocking, submit_queues=32) in one dual-socket, 32cores system.
Thanks,
Ming
Quoted 11 lines
Quoted 11 lines
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.
Quoted 4 lines
Quoted 1 line
Also it becomes much easier to add interface of async queue quiesce.
BTW, no obvious IOPS difference is observed with this patch applied when running io on
null_blk(blocking, submit_queues=32)in one dual-socket, 32cores system.
Thanks Ming, can you test for non-blocking on the same setup?
I can test some reset storms during traffic.
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
Quoted 11 lines
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Also it becomes much easier to add interface of async queue quiesce.
BTW, no obvious IOPS difference is observed with this patch applied when running io on
null_blk(blocking, submit_queues=32)in one dual-socket, 32cores system.Thanks Ming, can you test for non-blocking on the same setup?
OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?
Thanks,
Ming
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.Also it becomes much easier to add interface of async queue quiesce.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
BTW, no obvious IOPS difference is observed with this patch applied when running io on
null_blk(blocking, submit_queues=32)in one dual-socket, 32cores system.Thanks Ming, can you test for non-blocking on the same setup?
OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Quoted 17 lines
Quoted 11 lines
Quoted 11 lines
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.Quoted 4 lines
Quoted 1 line
Also it becomes much easier to add interface of async queue quiesce.
BTW, no obvious IOPS difference is observed with this patch applied when running io on
null_blk(blocking, submit_queues=32)in one dual-socket, 32cores system.Thanks Ming, can you test for non-blocking on the same setup?
Quoted 9 lines
Quoted 2 lines
OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
rcu_read_lock and rcu_read_unlock has been proved as efficient enough, and I don't think percpu_refcount is better than it, so I'd suggest to not switch non-blocking into this way.
BTW, in case of blocking, one hctx may dispatch at most one request because there is only single .run_work, even though when .queue_rq() is slept, that said blk_mq_submit_bio() queues bio in sync style. This way won't be very efficient. So percpu_refcount should be good enough for blocking code path, but may not be well enough for non-blocking case.
Thanks,
Ming
Quoted 25 lines
Quoted 25 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.Also it becomes much easier to add interface of async queue quiesce.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
BTW, no obvious IOPS difference is observed with this patch applied when running io on
null_blk(blocking, submit_queues=32)in one dual-socket, 32cores system.Thanks Ming, can you test for non-blocking on the same setup?
OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
Quoted 6 lines
Quoted 2 lines
If the cost is negligible, I think the upside is worth it.
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.
It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to percpu_ref is negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.
Quoted 5 lines
BTW, in case of blocking, one hctx may dispatch at most one request because there is only single .run_work, even though when .queue_rq() is slept, that said
blk_mq_submit_bio()queues bio in sync style. This way won't be very efficient. Sopercpu_refcountshould be good enough for blocking code path, but may not be well enough for non-blocking case.
Not sure what you mean, the percpu_ref is taken exactly where rcu is taken, not sure what is the difference.
Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
Quoted 6 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 8 lines
Quoted 3 lines
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.
FWIW, I proposed an hctx percpu_ref over a year ago (but for a completely different reason), and it was measured as too costly.
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 8 lines
Quoted 3 lines
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.
Quoted 4 lines
FWIW, I proposed an hctx
percpu_refover a year ago (but for a completely different reason), and it was measured as too costly.https://lore.kernel.org/linux-block/[email protected]/
If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.
Quoted 17 lines · Sagi Grimberg
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
Quoted 7 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.
Quoted 8 lines
Quoted 4 lines
FWIW, I proposed an hctx
percpu_refover a year ago (but for a completely different reason), and it was measured as too costly.https://lore.kernel.org/linux-block/[email protected]/If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.
Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.
Jens Axboe
maintainerQuoted 23 lines · Keith Busch
Quoted 22 lines · Sagi Grimberg
Quoted 21 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 8 lines
Quoted 3 lines
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.FWIW, I proposed an hctx
percpu_refover a year ago (but for a completely different reason), and it was measured as too costly.https://lore.kernel.org/linux-block/[email protected]/
Quoted 7 lines
Quoted 3 lines
If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.
Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.
I fully agree with that point.
Signature · 1 line
Jens Axboe
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 8 lines
Quoted 3 lines
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.Quoted 4 lines
FWIW, I proposed an hctx
percpu_refover a year ago (but for a completely different reason), and it was measured as too costly.https://lore.kernel.org/linux-block/[email protected]/If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.
Quoted 5 lines
Quoted 3 lines
Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.
I fully agree with that point.
I also agree, just said we should use the same mechanisms, IFF its not
expensive. But I'm concerned that if we use completely different mechanisms we are likely to get wrong assumptions and break one at some point.
Hence my suggestion to move back to srcu and place the rcu_head in the hctx.
Quoted 30 lines · Sagi Grimberg
Quoted 30 lines
Quoted 29 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 11 lines
Quoted 7 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.Quoted 8 lines
Quoted 4 lines
FWIW, I proposed an hctx
percpu_refover a year ago (but for a completely different reason), and it was measured as too costly.https://lore.kernel.org/linux-block/[email protected]/If this is the case, we shouldn't consider this as an alternative at all, and move forward with either the original proposal or what ming proposed to move a counter to the tagset.
Well, the point I was trying to make is that we shouldn't bother making blocking and non-blocking dispatchers use the same synchronization since non-blocking has a very cheap solution that blocking can't use.
Quoted 8 lines
Quoted 1 line
I fully agree with that point.
I also agree, just said we should use the same mechanisms, IFF its not expensive. But I'm concerned that if we use completely different mechanisms we are likely to get wrong assumptions and break one at some point.
Hence my suggestion to move back to srcu and place the
rcu_headin the hctx.
SRCU has been different enough compared with RCU, either implementation or API interface.
Then I'd still suggest to replace SRCU with percpu refcount. Then we can have a simpler quiesce implementation.
Thanks,
Ming
Quoted 17 lines · Keith Busch
Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
Quoted 6 lines
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 8 lines
Quoted 3 lines
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.
Quoted 4 lines
FWIW, I proposed an hctx
percpu_refover a year ago (but for a completely different reason), and it was measured as too costly.
That is why I don't want to switch non-blocking to percpu-refcount.
However, cost of srcu read lock/unlock is basically similar with percpu ref.
Thanks,
Ming
Quoted 28 lines · Sagi Grimberg
Quoted 28 lines
Quoted 27 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
In case of
BLK_MQ_F_BLOCKING, blk-mq uses SRCU to mark read critical section during dispatching request, then request queue quiesce is based on SRCU. What we want to get is low cost added in fast path.However, from srcu_read_lock/srcu_read_unlock implementation, not see it is quicker than percpu refcount, so use
percpu_refto implement queue quiesce. This usage is cleaner and simpler & enough for implementing queue quiesce. The main requirement is to make sure all read sections to observeQUEUE_FLAG_QUIESCEDonceblk_mq_quiesce_queue()returns.Also it becomes much easier to add interface of async queue quiesce.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
BTW, no obvious IOPS difference is observed with this patch applied when running io on
null_blk(blocking, submit_queues=32)in one dual-socket, 32cores system.Thanks Ming, can you test for non-blocking on the same setup?
OK, I can do that, but this patch supposes to not affect non-blocking, care to share your motivation for testing non-blocking?
I think it will be a significant improvement to have a single code path. The code will be more robust and we won't need to face issues that are specific for blocking.
If the cost is negligible, I think the upside is worth it.
Quoted 8 lines
Quoted 3 lines
rcu_read_lockandrcu_read_unlockhas been proved as efficient enough, and I don't thinkpercpu_refcountis better than it, so I'd suggest to not switch non-blocking into this way.It's not a matter of which is better, its a matter of making the code more robust because it has a single code-path. If moving to
percpu_refis negligible, I would suggest to move both, I don't want to have two completely different mechanism for blocking vs. non-blocking.
RCU and SRCU have been different mechanism already.
Quoted 9 lines
Quoted 6 lines
BTW, in case of blocking, one hctx may dispatch at most one request because there is only single .run_work, even though when .queue_rq() is slept, that said
blk_mq_submit_bio()queues bio in sync style. This way won't be very efficient. Sopercpu_refcountshould be good enough for blocking code path, but may not be well enough for non-blocking case.Not sure what you mean, the
percpu_refis taken exactly where rcu is taken, not sure what is the difference.
My point is that blocking can't be efficient enough, when .queue_rq() is slept, no any request can be queued to this hctx any more.
thanks,
Ming
On 28/07/2020 15:50, Ming Lei wrote:
[...]
Quoted 4 lines
block/blk-mq.c +1
-2-static void hctx_lock (struct blk_mq_hw_ctx *hctx,int *srcu_idx) - __acquires(hctx->srcu) +static void hctx_lock (struct blk_mq_hw_ctx *hctx) {
block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713
Quoted 10 lines
block/blk-mq.c +3
-5-if (!(hctx->flags &BLK_MQ_F_BLOCKING )) { -/* shut up gcc false positive */ - *srcu_idx =0 ; +if (!(hctx->flags &BLK_MQ_F_BLOCKING ))rcu_read_lock (); - }else - *srcu_idx =srcu_read_lock (hctx->srcu); +else +percpu_ref_get (&hctx->queue->dispatch_counter); }
I quite like this because it hides the internals of hctx_lock() (rcu vs srcu) from the callers.
Quoted 11 lines
/**
block/blk-mq.c +0
-2@@ -1486,8 +1479,6 @@ bool blk_mq_dispatch_rq_list(struct blk_mq_hw_ctx *hctx, struct list_head *list,*/ static void __blk_mq_run_hw_queue (struct blk_mq_hw_ctx *hctx) { -int srcu_idx; -/* * We should be running this queue from one of the CPUs that * are mapped to it.
block/blk-mq.c static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx) · line 1517
Quoted 8 lines
block/blk-mq.c +2
-2@@ -1521,9 +1512,9 @@ static void __blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx)might_sleep_if (hctx->flags &BLK_MQ_F_BLOCKING ); -hctx_lock (hctx, &srcu_idx); +hctx_lock (hctx);blk_mq_sched_dispatch_requests (hctx); -hctx_unlock (hctx, srcu_idx); +hctx_unlock (hctx);
blk_mq_sched_dispatch_requests also has this comment at the beginning:
/* RCU or SRCU read lock is needed before checking quiesced flag */ if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)))
I think the SRCU part needs to be changed to percpu_ref or the whole comment should probably just mention the hctx_lock().
Quoted 27 lines · Ming Lei
void
blk_mq_quiesce_queue(struct request_queue *q){block/blk-mq.c +6
-11-struct blk_mq_hw_ctx *hctx; -unsigned int i; -bool rcu = false; -blk_mq_quiesce_queue_nowait (q); -queue_for_each_hw_ctx (q, hctx, i) { -if (hctx->flags &BLK_MQ_F_BLOCKING ) -synchronize_srcu (hctx->srcu); -else - rcu = true; - } -if (rcu) +if (q->tag_set->flags &BLK_MQ_F_BLOCKING ) { +percpu_ref_kill (&q->dispatch_counter); +wait_event (q->mq_quiesce_wq, +percpu_ref_is_zero (&q->dispatch_counter)); + }else synchronize_rcu (); } +static void hctx_lock (struct blk_mq_hw_ctx *hctx) {
block/blk-mq.c EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_nowait);
Quoted 10 lines
block/blk-mq.c +3
-5-if (!(hctx->flags &BLK_MQ_F_BLOCKING )) { -/* shut up gcc false positive */ - *srcu_idx =0 ; +if (!(hctx->flags &BLK_MQ_F_BLOCKING ))rcu_read_lock (); - }else - *srcu_idx =srcu_read_lock (hctx->srcu); +else +percpu_ref_get (&hctx->queue->dispatch_counter); }
percpu_ref_get() will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observe blk_queue_quiesced(). That doesn't require that there are no current dispatchers.
Quoted 36 lines · Keith Busch
Quoted 35 lines · Ming Lei
void
blk_mq_quiesce_queue(struct request_queue *q){block/blk-mq.c +8
-16-struct blk_mq_hw_ctx *hctx; -unsigned int i; -bool rcu = false; -blk_mq_quiesce_queue_nowait (q); -queue_for_each_hw_ctx (q, hctx, i) { -if (hctx->flags &BLK_MQ_F_BLOCKING ) -synchronize_srcu (hctx->srcu); -else - rcu = true; - } -if (rcu) +if (q->tag_set->flags &BLK_MQ_F_BLOCKING ) { +percpu_ref_kill (&q->dispatch_counter); +wait_event (q->mq_quiesce_wq, +percpu_ref_is_zero (&q->dispatch_counter)); + }else synchronize_rcu (); } +static void hctx_lock (struct blk_mq_hw_ctx *hctx) { -if (!(hctx->flags &BLK_MQ_F_BLOCKING )) { -/* shut up gcc false positive */ - *srcu_idx =0 ; +if (!(hctx->flags &BLK_MQ_F_BLOCKING ))rcu_read_lock (); - }else - *srcu_idx =srcu_read_lock (hctx->srcu); +else
Quoted 8 lines
block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713
Quoted 2 lines
block/blk-mq.c +1
-0+percpu_ref_get (&hctx->queue->dispatch_counter); }
percpu_ref_get()will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observeblk_queue_quiesced(). That doesn't require that there are no current dispatchers.
IMO it shouldn't be one issue in reality, because:
- when dispatch can't make progress, the submission side will finally stop because we either run queue from submission side or request completion
- submission side stops because we always have very limited requests
- completion side stops because requests queued to device is limited too
We still can handle this case by not dispatch in case that percpu_ref_tryget() returns false, which will change the usage into the following way:
if (hctx_lock(hctx)) {
blk_mq_sched_dispatch_requests(hctx);
hctx_unlock(hctx);
}And __blk_mq_try_issue_directly() needs a bit special treatment because the request has to be inserted to queue after queue becomes quiesced.
Thanks,
Ming
Quoted 47 lines
Quoted 36 lines
Quoted 36 lines
void blk_mq_quiesce_queue(struct request_queue *q) {block/blk-mq.c +8
-16-struct blk_mq_hw_ctx *hctx; -unsigned int i; -bool rcu = false; -blk_mq_quiesce_queue_nowait (q); -queue_for_each_hw_ctx (q, hctx, i) { -if (hctx->flags &BLK_MQ_F_BLOCKING ) -synchronize_srcu (hctx->srcu); -else - rcu = true; - } -if (rcu) +if (q->tag_set->flags &BLK_MQ_F_BLOCKING ) { +percpu_ref_kill (&q->dispatch_counter); +wait_event (q->mq_quiesce_wq, +percpu_ref_is_zero (&q->dispatch_counter)); + }else synchronize_rcu (); } +static void hctx_lock (struct blk_mq_hw_ctx *hctx) { -if (!(hctx->flags &BLK_MQ_F_BLOCKING )) { -/* shut up gcc false positive */ - *srcu_idx =0 ; +if (!(hctx->flags &BLK_MQ_F_BLOCKING ))rcu_read_lock (); - }else - *srcu_idx =srcu_read_lock (hctx->srcu); +else Quoted 8 lines
block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713
Quoted 2 lines
block/blk-mq.c +1
-0+percpu_ref_get (&hctx->queue->dispatch_counter); }
percpu_ref_get()will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observeblk_queue_quiesced(). That doesn't require that there are no current dispatchers.IMO it shouldn't be one issue in reality, because:
Quoted 8 lines
- when dispatch can't make progress, the submission side will finally stop because we either run queue from submission side or request completion
- submission side stops because we always have very limited requests
- completion side stops because requests queued to device is limited too
I don't think that any requests should pass after the kill was called, otherwise how can we safely quiesce if requests can come in after it?
Quoted 2 lines
We still can handle this case by not dispatch in case that
percpu_ref_tryget()
You meant tryget_live right?
Quoted 9 lines
returns false, which will change the usage into the following way:
if (hctx_lock(hctx)) { blk_mq_sched_dispatch_requests(hctx); hctx_unlock(hctx); }And
__blk_mq_try_issue_directly()needs a bit special treatment because the request has to be inserted to queue after queue becomes quiesced.
Agreed.
Quoted 51 lines · Sagi Grimberg
Quoted 51 lines
Quoted 35 lines
Quoted 35 lines
void blk_mq_quiesce_queue(struct request_queue *q) {block/blk-mq.c +8
-16-struct blk_mq_hw_ctx *hctx; -unsigned int i; -bool rcu = false; -blk_mq_quiesce_queue_nowait (q); -queue_for_each_hw_ctx (q, hctx, i) { -if (hctx->flags &BLK_MQ_F_BLOCKING ) -synchronize_srcu (hctx->srcu); -else - rcu = true; - } -if (rcu) +if (q->tag_set->flags &BLK_MQ_F_BLOCKING ) { +percpu_ref_kill (&q->dispatch_counter); +wait_event (q->mq_quiesce_wq, +percpu_ref_is_zero (&q->dispatch_counter)); + }else synchronize_rcu (); } +static void hctx_lock (struct blk_mq_hw_ctx *hctx) { -if (!(hctx->flags &BLK_MQ_F_BLOCKING )) { -/* shut up gcc false positive */ - *srcu_idx =0 ; +if (!(hctx->flags &BLK_MQ_F_BLOCKING ))rcu_read_lock (); - }else - *srcu_idx =srcu_read_lock (hctx->srcu); +else Quoted 8 lines
block/blk-mq.c void blk_mq_complete_request(struct request *rq) · line 713
Quoted 2 lines
block/blk-mq.c +1
-0+percpu_ref_get (&hctx->queue->dispatch_counter); }
percpu_ref_get()will always succeed, even after quiesce kills it. Isn't it possible that 'percpu_ref_is_zero(&q->dispatch_counter))' may never reach 0? We only need to ensure that dispatchers will observeblk_queue_quiesced(). That doesn't require that there are no current dispatchers.IMO it shouldn't be one issue in reality, because:
- when dispatch can't make progress, the submission side will finally stop because we either run queue from submission side or request completion
- submission side stops because we always have very limited requests
Quoted 6 lines
Quoted 2 lines
- completion side stops because requests queued to device is limited too
I don't think that any requests should pass after the kill was called, otherwise how can we safely quiesce if requests can come in after it?
What we guarantee is that no request can be queued to LLD after blk_mq_quiesce_queue returns.
With percpu_refcount, once percpu_ref_is_zero(&q->dispatch_counter) returns true, all code path can observe the QUIESCED flag reliably just like what SRCU does, so no any request can pass to LLD after blk_mq_quiesce_queue returns.
Quoted 5 lines
Quoted 3 lines
We still can handle this case by not dispatch in case that
percpu_ref_tryget()You meant
tryget_liveright?
Both works, but tryget_live could be better.
Thanks,
Ming