Skip to contentSkip to conversation

blk-mq: order adding requests to hctx->dispatch and checking SCHED_RESTART

PATCH RESEND Applied Ming Lei 4 participants 5 messages Aug 17, 2020 – Aug 26, 2020

Subsystems (2)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH RESEND 1/1 blk-mq: order adding requests to hctx->dispatch and checking SCHED_RESTART

SCHED_RESTART code path is relied to re-run queue for dispatch requests in hctx->dispatch. Meantime the SCHED_RSTART flag is checked when adding requests to hctx->dispatch.

memory barriers have to be used for ordering the following two pair of OPs:

1) adding requests to hctx->dispatch and checking SCHED_RESTART in blk_mq_dispatch_rq_list()

2) clearing SCHED_RESTART and checking if there is request in hctx->dispatch in blk_mq_sched_restart().

Without the added memory barrier, either:

1) blk_mq_sched_restart() may miss requests added to hctx->dispatch meantime blk_mq_dispatch_rq_list() observes SCHED_RESTART, and not run queue in dispatch side

or

2) blk_mq_dispatch_rq_list still sees SCHED_RESTART, and not run queue in dispatch side, meantime checking if there is request in hctx->dispatch from blk_mq_sched_restart() is missed.

IO hang in ltp/fs_fill test is reported by kernel test robot:

	https://lkml.org/lkml/2020/7/26/77

Turns out it is caused by the above out-of-order OPs. And the IO hang can't be observed any more after applying this patch.

Cc Bart Van Assche <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc David Jeffery <[email protected]>
---
 block/blk-mq-sched.c | 9 +++++++++
 block/blk-mq.c       | 9 +++++++++
 2 files changed, 18 insertions(+)
block/blk-mq-sched.c +9 -0
index a19cdf159b75..d2790e5b06d1 100644
@@ -78,6 +78,15 @@ void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx)
 		return;
 	clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
 
+	/*
+	 * Order clearing SCHED_RESTART and list_empty_careful(&hctx->dispatch)
+	 * in blk_mq_run_hw_queue(). Its pair is the barrier in
+	 * blk_mq_dispatch_rq_list(). So dispatch code won't see SCHED_RESTART,
+	 * meantime new request added to hctx->dispatch is missed to check in
+	 * blk_mq_run_hw_queue().
+	 */
+	smp_mb();
+
 	blk_mq_run_hw_queue(hctx, true);
 }
 
block/blk-mq.c +9 -0
index 0015a1892153..6c1c3ad175a9 100644
@@ -1437,6 +1437,15 @@ bool blk_mq_dispatch_rq_list(struct blk_mq_hw_ctx *hctx, struct list_head *list,
 		list_splice_tail_init(list, &hctx->dispatch);
 		spin_unlock(&hctx->lock);
 
+		/*
+		 * Order adding requests to hctx->dispatch and checking
+		 * SCHED_RESTART flag. The pair of this smp_mb() is the one
+		 * in blk_mq_sched_restart(). Avoid restart code path to
+		 * miss the new added requests to hctx->dispatch, meantime
+		 * SCHED_RESTART is observed here.
+		 */
+		smp_mb();
+
 		/*
 		 * If SCHED_RESTART was set by the caller of this function and
 		 * it is no longer set that means that it was cleared by another
Signature · 1 line
2.25.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 31 lines · Ming Lei

SCHED_RESTART code path is relied to re-run queue for dispatch requests in hctx->dispatch. Meantime the SCHED_RSTART flag is checked when adding requests to hctx->dispatch.

memory barriers have to be used for ordering the following two pair of OPs:

1) adding requests to hctx->dispatch and checking SCHED_RESTART in blk_mq_dispatch_rq_list()

2) clearing SCHED_RESTART and checking if there is request in hctx->dispatch in blk_mq_sched_restart().

Without the added memory barrier, either:

1) blk_mq_sched_restart() may miss requests added to hctx->dispatch meantime blk_mq_dispatch_rq_list() observes SCHED_RESTART, and not run queue in dispatch side

or

2) blk_mq_dispatch_rq_list still sees SCHED_RESTART, and not run queue in dispatch side, meantime checking if there is request in hctx->dispatch from blk_mq_sched_restart() is missed.

IO hang in ltp/fs_fill test is reported by kernel test robot:

	https://lkml.org/lkml/2020/7/26/77

Turns out it is caused by the above out-of-order OPs. And the IO hang can't be observed any more after applying this patch.

Quoted 6 lines
Cc Bart Van Assche <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc David Jeffery <[email protected]>

Can you add a Fixes: tag so that the commit gets backported?

Otherwise looks good:

author
#2 lore raw
Quoted 32 lines · Christoph Hellwig
Quoted 31 lines · Ming Lei

SCHED_RESTART code path is relied to re-run queue for dispatch requests in hctx->dispatch. Meantime the SCHED_RSTART flag is checked when adding requests to hctx->dispatch.

memory barriers have to be used for ordering the following two pair of OPs:

1) adding requests to hctx->dispatch and checking SCHED_RESTART in blk_mq_dispatch_rq_list()

2) clearing SCHED_RESTART and checking if there is request in hctx->dispatch in blk_mq_sched_restart().

Without the added memory barrier, either:

1) blk_mq_sched_restart() may miss requests added to hctx->dispatch meantime blk_mq_dispatch_rq_list() observes SCHED_RESTART, and not run queue in dispatch side

or

2) blk_mq_dispatch_rq_list still sees SCHED_RESTART, and not run queue in dispatch side, meantime checking if there is request in hctx->dispatch from blk_mq_sched_restart() is missed.

IO hang in ltp/fs_fill test is reported by kernel test robot:

	https://lkml.org/lkml/2020/7/26/77

Turns out it is caused by the above out-of-order OPs. And the IO hang can't be observed any more after applying this patch.

Quoted 8 lines
Quoted 6 lines
Cc Bart Van Assche <[email protected]>
Cc Christoph Hellwig <[email protected]>
Cc David Jeffery <[email protected]>

Can you add a Fixes: tag so that the commit gets backported?

Fixes bd166ef183c2 ("blk-mq-sched: add framework for MQ capable IO schedulers")

Thanks,
Ming

maintainer Applied
#3 lore raw
In reply to Ming Lei
Quoted 24 lines · Ming Lei

SCHED_RESTART code path is relied to re-run queue for dispatch requests in hctx->dispatch. Meantime the SCHED_RSTART flag is checked when adding requests to hctx->dispatch.

memory barriers have to be used for ordering the following two pair of OPs:

1) adding requests to hctx->dispatch and checking SCHED_RESTART in blk_mq_dispatch_rq_list()

2) clearing SCHED_RESTART and checking if there is request in hctx->dispatch in blk_mq_sched_restart().

Without the added memory barrier, either:

1) blk_mq_sched_restart() may miss requests added to hctx->dispatch meantime blk_mq_dispatch_rq_list() observes SCHED_RESTART, and not run queue in dispatch side

or

2) blk_mq_dispatch_rq_list still sees SCHED_RESTART, and not run queue in dispatch side, meantime checking if there is request in hctx->dispatch from blk_mq_sched_restart() is missed.

Quoted 6 lines

IO hang in ltp/fs_fill test is reported by kernel test robot:

	https://lkml.org/lkml/2020/7/26/77

Turns out it is caused by the above out-of-order OPs. And the IO hang can't be observed any more after applying this patch.

Applied, thanks.

Signature · 1 line
Jens Axboe
maintainer
#4 lore raw
In reply to Ming Lei

Hi

[This is an automated email]

This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: all

The bot has tested the following trees: v5.8.2, v5.7.16, v5.4.59, v4.19.140, v4.14.193, v4.9.232, v4.4.232.

v5.8.2: Build OK!
v5.7.16: Build OK!
v5.4.59: Build OK!
v4.19.140: Build OK!
v4.14.193: Failed to apply! Possible dependencies:
1f460b63d4b3 ("blk-mq: don't restart queue when .get_budget returns BLK_STS_RESOURCE") 358a3a6bccb7 ("blk-mq: don't handle TAG_SHARED in restart") 97889f9ac24f ("blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()") b347689ffbca ("blk-mq-sched: improve dispatching from sw queue") caf8eb0d604a ("blk-mq-sched: move actual dispatching into one helper") de1482974080 ("blk-mq: introduce .get_budget and .put_budget in blk_mq_ops")

v4.9.232: Failed to apply! Possible dependencies:
8e8320c9315c ("blk-mq: fix performance regression with shared tags") 97889f9ac24f ("blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()") bd166ef183c2 ("blk-mq-sched: add framework for MQ capable IO schedulers") cf43e6be865a ("block: add scalable completion tracking of requests") e806402130c9 ("block: split out request-only flags into a new namespace") f1ba82616c33 ("blk-mq: pass bio to blk_mq_sched_get_rq_priv")

v4.4.232: Failed to apply! Possible dependencies:
511cbce2ff8b ("irq_poll: make blk-iopoll available outside the block layer") 6f3b0e8bcf3c ("blk-mq: add a flags parameter to blk_mq_alloc_request") 88459642cba4 ("blk-mq: abstract tag allocation out into sbitmap library") 8e8320c9315c ("blk-mq: fix performance regression with shared tags") 9467f85960a3 ("blk-mq/cpu-notif: Convert to new hotplug state machine") 97889f9ac24f ("blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()") 9e0e252a048b ("badblocks: Add core badblock management code") bd166ef183c2 ("blk-mq-sched: add framework for MQ capable IO schedulers") cf43e6be865a ("block: add scalable completion tracking of requests") e57690fe009b ("blk-mq: don't overwrite rq->mq_ctx")
f1ba82616c33 ("blk-mq: pass bio to blk_mq_sched_get_rq_priv")

NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

Signature · 2 lines
Thanks
Sasha