Skip to contentSkip to conversation

blk-mq: insert request not through ->queue_rq into sw/scheduler queue

PATCH RESEND Applied Ming Lei 3 participants 5 messages Aug 18, 2020 – Aug 19, 2020

Subsystems (2)
lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH RESEND 1/1 blk-mq: insert request not through ->queue_rq into sw/scheduler queue

c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list") supposed to add request which has been through ->queue_rq() to the hw queue dispatch list, however it adds request running out of budget or driver tag to hw queue too. This way basically bypasses request merge, and causes too many request dispatched to LLD, and system% is unnecessary increased.

Fixes this issue by adding request not through ->queue_rq into sw/scheduler queue, and this way is safe because no ->queue_rq is called on this request yet.

High %system can be observed on Azure storvsc device, and even soft lock is observed. This patch reduces %system during heavy sequential IO, meantime decreases soft lockup risk.

Fixes c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list")
Cc Christoph Hellwig <[email protected]>
Cc Bart Van Assche <[email protected]>
Cc Mike Snitzer <[email protected]>
---
 block/blk-mq.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
block/blk-mq.c +2 -1
index 5ac80bfac325..f50c38ccac3c 100644
@@ -2039,7 +2039,8 @@ static blk_status_t __blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx,
 	if (bypass_insert)
 		return BLK_STS_RESOURCE;
 
-	blk_mq_request_bypass_insert(rq, false, run_queue);
+	blk_mq_sched_insert_request(rq, false, run_queue, false);
+
 	return BLK_STS_OK;
 }
 
Signature · 1 line
2.25.2
maintainer Applied
#1 lore raw
Quoted 6 lines · Ming Lei

c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list") supposed to add request which has been through ->queue_rq() to the hw queue dispatch list, however it adds request running out of budget or driver tag to hw queue too. This way basically bypasses request merge, and causes too many request dispatched to LLD, and system% is unnecessary increased.

Quoted 7 lines

Fixes this issue by adding request not through ->queue_rq into sw/scheduler queue, and this way is safe because no ->queue_rq is called on this request yet.

High %system can be observed on Azure storvsc device, and even soft lock is observed. This patch reduces %system during heavy sequential IO, meantime decreases soft lockup risk.

Applied, thanks Ming.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Ming Lei

c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list") supposed to add request which has been through ->queue_rq() to the hw queue dispatch list, however it adds request running out of budget or driver tag to hw queue too. This way basically bypasses request merge, and causes too many request dispatched to LLD, and system% is unnecessary increased.

Fixes this issue by adding request not through ->queue_rq into sw/scheduler queue, and this way is safe because no ->queue_rq is called on this request yet.

Quoted 5 lines
Quoted 3 lines

High %system can be observed on Azure storvsc device, and even soft lock is observed. This patch reduces %system during heavy sequential IO, meantime decreases soft lockup risk.

Applied, thanks Ming.

Hmm, strikes me as strange that this is occurring given the direct insertion into blk-mq queue (bypassing scheduler) is meant to avoid 2 layers of IO merging when dm-mulipath is stacked on blk-mq path(s). The dm-mpath IO scheduler does all merging and underlying paths' blk-mq request_queues are meant to just dispatch the top-level's requests.

So this change concerns me. Feels like this design has broken down.

Could be that some other entry point was added for the __blk_mq_try_issue_directly() code? And it needs to be untangled away from the dm-multipath use-case?

Apologies for not responding to this patch until now.

Mike

author
#3 lore raw
Quoted 20 lines · Mike Snitzer
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Ming Lei

c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list") supposed to add request which has been through ->queue_rq() to the hw queue dispatch list, however it adds request running out of budget or driver tag to hw queue too. This way basically bypasses request merge, and causes too many request dispatched to LLD, and system% is unnecessary increased.

Fixes this issue by adding request not through ->queue_rq into sw/scheduler queue, and this way is safe because no ->queue_rq is called on this request yet.

Quoted 5 lines
Quoted 3 lines

High %system can be observed on Azure storvsc device, and even soft lock is observed. This patch reduces %system during heavy sequential IO, meantime decreases soft lockup risk.

Applied, thanks Ming.

Quoted 8 lines

Hmm, strikes me as strange that this is occurring given the direct insertion into blk-mq queue (bypassing scheduler) is meant to avoid 2 layers of IO merging when dm-mulipath is stacked on blk-mq path(s). The dm-mpath IO scheduler does all merging and underlying paths' blk-mq request_queues are meant to just dispatch the top-level's requests.

So this change concerns me. Feels like this design has broken down.

'bypass_insert' is 'true' when blk_insert_cloned_request() is called from device mapper code, so this patch doesn't affect dm.

Quoted 3 lines

Could be that some other entry point was added for the __blk_mq_try_issue_directly() code? And it needs to be untangled away from the dm-multipath use-case?

__blk_mq_try_issue_directly() can be called from blk-mq directly, that is the case this patch is addressing, if one request can't be queued to LLD because of running out of budget or driver tag, it should be added to scheduler queue for improving io merge, meantime we can avoid too many requests dispatched to hardware.

Thanks,
Ming

#4 lore raw
Quoted 27 lines · Ming Lei
Quoted 26 lines · Mike Snitzer
Quoted 11 lines · Jens Axboe
Quoted 10 lines · Ming Lei

c616cbee97ae ("blk-mq: punt failed direct issue to dispatch list") supposed to add request which has been through ->queue_rq() to the hw queue dispatch list, however it adds request running out of budget or driver tag to hw queue too. This way basically bypasses request merge, and causes too many request dispatched to LLD, and system% is unnecessary increased.

Fixes this issue by adding request not through ->queue_rq into sw/scheduler queue, and this way is safe because no ->queue_rq is called on this request yet.

Quoted 5 lines
Quoted 3 lines

High %system can be observed on Azure storvsc device, and even soft lock is observed. This patch reduces %system during heavy sequential IO, meantime decreases soft lockup risk.

Applied, thanks Ming.

Hmm, strikes me as strange that this is occurring given the direct insertion into blk-mq queue (bypassing scheduler) is meant to avoid 2 layers of IO merging when dm-mulipath is stacked on blk-mq path(s). The dm-mpath IO scheduler does all merging and underlying paths' blk-mq request_queues are meant to just dispatch the top-level's requests.

Quoted 5 lines
Quoted 2 lines

So this change concerns me. Feels like this design has broken down.

'bypass_insert' is 'true' when blk_insert_cloned_request() is called from device mapper code, so this patch doesn't affect dm.

Great.

Quoted 9 lines
Quoted 3 lines

Could be that some other entry point was added for the __blk_mq_try_issue_directly() code? And it needs to be untangled away from the dm-multipath use-case?

__blk_mq_try_issue_directly() can be called from blk-mq directly, that is the case this patch is addressing, if one request can't be queued to LLD because of running out of budget or driver tag, it should be added to scheduler queue for improving io merge, meantime we can avoid too many requests dispatched to hardware.

I see, so if retry is needed best to attempt merge again.

Thanks for the explanation.

Mike