block: make rewriting the current I/O scheduler cheap again
cover block: make rewriting the current I/O scheduler cheap again
Writing the scheduler a queue already uses to queue/scheduler takes about 10 ms and freezes the queue, which stalls I/O on live disks every time a udev rule or tuning tool sets the scheduler unconditionally.
Patch 1 stops requesting a "none-iosched" module on every write of "none". Patch 2 returns before freezing the queue when the elevator wouldn't change.
On a null_blk disk (median of 20 writes, all kernels measured in the same run):
operation base (ms) patch 1 (ms) patch 1+2 (ms)
none -> none 13.09 9.06 0.08 mq-deadline -> mq-deadline 9.19 9.14 0.08 mq-deadline -> none 12.00 7.96 7.97
---
Qiliang Yuan (2):
block: don't request a module for the none elevator
block: return early when the elevator doesn't change
block/elevator.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
---
base-commit: 79eb6679ef6a0d5ea68da53a8321fb1dc72299b9
change-id: 20261111-bug-block-elevator-unchanged-a6cd2d761d38
Best regards,
Signature · 1 line
Qiliang Yuan <[email protected]>
PATCH 1/2 block: don't request a module for the none elevator
elv_iosched_store() calls elv_iosched_load_module() for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up in request_module("none-iosched").
That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.
Return early for "none", like elevator_change() did before the module loading moved into elv_iosched_load_module().
Writing queue/scheduler of a null_blk disk 20 times takes (median):
operation before (ms) after (ms) change
none -> none 13.09 9.06 -31% mq-deadline -> none 11.99 7.96 -34%
--- block/elevator.c | 3 +++ 1 file changed, 3 insertions(+)
block/elevator.c +3 -0
@@ -775,6 +775,9 @@ static void elv_iosched_load_module(const char *elevator_name) {struct elevator_type *found; +if (!strncmp (elevator_name,"none" ,4 )) +return ; +spin_lock (&elv_list_lock); found =__elevator_find (elevator_name);spin_unlock (&elv_list_lock);
Signature · 1 line
2.43.0
Re: [PATCH 1/2] block: don't request a module for the none elevator
Looks good:
Re: [PATCH 1/2] block: don't request a module for the none elevator
In reply to Qiliang YuanQuoted 12 lines · Qiliang Yuan
elv_iosched_store()callselv_iosched_load_module()for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up inrequest_module("none-iosched").That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.
Return early for "none", like
elevator_change()did before the module loading moved intoelv_iosched_load_module().
Quoted 8 lines
Writing queue/scheduler of a
null_blkdisk 20 times takes (median):operation before (ms) after (ms) change none -> none 13.09 9.06 -31% mq-deadline -> none 11.99 7.96 -34%Fixes 734e1a8603128 ("block: Prevent deadlocks when switching elevators")Signed-off-by Qiliang Yuan<[email protected]>
Looks good to me:
PATCH 2/2 block: return early when the elevator doesn't change
In reply to Qiliang YuanWriting the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.
Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.
Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. Elevator changes of a queue are already serialized by QUEUE_FLAG_NO_ELV_SWITCH, registration and update_nr_hwq_lock, so drop the check under the freeze.
Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):
operation before (ms) after (ms) change
none -> none 9.06 0.08 -99% mq-deadline -> mq-deadline 9.14 0.08 -99%
4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:
metric before after change
IOPS 342609 395915 +16%
max latency (us) 10839 978 -91%
--- block/elevator.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
block/elevator.c +15 -2
@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) {unsigned int memflags;struct blk_mq_tag_set *set = q->tag_set; +bool unchanged;int ret =0 ;lockdep_assert_held (&set->update_nr_hwq_lock); +/* +* Callers serialize elevator changes of this queue, so q->elevator is +* stable here. Skip freezing the queue if nothing would change. +*/ +mutex_lock (&q->elevator_lock); +if (q->elevator) + unchanged =elevator_match (q->elevator->type, ctx->name); +else + unchanged = !strncmp (ctx->name,"none" ,4 ); +mutex_unlock (&q->elevator_lock); +if (unchanged) +return 0 ; +if (strncmp (ctx->name,"none" ,4 )) { ret =blk_mq_alloc_sched_res (q, ctx->type, &ctx->res, set->nr_hw_queues);
@@ -677,8 +691,7 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)*/ blk_mq_cancel_work_sync (q);mutex_lock (&q->elevator_lock); -if (!(q->elevator &&elevator_match (q->elevator->type, ctx->name))) - ret =elevator_switch (q, ctx); + ret =elevator_switch (q, ctx);mutex_unlock (&q->elevator_lock);blk_mq_unfreeze_queue (q, memflags);if (!ret)
Signature · 1 line
2.43.0
Re: [PATCH 2/2] block: return early when the elevator doesn't change
Quoted 50 lines · Qiliang Yuan
Writing the scheduler a queue already uses to queue/scheduler used to be a no-op.
elevator_change()now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue inelevator_switch()and resetsnr_requests.Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.
Check under
elevator_lockwhether anything would change before allocating scheduler resources and freezing the queue. Elevator changes of a queue are already serialized byQUEUE_FLAG_NO_ELV_SWITCH, registration andupdate_nr_hwq_lock, so drop the check under the freeze.Writing the current scheduler to queue/scheduler of a
null_blkdisk 20 times takes (median):operation before (ms) after (ms) change
none -> none 9.06 0.08 -99% mq-deadline -> mq-deadline 9.14 0.08 -99%4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:
metric before after change
IOPS 342609 395915 +16%
max latency (us) 10839 978 -91%Fixes b07a889e83355 ("block: move q->sysfs_lock and queue-freeze under show/store method")Signed-off-by Qiliang Yuan <[email protected]>--- block/elevator.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)block/elevator.c +3
-0@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) {unsigned int memflags;struct blk_mq_tag_set *set = q->tag_set; +bool unchanged;int ret =0 ;lockdep_assert_held (&set->update_nr_hwq_lock); +/* +* Callers serialize elevator changes of this queue, so q->elevator is
block/elevator.c static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) · line 673
Quoted 10 lines
block/elevator.c +10
-0+* stable here. Skip freezing the queue if nothing would change. +*/ +mutex_lock (&q->elevator_lock); +if (q->elevator) + unchanged =elevator_match (q->elevator->type, ctx->name); +else + unchanged = !strncmp (ctx->name,"none" ,4 ); +mutex_unlock (&q->elevator_lock); +if (unchanged) +return 0 ;
Strictly speaking this is racy with another thread changing the elevator. I don't think we actually care about that, but maybe add a comment to that effect?
block/elevator.c static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) · line 694
Quoted 12 lines
block/elevator.c +1
-0+if (strncmp (ctx->name,"none" ,4 )) { ret = blk_mq_alloc_sched_res(q, ctx->type, &ctx->res,block/elevator.c +1
-2set->nr_hw_queues);@@ -677,8 +691,7 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)*/ blk_mq_cancel_work_sync (q);mutex_lock (&q->elevator_lock); -if (!(q->elevator &&elevator_match (q->elevator->type, ctx->name))) - ret =elevator_switch (q, ctx); + ret =elevator_switch (q, ctx);
However, the raciness means we still need to keep this check I think.
Nit, but I think the Subject should say "change" instead of "rewrite".