block: make changing to the current I/O scheduler cheap again
v3 cover block: make changing to the current I/O scheduler cheap again
Writing the scheduler a queue already uses to queue/scheduler takes about 10 ms and freezes the queue, which stalls I/O on live disks every time a udev rule or tuning tool sets the scheduler unconditionally.
Patch 1 stops requesting a "none-iosched" module on every write of "none". Patch 2 returns before freezing the queue when the elevator wouldn't change.
On a null_blk disk (median of 20 writes, all kernels measured in the same run):
operation base (ms) patch 1 (ms) patch 1+2 (ms)
none -> none 13.10 9.11 0.08 mq-deadline -> mq-deadline 10.95 9.05 0.08 mq-deadline -> none 13.00 7.97 6.03
---
V2 -> V3:
- Point Fixes in patch 2 at the commit that added the freeze, as Christoph pointed out
- Add Christoph's Reviewed-by to patch 2
- Rebase onto v7.4-rc2-1025-g666539eecdf4
V1 -> V2:
- Keep the elevator check under the freeze and note the race in the comment, as Christoph suggested
- Say "change" instead of "rewrite" in the cover letter subject
- Rebase onto v7.4-rc2-470-g7e1392a42ea2 and update the numbers from a new run
v2: https://lore.kernel.org/r/20261114-bug-block-elevator-unchanged-v2-0-5a1fe7d72fa0@linux.dev
v1: https://lore.kernel.org/r/20261111-bug-block-elevator-unchanged-v1-0-583198bd4e21@linux.dev
---
Qiliang Yuan (2):
block: don't request a module for the none elevator
block: return early when the elevator doesn't change
block/elevator.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
---
base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
change-id: 20261111-bug-block-elevator-unchanged-a6cd2d761d38
Best regards,
Signature · 1 line
Qiliang Yuan <[email protected]>
PATCH v3 1/2 block: don't request a module for the none elevator
elv_iosched_store() calls elv_iosched_load_module() for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up in request_module("none-iosched").
That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.
Return early for "none", like elevator_change() did before the module loading moved into elv_iosched_load_module().
Writing queue/scheduler of a null_blk disk 20 times takes (median):
operation before (ms) after (ms) change
none -> none 13.10 9.11 -30% mq-deadline -> none 13.00 7.97 -39%
--- block/elevator.c | 3 +++ 1 file changed, 3 insertions(+)
block/elevator.c +3 -0
@@ -775,6 +775,9 @@ static void elv_iosched_load_module(const char *elevator_name) {struct elevator_type *found; +if (!strncmp (elevator_name,"none" ,4 )) +return ; +spin_lock (&elv_list_lock); found =__elevator_find (elevator_name);spin_unlock (&elv_list_lock);
Signature · 1 line
2.43.0
PATCH v3 2/2 block: return early when the elevator doesn't change
In reply to Qiliang YuanWriting the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.
Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.
Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. The check can race with another elevator change, which at worst leads to a redundant switch, so keep the one done under the freeze as well.
Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):
operation before (ms) after (ms) change
none -> none 9.11 0.08 -99% mq-deadline -> mq-deadline 9.05 0.08 -99%
4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:
metric before after change
IOPS 340562 390392 +15%
max latency (us) 16302 1224 -92%
--- block/elevator.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
block/elevator.c +14 -0
@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) {unsigned int memflags;struct blk_mq_tag_set *set = q->tag_set; +bool unchanged;int ret =0 ;lockdep_assert_held (&set->update_nr_hwq_lock); +/* +* Skip freezing the queue if nothing would change. This can race with +* another elevator change, which at worst leads to a redundant switch. +*/ +mutex_lock (&q->elevator_lock); +if (q->elevator) + unchanged =elevator_match (q->elevator->type, ctx->name); +else + unchanged = !strncmp (ctx->name,"none" ,4 ); +mutex_unlock (&q->elevator_lock); +if (unchanged) +return 0 ; +if (strncmp (ctx->name,"none" ,4 )) { ret =blk_mq_alloc_sched_res (q, ctx->type, &ctx->res, set->nr_hw_queues);
Signature · 1 line
2.43.0
Re: [PATCH v3 2/2] block: return early when the elevator doesn't change
Quoted 25 lines · Qiliang Yuan
Writing the scheduler a queue already uses to queue/scheduler used to be a no-op.
elevator_change()now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue inelevator_switch()and resetsnr_requests.Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.
Check under
elevator_lockwhether anything would change before allocating scheduler resources and freezing the queue. The check can race with another elevator change, which at worst leads to a redundant switch, so keep the one done under the freeze as well.Writing the current scheduler to queue/scheduler of a
null_blkdisk 20 times takes (median):operation before (ms) after (ms) change none -> none 9.11 0.08 -99% mq-deadline -> mq-deadline 9.05 0.08 -99%4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:
Quoted 7 lines
metric before after change IOPS 340562 390392 +15% max latency (us) 16302 1224 -92%Fixes af2814149883e ("block: freeze the queue in queue_attr_store")Signed-off-by Qiliang Yuan<[email protected]>Reviewed-by Christoph Hellwig<[email protected]>
Looks good to me:
Re: [PATCH v3 1/2] block: don't request a module for the none elevator
In reply to Qiliang YuanQuoted 32 lines · Qiliang Yuan
elv_iosched_store()callselv_iosched_load_module()for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up inrequest_module("none-iosched").That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.
Return early for "none", like
elevator_change()did before the module loading moved intoelv_iosched_load_module().Writing queue/scheduler of a
null_blkdisk 20 times takes (median):operation before (ms) after (ms) change none -> none 13.10 9.11 -30% mq-deadline -> none 13.00 7.97 -39%Fixes 734e1a8603128 ("block: Prevent deadlocks when switching elevators")Signed-off-by Qiliang Yuan <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Nilay Shroff <[email protected]>--- block/elevator.c | 3 +++ 1 file changed, 3 insertions(+)block/elevator.c +0
-0
block/elevator.c static void elv_iosched_load_module(const char *elevator_name) · line 783
Quoted 9 lines
block/elevator.c +3
-0@@ -775,6 +775,9 @@ static void elv_iosched_load_module(const char *elevator_name) {struct elevator_type *found; +if (!strncmp (elevator_name,"none" ,4 )) +return ; +spin_lock (&elv_list_lock); found =__elevator_find (elevator_name);spin_unlock (&elv_list_lock);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH v3 2/2] block: return early when the elevator doesn't change
In reply to Qiliang YuanQuoted 56 lines · Qiliang Yuan
Writing the scheduler a queue already uses to queue/scheduler used to be a no-op.
elevator_change()now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue inelevator_switch()and resetsnr_requests.Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.
Check under
elevator_lockwhether anything would change before allocating scheduler resources and freezing the queue. The check can race with another elevator change, which at worst leads to a redundant switch, so keep the one done under the freeze as well.Writing the current scheduler to queue/scheduler of a
null_blkdisk 20 times takes (median):operation before (ms) after (ms) change none -> none 9.11 0.08 -99% mq-deadline -> mq-deadline 9.05 0.08 -99%4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:
metric before after change IOPS 340562 390392 +15% max latency (us) 16302 1224 -92%Fixes af2814149883e ("block: freeze the queue in queue_attr_store")Signed-off-by Qiliang Yuan <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>--- block/elevator.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)block/elevator.c +7
-0@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) {unsigned int memflags;struct blk_mq_tag_set *set = q->tag_set; +bool unchanged;int ret =0 ;lockdep_assert_held (&set->update_nr_hwq_lock); +/* +* Skip freezing the queue if nothing would change. This can race with +* another elevator change, which at worst leads to a redundant switch. +*/ +mutex_lock (&q->elevator_lock); +if (q->elevator)
block/elevator.c static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) · line 677
Quoted 10 lines
block/elevator.c +7
-0+ unchanged =elevator_match (q->elevator->type, ctx->name); + else + unchanged = !strncmp (ctx->name,"none" ,4 ); +mutex_unlock (&q->elevator_lock); +if (unchanged) +return 0 ; +if (strncmp (ctx->name,"none" ,4 )) { ret =blk_mq_alloc_sched_res (q, ctx->type, &ctx->res, set->nr_hw_queues);
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich