Skip to contentSkip to conversation

block: make changing to the current I/O scheduler cheap again

PATCH In review Qiliang Yuan 3 participants 6 messages Nov 17 Block layer

lore
Share & tools
lore
#0 lore raw

v3 cover block: make changing to the current I/O scheduler cheap again

Writing the scheduler a queue already uses to queue/scheduler takes about 10 ms and freezes the queue, which stalls I/O on live disks every time a udev rule or tuning tool sets the scheduler unconditionally.

Patch 1 stops requesting a "none-iosched" module on every write of "none". Patch 2 returns before freezing the queue when the elevator wouldn't change.

On a null_blk disk (median of 20 writes, all kernels measured in the same run):

operation base (ms) patch 1 (ms) patch 1+2 (ms)

  none -> none                    13.10          9.11            0.08
  mq-deadline -> mq-deadline      10.95          9.05            0.08
  mq-deadline -> none             13.00          7.97            6.03

---
V2 -> V3:
- Point Fixes in patch 2 at the commit that added the freeze, as Christoph pointed out
- Add Christoph's Reviewed-by to patch 2
- Rebase onto v7.4-rc2-1025-g666539eecdf4

V1 -> V2:
- Keep the elevator check under the freeze and note the race in the comment, as Christoph suggested
- Say "change" instead of "rewrite" in the cover letter subject
- Rebase onto v7.4-rc2-470-g7e1392a42ea2 and update the numbers from a new run

v2: https://lore.kernel.org/r/20261114-bug-block-elevator-unchanged-v2-0-5a1fe7d72fa0@linux.dev
v1: https://lore.kernel.org/r/20261111-bug-block-elevator-unchanged-v1-0-583198bd4e21@linux.dev

---
Qiliang Yuan (2):
block: don't request a module for the none elevator
block: return early when the elevator doesn't change

 block/elevator.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

---
base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4
change-id: 20261111-bug-block-elevator-unchanged-a6cd2d761d38

Best regards,

Signature · 1 line
Qiliang Yuan <[email protected]>
#1 lore raw

PATCH v3 1/2 block: don't request a module for the none elevator

elv_iosched_store() calls elv_iosched_load_module() for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up in request_module("none-iosched").

That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.

Return early for "none", like elevator_change() did before the module loading moved into elv_iosched_load_module().

Writing queue/scheduler of a null_blk disk 20 times takes (median):

operation before (ms) after (ms) change

  none -> none                13.10        9.11    -30%
  mq-deadline -> none         13.00        7.97    -39%
Fixes 734e1a8603128 ("block: Prevent deadlocks when switching elevators")
---
 block/elevator.c | 3 +++
 1 file changed, 3 insertions(+)
block/elevator.c +3 -0
index 66edf1a5768a8..9d9b2e84a1288 100644
@@ -775,6 +775,9 @@ static void elv_iosched_load_module(const char *elevator_name)
 {
 	struct elevator_type *found;
 
+	if (!strncmp(elevator_name, "none", 4))
+		return;
+
 	spin_lock(&elv_list_lock);
 	found = __elevator_find(elevator_name);
 	spin_unlock(&elv_list_lock);
Signature · 1 line
2.43.0
Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] block: don't request a module for the none elevator

Quoted 32 lines · Qiliang Yuan

elv_iosched_store() calls elv_iosched_load_module() for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up in request_module("none-iosched").

That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.

Return early for "none", like elevator_change() did before the module loading moved into elv_iosched_load_module().

Writing queue/scheduler of a null_blk disk 20 times takes (median):

   operation             before (ms)  after (ms)  change
   none -> none                13.10        9.11    -30%
   mq-deadline -> none         13.00        7.97    -39%
Fixes 734e1a8603128 ("block: Prevent deadlocks when switching elevators")
---
  block/elevator.c | 3 +++
  1 file changed, 3 insertions(+)
block/elevator.c +0 -0
index 66edf1a5768a8..9d9b2e84a1288 100644

block/elevator.c static void elv_iosched_load_module(const char *elevator_name) · line 783

Quoted 9 lines
block/elevator.c +3 -0
@@ -775,6 +775,9 @@ static void elv_iosched_load_module(const char *elevator_name)
  {
  	struct elevator_type *found;
 
+	if (!strncmp(elevator_name, "none", 4))
+		return;
+
  	spin_lock(&elv_list_lock);
  	found = __elevator_find(elevator_name);
  	spin_unlock(&elv_list_lock);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#3 lore raw

PATCH v3 2/2 block: return early when the elevator doesn't change

In reply to Qiliang Yuan

Writing the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.

Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.

Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. The check can race with another elevator change, which at worst leads to a redundant switch, so keep the one done under the freeze as well.

Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):

operation before (ms) after (ms) change

  none -> none                     9.11        0.08    -99%
  mq-deadline -> mq-deadline       9.05        0.08    -99%

4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:

metric before after change
IOPS 340562 390392 +15%
max latency (us) 16302 1224 -92%

Fixes af2814149883e ("block: freeze the queue in queue_attr_store")
---
 block/elevator.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
block/elevator.c +14 -0
index 9d9b2e84a1288..549cf51543a8b 100644
@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
 {
 	unsigned int memflags;
 	struct blk_mq_tag_set *set = q->tag_set;
+	bool unchanged;
 	int ret = 0;
 
 	lockdep_assert_held(&set->update_nr_hwq_lock);
 
+	/*
+	 * Skip freezing the queue if nothing would change. This can race with
+	 * another elevator change, which at worst leads to a redundant switch.
+	 */
+	mutex_lock(&q->elevator_lock);
+	if (q->elevator)
+		unchanged = elevator_match(q->elevator->type, ctx->name);
+	else
+		unchanged = !strncmp(ctx->name, "none", 4);
+	mutex_unlock(&q->elevator_lock);
+	if (unchanged)
+		return 0;
+
 	if (strncmp(ctx->name, "none", 4)) {
 		ret = blk_mq_alloc_sched_res(q, ctx->type, &ctx->res,
 				set->nr_hw_queues);
Signature · 1 line
2.43.0
Reviewed-by on v3 2/2
#4 lore raw

Re: [PATCH v3 2/2] block: return early when the elevator doesn't change

Quoted 25 lines · Qiliang Yuan

Writing the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.

Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.

Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. The check can race with another elevator change, which at worst leads to a redundant switch, so keep the one done under the freeze as well.

Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):

   operation                 before (ms)  after (ms)  change
   none -> none                     9.11        0.08    -99%
   mq-deadline -> mq-deadline       9.05        0.08    -99%

4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:

Quoted 7 lines
   metric             before  after  change
   IOPS               340562 390392    +15%
   max latency (us)    16302   1224    -92%
Fixes af2814149883e ("block: freeze the queue in queue_attr_store")

Looks good to me:

Reviewed-by on v3 2/2
#5 lore raw

Re: [PATCH v3 2/2] block: return early when the elevator doesn't change

In reply to Qiliang Yuan
Quoted 56 lines · Qiliang Yuan

Writing the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.

Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.

Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. The check can race with another elevator change, which at worst leads to a redundant switch, so keep the one done under the freeze as well.

Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):

   operation                 before (ms)  after (ms)  change
   none -> none                     9.11        0.08    -99%
   mq-deadline -> mq-deadline       9.05        0.08    -99%

4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:

   metric             before  after  change
   IOPS               340562 390392    +15%
   max latency (us)    16302   1224    -92%
Fixes af2814149883e ("block: freeze the queue in queue_attr_store")
---
  block/elevator.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
block/elevator.c +7 -0
index 9d9b2e84a1288..549cf51543a8b 100644
@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
  {
  	unsigned int memflags;
  	struct blk_mq_tag_set *set = q->tag_set;
+	bool unchanged;
  	int ret = 0;
 
  	lockdep_assert_held(&set->update_nr_hwq_lock);
 
+	/*
+	 * Skip freezing the queue if nothing would change. This can race with
+	 * another elevator change, which at worst leads to a redundant switch.
+	 */
+	mutex_lock(&q->elevator_lock);
+	if (q->elevator)

block/elevator.c static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) · line 677

Quoted 10 lines
block/elevator.c +7 -0
+		unchanged = elevator_match(q->elevator->type, ctx->name);
+	else
+		unchanged = !strncmp(ctx->name, "none", 4);
+	mutex_unlock(&q->elevator_lock);
+	if (unchanged)
+		return 0;
+
  	if (strncmp(ctx->name, "none", 4)) {
  		ret = blk_mq_alloc_sched_res(q, ctx->type, &ctx->res,
  				set->nr_hw_queues);

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich