Skip to contentSkip to conversation

block: make rewriting the current I/O scheduler cheap again

PATCH Superseded Qiliang Yuan 3 participants 7 messages Nov 12 – Nov 13 Block layer

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover block: make rewriting the current I/O scheduler cheap again

Writing the scheduler a queue already uses to queue/scheduler takes about 10 ms and freezes the queue, which stalls I/O on live disks every time a udev rule or tuning tool sets the scheduler unconditionally.

Patch 1 stops requesting a "none-iosched" module on every write of "none". Patch 2 returns before freezing the queue when the elevator wouldn't change.

On a null_blk disk (median of 20 writes, all kernels measured in the same run):

operation base (ms) patch 1 (ms) patch 1+2 (ms)

  none -> none                    13.09          9.06            0.08
  mq-deadline -> mq-deadline       9.19          9.14            0.08
  mq-deadline -> none             12.00          7.96            7.97

---
Qiliang Yuan (2):
block: don't request a module for the none elevator
block: return early when the elevator doesn't change

 block/elevator.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

---
base-commit: 79eb6679ef6a0d5ea68da53a8321fb1dc72299b9
change-id: 20261111-bug-block-elevator-unchanged-a6cd2d761d38

Best regards,

Signature · 1 line
Qiliang Yuan <[email protected]>
#1 lore raw

PATCH 1/2 block: don't request a module for the none elevator

elv_iosched_store() calls elv_iosched_load_module() for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up in request_module("none-iosched").

That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.

Return early for "none", like elevator_change() did before the module loading moved into elv_iosched_load_module().

Writing queue/scheduler of a null_blk disk 20 times takes (median):

operation before (ms) after (ms) change

  none -> none                13.09        9.06    -31%
  mq-deadline -> none         11.99        7.96    -34%
Fixes 734e1a8603128 ("block: Prevent deadlocks when switching elevators")
---
 block/elevator.c | 3 +++
 1 file changed, 3 insertions(+)
block/elevator.c +3 -0
index 66edf1a5768a8..9d9b2e84a1288 100644
@@ -775,6 +775,9 @@ static void elv_iosched_load_module(const char *elevator_name)
 {
 	struct elevator_type *found;
 
+	if (!strncmp(elevator_name, "none", 4))
+		return;
+
 	spin_lock(&elv_list_lock);
 	found = __elevator_find(elevator_name);
 	spin_unlock(&elv_list_lock);
Signature · 1 line
2.43.0
Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] block: don't request a module for the none elevator

In reply to Qiliang Yuan
Quoted 12 lines · Qiliang Yuan

elv_iosched_store() calls elv_iosched_load_module() for the requested name before anything else, and loads "<name>-iosched" if no elevator type of that name is registered. "none" never is, so every write of "none" to queue/scheduler ends up in request_module("none-iosched").

That runs a modprobe helper which can never find the module, and adds several milliseconds to every switch to "none", including rewrites of it by udev rules or tuning tools that set the scheduler on each disk.

Return early for "none", like elevator_change() did before the module loading moved into elv_iosched_load_module().

Quoted 8 lines

Writing queue/scheduler of a null_blk disk 20 times takes (median):

   operation             before (ms)  after (ms)  change
   none -> none                13.09        9.06    -31%
   mq-deadline -> none         11.99        7.96    -34%
Fixes 734e1a8603128 ("block: Prevent deadlocks when switching elevators")

Looks good to me:

#4 lore raw

PATCH 2/2 block: return early when the elevator doesn't change

In reply to Qiliang Yuan

Writing the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.

Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.

Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. Elevator changes of a queue are already serialized by QUEUE_FLAG_NO_ELV_SWITCH, registration and update_nr_hwq_lock, so drop the check under the freeze.

Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):

operation before (ms) after (ms) change

  none -> none                     9.06        0.08    -99%
  mq-deadline -> mq-deadline       9.14        0.08    -99%

4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:

metric before after change
IOPS 342609 395915 +16%
max latency (us) 10839 978 -91%

Fixes b07a889e83355 ("block: move q->sysfs_lock and queue-freeze under show/store method")
---
 block/elevator.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
block/elevator.c +15 -2
index 9d9b2e84a1288..979203f518ada 100644
@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
 {
 	unsigned int memflags;
 	struct blk_mq_tag_set *set = q->tag_set;
+	bool unchanged;
 	int ret = 0;
 
 	lockdep_assert_held(&set->update_nr_hwq_lock);
 
+	/*
+	 * Callers serialize elevator changes of this queue, so q->elevator is
+	 * stable here. Skip freezing the queue if nothing would change.
+	 */
+	mutex_lock(&q->elevator_lock);
+	if (q->elevator)
+		unchanged = elevator_match(q->elevator->type, ctx->name);
+	else
+		unchanged = !strncmp(ctx->name, "none", 4);
+	mutex_unlock(&q->elevator_lock);
+	if (unchanged)
+		return 0;
+
 	if (strncmp(ctx->name, "none", 4)) {
 		ret = blk_mq_alloc_sched_res(q, ctx->type, &ctx->res,
 				set->nr_hw_queues);
@@ -677,8 +691,7 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
 	 */
 	blk_mq_cancel_work_sync(q);
 	mutex_lock(&q->elevator_lock);
-	if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
-		ret = elevator_switch(q, ctx);
+	ret = elevator_switch(q, ctx);
 	mutex_unlock(&q->elevator_lock);
 	blk_mq_unfreeze_queue(q, memflags);
 	if (!ret)
Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH 2/2] block: return early when the elevator doesn't change

Quoted 50 lines · Qiliang Yuan

Writing the scheduler a queue already uses to queue/scheduler used to be a no-op. elevator_change() now only checks for an unchanged elevator after freezing the queue, and doesn't treat "none" on a queue without an elevator as unchanged at all, so that case also quiesces the queue in elevator_switch() and resets nr_requests.

Every such write therefore drains in-flight I/O and waits for an RCU grace period. udev rules and tuning tools that set the scheduler unconditionally stall I/O on live disks each time they run.

Check under elevator_lock whether anything would change before allocating scheduler resources and freezing the queue. Elevator changes of a queue are already serialized by QUEUE_FLAG_NO_ELV_SWITCH, registration and update_nr_hwq_lock, so drop the check under the freeze.

Writing the current scheduler to queue/scheduler of a null_blk disk 20 times takes (median):

operation before (ms) after (ms) change

  none -> none                     9.06        0.08    -99%
  mq-deadline -> mq-deadline       9.14        0.08    -99%

4k random reads at queue depth 1 for 10 seconds, while "none" is written to queue/scheduler every 50 ms:

metric before after change
IOPS 342609 395915 +16%
max latency (us) 10839 978 -91%

Fixes b07a889e83355 ("block: move q->sysfs_lock and queue-freeze under show/store method")
---
 block/elevator.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
block/elevator.c +3 -0
index 9d9b2e84a1288..979203f518ada 100644
@@ -654,10 +654,24 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
 {
 	unsigned int memflags;
 	struct blk_mq_tag_set *set = q->tag_set;
+	bool unchanged;
 	int ret = 0;
 
 	lockdep_assert_held(&set->update_nr_hwq_lock);
 
+	/*
+	 * Callers serialize elevator changes of this queue, so q->elevator is

block/elevator.c static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) · line 673

Quoted 10 lines
block/elevator.c +10 -0
+	 * stable here. Skip freezing the queue if nothing would change.
+	 */
+	mutex_lock(&q->elevator_lock);
+	if (q->elevator)
+		unchanged = elevator_match(q->elevator->type, ctx->name);
+	else
+		unchanged = !strncmp(ctx->name, "none", 4);
+	mutex_unlock(&q->elevator_lock);
+	if (unchanged)
+		return 0;

Strictly speaking this is racy with another thread changing the elevator. I don't think we actually care about that, but maybe add a comment to that effect?

block/elevator.c static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx) · line 694

Quoted 12 lines
block/elevator.c +1 -0
+
 	if (strncmp(ctx->name, "none", 4)) {
 
 		ret = blk_mq_alloc_sched_res(q, ctx->type, &ctx->res,
block/elevator.c +1 -2
 				set->nr_hw_queues);
@@ -677,8 +691,7 @@ static int elevator_change(struct request_queue *q, struct elv_change_ctx *ctx)
 	 */
 	blk_mq_cancel_work_sync(q);
 	mutex_lock(&q->elevator_lock);
-	if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
-		ret = elevator_switch(q, ctx);
+	ret = elevator_switch(q, ctx);

However, the raciness means we still need to keep this check I think.