Skip to contentSkip to conversation

block: blk-rq-qos: replace static key with atomic bitop

PATCH Applied Nilay Shroff 5 participants 25 messages Aug 14, 2025 – Aug 21, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover block: blk-rq-qos: replace static key with atomic bitop

Hi,

This patchset replaces the use of a static key in the I/O path (rq_qos_ xxx()) with an atomic queue flag (QUEUE_FLAG_QOS_ENABLED). This change is made to eliminate a potential deadlock introduced by the use of static keys in the blk-rq-qos infrastructure, as reported by lockdep during blktests block/005[1].

The original static key approach was introduced to avoid unnecessary dereferencing of q->rq_qos when no blk-rq-qos module (e.g., blk-wbt or blk-iolatency) is configured. While efficient, enabling a static key at runtime requires taking cpu_hotplug_lock and jump_label_mutex, which becomes problematic if the queue is already frozen — causing a reverse dependency on ->freeze_lock. This results in a lockdep splat indicating a potential deadlock.

To resolve this, we now gate q->rq_qos access with a q->queue_flags bitop (QUEUE_FLAG_QOS_ENABLED), avoiding the static key and the associated locking altogether.

I compared both static key and atomic bitop implementations using ftrace function graph tracer over ~50 invocations of rq_qos_issue() while ensuring blk-wbt/blk-iolatency were disabled (i.e., no QoS functionality). For easy comparision, I made rq_qos_issue() noinline. The comparision was made on PowerPC machine.

Static Key disabled (QoS is not configured):

5d0: 00 00 00 60     nop    # patched in by static key framework
5d4: 20 00 80 4e     blr    # return (branch to link register)

Only a nop and blr (branch to link register) are executed — very lightweight.

atomic bitop (QoS is not configured):

5d0: 20 00 23 e9     ld      r9,32(r3)     # load q->queue_flags
5d4: 00 80 29 71     andi.   r9,r9,32768   # check QUEUE_FLAG_QOS_ENABLED (bit 15)
5d8: 20 00 82 4d     beqlr                 # return if bit not set

This performs an ld and andi. before returning. Slightly more work, but q->queue_flags is typically hot in cache during I/O submission.

With Static Key (disabled):
Duration (us): min=0.668 max=0.816 avg≈0.750

With atomic bitop QUEUE_FLAG_QOS_ENABLED (bit not set):
Duration (us): min=0.684 max=0.834 avg≈0.759

As expected, both versions are almost similar in cost. The added latency from an extra ld and andi. is in the range of ~9ns.

There're three patches in the series:
- First patch is a minor optimization which skips evaluating q->rq_qos check in the re_qos_done_bio() as it's not needed.
- Second patch fixes a subtle issue in rq_qos_del() to ensure that we decrement the block_rq_qos static key in rq_qos_del() when a QoS policy is detached from the queue.
- Third patch replaces usage of block_rq_qos static key with atomic flag QUEUE_FLAG_QOS_ENABLED and thus helps break cpu_hotplug_lock depedency on freeze_lock and that eventually help to fix the lockdep splat.

As usual, feedback and review comments are welcome!

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

Changes from v2:
- Added a change to skip the q->rq_qos check in rq_qos_done_bio(). This is now part of the first patch in this series. (Yu Kuai)
- Added a separate patch to ensure block_rq_qos is decremented when detaching a QoS policy in rq_qos_del(). This is now the second patch in this series. (Yu Kuai)
- Folded the third patch from v2 into the new third patch, as patch ordering changed (the second patch from v2 is now the third patch in v3).

Link to v2: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- For debugging I made rq_qos_issue() noinline in my local workspace, but then inadvertently it slipped through the patchset upstream. So reverted it and made rq_qos_issue() inline again as earlier.
- Added Reported-by and Closes tags in the first patch, which I obviously missed to add in the first version.

Link to v1: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (3):
block: skip q->rq_qos check in rq_qos_done_bio()
block: decrement block_rq_qos static key in rq_qos_del()
block: avoid cpu_hotplug_lock depedency on freeze_lock

 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  8 +++----
 block/blk-rq-qos.h     | 48 +++++++++++++++++++++++++++---------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 21 deletions(-)
Signature · 1 line
2.50.1
#1 lore raw

PATCH v3 1/3 block: skip q->rq_qos check in rq_qos_done_bio()

If a bio has BIO_QOS_THROTTLED or BIO_QOS_MERGED set, it implicitly guarantees that q->rq_qos is present. Avoid re-checking q->rq_qos in this case and call __rq_qos_done_bio() directly as a minor optimization.

---
 block/blk-rq-qos.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
block/blk-rq-qos.h +8 -2
index 39749f4066fb..28125fc49eff 100644
@@ -142,8 +142,14 @@ static inline void rq_qos_done_bio(struct bio *bio)
 	    bio->bi_bdev && (bio_flagged(bio, BIO_QOS_THROTTLED) ||
 			     bio_flagged(bio, BIO_QOS_MERGED))) {
 		struct request_queue *q = bdev_get_queue(bio->bi_bdev);
-		if (q->rq_qos)
-			__rq_qos_done_bio(q->rq_qos, bio);
+
+		/*
+		 * If a bio has BIO_QOS_xxx set, it implicitly implies that
+		 * q->rq_qos is present. So, we skip re-checking q->rq_qos
+		 * here as an extra optimization and directly call
+		 * __rq_qos_done_bio().
+		 */
+		__rq_qos_done_bio(q->rq_qos, bio);
 	}
 }
 
Signature · 1 line
2.50.1
Reviewed-by on v3 1/3
#2 lore raw

Re: [PATCHv3 1/3] block: skip q->rq_qos check in rq_qos_done_bio()

Quoted 10 lines · Nilay Shroff

If a bio has BIO_QOS_THROTTLED or BIO_QOS_MERGED set, it implicitly guarantees that q->rq_qos is present. Avoid re-checking q->rq_qos in this case and call __rq_qos_done_bio() directly as a minor optimization.

---
  block/blk-rq-qos.h | 10 ++++++++--
  1 file changed, 8 insertions(+), 2 deletions(-)

LGTM

Reviewed-by on v3 1/3
#3 lore raw

Re: [PATCHv3 1/3] block: skip q->rq_qos check in rq_qos_done_bio()

In reply to Nilay Shroff
Quoted 7 lines · Nilay Shroff

If a bio has BIO_QOS_THROTTLED or BIO_QOS_MERGED set, it implicitly guarantees that q->rq_qos is present. Avoid re-checking q->rq_qos in this case and call __rq_qos_done_bio() directly as a minor optimization.

Thanks,
Ming

#4 lore raw

PATCH v3 2/3 block: decrement block_rq_qos static key in rq_qos_del()

In reply to Nilay Shroff

rq_qos_add() increments the block_rq_qos static key when a QoS policy is attached. When a QoS policy is removed via rq_qos_del(), we must symmetrically decrement the static key. If this removal drops the last QoS policy from the queue (q->rq_qos becomes NULL), the static branch can be disabled and the jump label patched to a NOP, avoiding overhead on the hot path.

This change ensures rq_qos_add()/rq_qos_del() keep the block_rq_qos static key balanced and prevents leaving the branch permanently enabled after the last policy is removed.

Fixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")
---
 block/blk-rq-qos.c | 1 +
 1 file changed, 1 insertion(+)
block/blk-rq-qos.c +1 -0
index 848591fb3c57..b1e24bb85ad2 100644
@@ -374,6 +374,7 @@ void rq_qos_del(struct rq_qos *rqos)
 	for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {
 		if (*cur == rqos) {
 			*cur = rqos->next;
+			static_branch_dec(&block_rq_qos);
 			break;
 		}
 	}
Signature · 1 line
2.50.1
Reviewed-by on v3 2/3
#5 lore raw

Re: [PATCHv3 2/3] block: decrement block_rq_qos static key in rq_qos_del()

Hi,

Quoted 11 lines · Nilay Shroff

rq_qos_add() increments the block_rq_qos static key when a QoS policy is attached. When a QoS policy is removed via rq_qos_del(), we must symmetrically decrement the static key. If this removal drops the last QoS policy from the queue (q->rq_qos becomes NULL), the static branch can be disabled and the jump label patched to a NOP, avoiding overhead on the hot path.

This change ensures rq_qos_add()/rq_qos_del() keep the block_rq_qos static key balanced and prevents leaving the branch permanently enabled after the last policy is removed.

BTW, this problem is unlikely to happen in real users cases, because rq_qos_del is only called from error path, where activate cgroup policy failed with -ENOMEM.

Quoted 12 lines
Fixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")
---
  block/blk-rq-qos.c | 1 +
  1 file changed, 1 insertion(+)
block/blk-rq-qos.c +0 -0
index 848591fb3c57..b1e24bb85ad2 100644

block/blk-rq-qos.c void rq_qos_del(struct rq_qos *rqos) · line 380

Quoted 7 lines
block/blk-rq-qos.c +1 -0
@@ -374,6 +374,7 @@ void rq_qos_del(struct rq_qos *rqos)
  	for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {
  		if (*cur == rqos) {
  			*cur = rqos->next;
+			static_branch_dec(&block_rq_qos);
  			break;
  		}
  	}

Thanks

Reviewed-by on v3 2/3
#6 lore raw

Re: [PATCHv3 2/3] block: decrement block_rq_qos static key in rq_qos_del()

In reply to Nilay Shroff
Quoted 7 lines · Nilay Shroff

rq_qos_add() increments the block_rq_qos static key when a QoS policy is attached. When a QoS policy is removed via rq_qos_del(), we must symmetrically decrement the static key. If this removal drops the last QoS policy from the queue (q->rq_qos becomes NULL), the static branch can be disabled and the jump label patched to a NOP, avoiding overhead on the hot path.

Quoted 6 lines

This change ensures rq_qos_add()/rq_qos_del() keep the block_rq_qos static key balanced and prevents leaving the branch permanently enabled after the last policy is removed.

Fixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")

Thanks,
Ming

#7 lore raw

PATCH v3 3/3 block: avoid cpu_hotplug_lock depedency on freeze_lock

In reply to Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +4 -5
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);
 
 	blk_mq_unfreeze_queue(q, memflags);
 
@@ -374,10 +372,11 @@ void rq_qos_del(struct rq_qos *rqos)
 	for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {
 		if (*cur == rqos) {
 			*cur = rqos->next;
-			static_branch_dec(&block_rq_qos);
 			break;
 		}
 	}
+	if (!q->rq_qos)
+		blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	blk_mq_unfreeze_queue(q, memflags);
 
 	mutex_lock(&q->debugfs_mutex);
block/blk-rq-qos.h +31 -23

108 lines not on this page · raw message

include/linux/blkdev.h +1 -0
index 95886b404b16..fe1797bbec42 100644
@@ -656,6 +656,7 @@ enum {
 	QUEUE_FLAG_SQ_SCHED,		/* single queue style io dispatch */
 	QUEUE_FLAG_DISABLE_WBT_DEF,	/* for sched to disable/enable wbt */
 	QUEUE_FLAG_NO_ELV_SWITCH,	/* can't switch elevator any more */
+	QUEUE_FLAG_QOS_ENABLED,		/* qos is enabled */
 	QUEUE_FLAG_MAX
 };
 
Signature · 1 line
2.50.1
Reviewed-by on v3 3/3
#8 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 34 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1]https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

Quoted 6 lines
---
  block/blk-mq-debugfs.c |  1 +
  block/blk-rq-qos.c     |  9 ++++---
  block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
  include/linux/blkdev.h |  1 +
  4 files changed, 37 insertions(+), 28 deletions(-)

LGTM

#9 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

In reply to Nilay Shroff
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

Thanks,
Ming

#10 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

And yes we do have code path leading to rq_qos_exit() which freezes queue first:

__del_gendisk() => freezes queue
-> rq_qos_exit()

And similarly for rq_qos_delete():
ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_del()

So even if we move static_branch_inc() out of freeze queue in rq_qos_add(), it'd still cause the observed lockdep splat.

Thanks,
--Nilay

#11 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 102 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

Quoted 8 lines

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Quoted 13 lines

And yes we do have code path leading to rq_qos_exit() which freezes queue first:

__del_gendisk() => freezes queue
-> rq_qos_exit()

And similarly for rq_qos_delete():
ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_del()

Quoted 3 lines

So even if we move static_branch_inc() out of freeze queue in rq_qos_add(), it'd still cause the observed lockdep splat.

rqos won't be called into for passthrough request, so rq_qos_exit() may be moved to end of __del_gendisk(), when the freeze lock map is released.

Thanks,
Ming

#12 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

Quoted 7 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

Thanks,
--Nilay

#13 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

Quoted 7 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Quoted 6 lines

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

But it is that in-tree code is doing, isn't it?

static branch is only evaluated iff at least one rqos is added.

Thanks,
Ming

#14 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Hi,

Quoted 112 lines · Ming Lei
Quoted 111 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
  block/blk-mq-debugfs.c |  1 +
  block/blk-rq-qos.c     |  9 ++++---
  block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
  include/linux/blkdev.h |  1 +
  4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
  	QUEUE_FLAG_NAME(SQ_SCHED),
  	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
  	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
  };
  #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
  #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
  /*
   * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
   * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
  		struct rq_qos *rqos = q->rq_qos;
  		q->rq_qos = rqos->next;
  		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
  	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
  	mutex_unlock(&q->rq_qos_mutex);
  }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
  		goto ebusy;
  	rqos->next = q->rq_qos;
  	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue -> blk_iocost_init()

    -> rq_qos_add()
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

I thought about this, we'll need some return value to know if rq_qos is really added and I feel code will be much complex. We'll need at least two different APIs for cgroup based policy iocost/iolatency and pure rq_qos policy wbt.

Quoted 6 lines
Quoted 6 lines

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

I don't quite understand, do you mean between the window that static branch counter is increased and queue is not freezed? I think this is not hot path.

Quoted 5 lines

But it is that in-tree code is doing, isn't it?

static branch is only evaluated iff at least one rqos is added.

And yes.

Thanks,
Kuai

#15 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 115 lines · Yu Kuai

Hi,

Quoted 112 lines · Ming Lei
Quoted 111 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 106 lines · Nilay Shroff
Quoted 83 lines · Ming Lei
Quoted 73 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
  block/blk-mq-debugfs.c |  1 +
  block/blk-rq-qos.c     |  9 ++++---
  block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
  include/linux/blkdev.h |  1 +
  4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
  	QUEUE_FLAG_NAME(SQ_SCHED),
  	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
  	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
  };
  #undef QUEUE_FLAG_NAME
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
  #include "blk-rq-qos.h"
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
  /*
   * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
   * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
  		struct rq_qos *rqos = q->rq_qos;
  		q->rq_qos = rqos->next;
  		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
  	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
  	mutex_unlock(&q->rq_qos_mutex);
  }

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
  		goto ebusy;
  	rqos->next = q->rq_qos;
  	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue -> blk_iocost_init()

    -> rq_qos_add()

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

I thought about this, we'll need some return value to know if rq_qos is really added and I feel code will be much complex. We'll need at least two different APIs for cgroup based policy iocost/iolatency and pure rq_qos policy wbt.

Yes, but not too bad, such as:

block/blk-iocost.c +4 -1
index 5bfd70311359..05b13235ebb3 100644
@@ -3227,6 +3227,8 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	blkg_conf_init(&ctx, input);
 
+	rq_qos_prep_add();
+
 	memflags = blkg_conf_open_bdev_frozen(&ctx);
 	if (IS_ERR_VALUE(memflags)) {
 		ret = memflags;
@@ -3344,7 +3346,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 	if (enable)
 		wbt_disable_default(disk);
 	else
-		wbt_enable_default(disk);
+		wbt_enable_default(disk, false);
 
 	blk_mq_unquiesce_queue(disk->queue);
 
@@ -3356,6 +3358,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 	ret = -EINVAL;
 err:
 	blkg_conf_exit_frozen(&ctx, memflags);
+	rq_qos_prep_del();
 	return ret;
 }
 
block/blk-rq-qos.c +1 -2
index 848591fb3c57..27047f661e3f 100644
@@ -319,7 +319,7 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
+		rq_qos_prep_del();
 	}
 	mutex_unlock(&q->rq_qos_mutex);
 }
@@ -346,7 +346,6 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
 
 	blk_mq_unfreeze_queue(q, memflags);
 
block/blk-rq-qos.h +10 -0
index 39749f4066fb..38572a7eb2b7 100644
@@ -179,4 +179,14 @@ static inline void rq_qos_queue_depth_changed(struct request_queue *q)
 
 void rq_qos_exit(struct request_queue *);
 
+static inline void rq_qos_prep_add(void)
+{
+	static_branch_inc(&block_rq_qos);
+}
+
+static inline void rq_qos_prep_del(void)
+{
+	static_branch_dec(&block_rq_qos);
+}
+
 #endif
block/blk-sysfs.c +7 -1
index 396cded255ea..3801f35f206f 100644
@@ -613,6 +613,7 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page,
 	ssize_t ret;
 	s64 val;
 	unsigned int memflags;
+	bool rqos_added = false;
 
 	ret = queue_var_store64(&val, page);
 	if (ret < 0)
@@ -620,6 +621,7 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page,
 	if (val < -1)
 		return -EINVAL;
 
+	rq_qos_prep_add();
 	memflags = blk_mq_freeze_queue(q);
 
 	rqos = wbt_rq_qos(q);
@@ -627,6 +629,7 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page,
 		ret = wbt_init(disk);
 		if (ret)
 			goto out;
+		rqos_added = true;
 	}
 
 	ret = count;
@@ -653,6 +656,9 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page,
 out:
 	blk_mq_unfreeze_queue(q, memflags);
 
+	if (!rqos_added)
+		rq_qos_prep_del();
+
 	return ret;
 }
 
@@ -903,7 +909,7 @@ int blk_register_queue(struct gendisk *disk)
 
 	if (queue_is_mq(q))
 		elevator_set_default(q);
-	wbt_enable_default(disk);
+	wbt_enable_default(disk, true);
 
 	blk_queue_flag_set(QUEUE_FLAG_REGISTERED, q);
 
block/blk-wbt.c +13 -3
index a50d4cd55f41..9dc5926adfcc 100644
@@ -698,7 +698,7 @@ static void wbt_requeue(struct rq_qos *rqos, struct request *rq)
 /*
  * Enable wbt if defaults are configured that way
  */
-void wbt_enable_default(struct gendisk *disk)
+void wbt_enable_default(struct gendisk *disk, bool track_rqos_add)
 {
 	struct request_queue *q = disk->queue;
 	struct rq_qos *rqos;
@@ -723,8 +723,18 @@ void wbt_enable_default(struct gendisk *disk)
 	if (!blk_queue_registered(q))
 		return;
 
-	if (queue_is_mq(q) && enable)
-		wbt_init(disk);
+	if (queue_is_mq(q) && enable) {
+		if (track_rqos_add) {
+			int ret;
+
+			rq_qos_prep_add();
+			ret = wbt_init(disk);
+			if (ret)
+				rq_qos_prep_del();
+		} else {
+			wbt_init(disk);
+		}
+	}
 }
 EXPORT_SYMBOL_GPL(wbt_enable_default);
 
block/blk-wbt.h +1 -1
index e5fc653b9b76..fea0e54d7c1f 100644
@@ -6,7 +6,7 @@
 
 int wbt_init(struct gendisk *disk);
 void wbt_disable_default(struct gendisk *disk);
-void wbt_enable_default(struct gendisk *disk);
+void wbt_enable_default(struct gendisk *disk, bool track_rqos_add);
 
 u64 wbt_get_min_lat(struct request_queue *q);
 void wbt_set_min_lat(struct request_queue *q, u64 val);
block/elevator.c +1 -1
index fe96c6f4753c..cc2ca78ecb2b 100644
@@ -647,7 +647,7 @@ static int elevator_change_done(struct request_queue *q,
 		blk_mq_free_sched_tags(ctx->old->et, q->tag_set);
 		kobject_put(&ctx->old->kobj);
 		if (enable_wbt)
-			wbt_enable_default(q->disk);
+			wbt_enable_default(q->disk, true);
 	}
 	if (ctx->new) {
 		ret = elv_register_queue(q, ctx->new, !ctx->no_uevent);
block/genhd.c +1 -2
index c26733f6324b..2dd9d4ec66e4 100644
@@ -762,8 +762,6 @@ static void __del_gendisk(struct gendisk *disk)
 	if (queue_is_mq(q))
 		blk_mq_cancel_work_sync(q);
 
-	rq_qos_exit(q);
-
 	/*
 	 * If the disk does not own the queue, allow using passthrough requests
 	 * again.  Else leave the queue frozen to fail all I/O.
@@ -775,6 +773,7 @@ static void __del_gendisk(struct gendisk *disk)
 
 	if (start_drain)
 		blk_unfreeze_release_lock(q);
+	rq_qos_exit(q);
 }
 
 static void disable_elv_switch(struct request_queue *q)

Thanks,
Ming

#16 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Hi,

Quoted 150 lines · Ming Lei
Quoted 115 lines · Yu Kuai

Hi,

Quoted 112 lines · Ming Lei
Quoted 111 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 106 lines · Nilay Shroff
Quoted 83 lines · Ming Lei
Quoted 73 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
   block/blk-mq-debugfs.c |  1 +
   block/blk-rq-qos.c     |  9 ++++---
   block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
   include/linux/blkdev.h |  1 +
   4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
   	QUEUE_FLAG_NAME(SQ_SCHED),
   	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
   	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
   };
   #undef QUEUE_FLAG_NAME
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
   #include "blk-rq-qos.h"
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
   /*
    * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
    * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
   		struct rq_qos *rqos = q->rq_qos;
   		q->rq_qos = rqos->next;
   		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
   	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
   	mutex_unlock(&q->rq_qos_mutex);
   }

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
   		goto ebusy;
   	rqos->next = q->rq_qos;
   	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue -> blk_iocost_init()

     -> rq_qos_add()

queue_wb_lat_store() => freezes queue
-> wbt_init()

    -> rq_qos_add()
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

I thought about this, we'll need some return value to know if rq_qos is really added and I feel code will be much complex. We'll need at least two different APIs for cgroup based policy iocost/iolatency and pure rq_qos policy wbt.

Yes, but not too bad, such as:

block/blk-iocost.c +3 -1
index 5bfd70311359..05b13235ebb3 100644
@@ -3227,6 +3227,8 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
  	blkg_conf_init(&ctx, input);
 
+	rq_qos_prep_add();
+
  	memflags = blkg_conf_open_bdev_frozen(&ctx);
  	if (IS_ERR_VALUE(memflags)) {
  		ret = memflags;
@@ -3344,7 +3346,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
  	if (enable)
  		wbt_disable_default(disk);
  	else
-		wbt_enable_default(disk);
+		wbt_enable_default(disk, false);
 
  	blk_mq_unquiesce_queue(disk->queue);
 

block/blk-iocost.c static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, · line 3363

Quoted 6 lines
block/blk-iocost.c +1 -0
@@ -3356,6 +3358,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
  	ret = -EINVAL;
  err:
  	blkg_conf_exit_frozen(&ctx, memflags);
+	rq_qos_prep_del();
  	return ret;
  }

This is not enough for iocost:

1) ioc_qos_write() can be called with iocost already registered, we need to call rq_qos_prep_del() in the succeed branch as well.
2) ioc_cost_model_write() have to have the same treatment.

Quoted 31 lines
block/blk-rq-qos.c +1 -2
index 848591fb3c57..27047f661e3f 100644
@@ -319,7 +319,7 @@ void rq_qos_exit(struct request_queue *q)
  		struct rq_qos *rqos = q->rq_qos;
  		q->rq_qos = rqos->next;
  		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
+		rq_qos_prep_del();
  	}
  	mutex_unlock(&q->rq_qos_mutex);
  }
@@ -346,7 +346,6 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
  		goto ebusy;
  	rqos->next = q->rq_qos;
  	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
 
  	blk_mq_unfreeze_queue(q, memflags);
 
block/blk-rq-qos.h +1 -0
index 39749f4066fb..38572a7eb2b7 100644
@@ -179,4 +179,14 @@ static inline void rq_qos_queue_depth_changed(struct request_queue *q)
 
  void rq_qos_exit(struct request_queue *);
 
+static inline void rq_qos_prep_add(void)

block/blk-rq-qos.h static inline void rq_qos_queue_depth_changed(struct request_queue *q) · line 192

Quoted 10 lines
block/blk-rq-qos.h +9 -0
+{
+	static_branch_inc(&block_rq_qos);
+}
+
+static inline void rq_qos_prep_del(void)
+{
+	static_branch_dec(&block_rq_qos);
+}
+
  #endif

Wonder can we simplify this, we already have a disk level lock rq_qos_mutex, that is held before freeze queue for iocost and iolatency, and we can grab it before freeze queue for wbt as well, then we can just do the static_branch_inc/dec with rq_qos_mutex held easily since everything is serialized.

+static inline bool rq_qos_prep_add(struct request_queue *q, enum rq_qos_id id) +{
+ lockdep_assert_held(q->rq_qos_mutex);
+
+ if (!rq_qos_id(q, id)) {
+ static_branch_inc(&block_rq_qos);
+ return true;
+ }
+
+ return false;
+}
+
+/* paired with rq_qos_prep_add */
+static inline bool rq_qos_prep_del(struct request_queue *q, enum rq_qos_id id, bool prepared) +{
+ lockdep_assert_held(q->rq_qos_mutex);
+
+ if (prepared && !rq_qos_id(q, id))
+ static_branch_dec(&block_rq_qos);
+}
+

#17 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

In reply to Ming Lei
Quoted 126 lines · Ming Lei
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

Quoted 7 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Quoted 6 lines

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

Quoted 4 lines

But it is that in-tree code is doing, isn't it?

static branch is only evaluated iff at least one rqos is added.

In the current in-tree implementation, the static branch is evaluated only if at least one rq_qos is added.

Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch q->rq_qos, even though q->rq_qos would still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additional q->rq_qos fetch.

In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qos becomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.

With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that q->rq_qos is non-NULL. Thus, we either:
- Skip evaluating the static branch key (and q->rq_qos) entirely, or
- If the static branch key is enabled, also have a valid q->rq_qos.

In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but q->rq_qos is still NULL, incurring unnecessary fetch overhead in the I/O hotpath.

Thanks,
--Nilay

#18 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 149 lines · Nilay Shroff
Quoted 126 lines · Ming Lei
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

Quoted 7 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Quoted 6 lines

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

Quoted 4 lines

But it is that in-tree code is doing, isn't it?

static branch is only evaluated iff at least one rqos is added.

In the current in-tree implementation, the static branch is evaluated only if at least one rq_qos is added.

Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch q->rq_qos, even though q->rq_qos would still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additional q->rq_qos fetch.

In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qos becomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.

Quoted 8 lines

With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that q->rq_qos is non-NULL. Thus, we either:
- Skip evaluating the static branch key (and q->rq_qos) entirely, or
- If the static branch key is enabled, also have a valid q->rq_qos.

In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but q->rq_qos is still NULL, incurring unnecessary fetch overhead in the I/O hotpath.

Yes, but the window is pretty small, so the extra overhead isn't something matters. More importantly it shows correct static_branch_*() usage, which is supposed to be called safely without subsystem lock.

Thanks,
Ming

#19 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Quoted 155 lines · Ming Lei
Quoted 154 lines · Nilay Shroff
Quoted 126 lines · Ming Lei
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  9 ++++---
 block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
 	QUEUE_FLAG_NAME(SQ_SCHED),
 	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
 	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
 };
 #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
 #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
 /*
  * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
  * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
 		struct rq_qos *rqos = q->rq_qos;
 		q->rq_qos = rqos->next;
 		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
 	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
 	mutex_unlock(&q->rq_qos_mutex);
 }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
 		goto ebusy;
 	rqos->next = q->rq_qos;
 	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()

Quoted 7 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Quoted 6 lines

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

Quoted 4 lines

But it is that in-tree code is doing, isn't it?

static branch is only evaluated iff at least one rqos is added.

In the current in-tree implementation, the static branch is evaluated only if at least one rq_qos is added.

Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch q->rq_qos, even though q->rq_qos would still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additional q->rq_qos fetch.

In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qos becomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.

With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that q->rq_qos is non-NULL. Thus, we either:
- Skip evaluating the static branch key (and q->rq_qos) entirely, or
- If the static branch key is enabled, also have a valid q->rq_qos.

Quoted 8 lines
Quoted 3 lines

In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but q->rq_qos is still NULL, incurring unnecessary fetch overhead in the I/O hotpath.

Yes, but the window is pretty small, so the extra overhead isn't something matters. More importantly it shows correct static_branch_*() usage, which is supposed to be called safely without subsystem lock.

I see your point about static_branch_*() usage being independent of the subsystem lock, but in this case that “small window” still sits directly in the I/O hotpath and will be hit by all in-flight requests during queue freeze. The current ordering is intentionally structured so that:

1. The branch is only enabled after q->rq_qos is guaranteed non-NULL.
2. Any hotpath evaluation of the static key implies a valid pointer dereference with no wasted cache miss.
Even if the window is short, we’d still pay for unnecessary q->rq_qos loads and possible cacheline misses for every inflight I/O during that period. In practice, that’s why this patch replaces the static key with a queue_flags bit: it’s lock- free to update, eliminates the deadlock risk, and preserves the “no penalty until active” property without depending on lock ordering subtlety.

Having said that, I’m not opposed to reworking the patch per your proposal if all agree the minor hotpath cost is acceptable, but wanted to make sure the trade-off is clear.

Thanks,
--Nilay

#20 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

Hi,

Quoted 172 lines · Nilay Shroff
Quoted 149 lines · Ming Lei
Quoted 148 lines · Nilay Shroff
Quoted 121 lines · Ming Lei
Quoted 114 lines · Nilay Shroff
Quoted 106 lines · Ming Lei
Quoted 105 lines · Nilay Shroff
Quoted 86 lines · Ming Lei
Quoted 77 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

---
  block/blk-mq-debugfs.c |  1 +
  block/blk-rq-qos.c     |  9 ++++---
  block/blk-rq-qos.h     | 54 ++++++++++++++++++++++++------------------
  include/linux/blkdev.h |  1 +
  4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
index 7ed3e71f2fc0..32c65efdda46 100644
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {
  	QUEUE_FLAG_NAME(SQ_SCHED),
  	QUEUE_FLAG_NAME(DISABLE_WBT_DEF),
  	QUEUE_FLAG_NAME(NO_ELV_SWITCH),
+	QUEUE_FLAG_NAME(QOS_ENABLED),
  };
  #undef QUEUE_FLAG_NAME
 
block/blk-rq-qos.c +1 -3
index b1e24bb85ad2..654478dfbc20 100644
@@ -2,8 +2,6 @@
 
  #include "blk-rq-qos.h"
 
-__read_mostly DEFINE_STATIC_KEY_FALSE(block_rq_qos);
-
  /*
   * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded,
   * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)
  		struct rq_qos *rqos = q->rq_qos;
  		q->rq_qos = rqos->next;
  		rqos->ops->exit(rqos);
-		static_branch_dec(&block_rq_qos);
  	}
+	blk_queue_flag_clear(QUEUE_FLAG_QOS_ENABLED, q);
  	mutex_unlock(&q->rq_qos_mutex);
  }
 

block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347

Quoted 5 lines
block/blk-rq-qos.c +1 -1
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id,
  		goto ebusy;
  	rqos->next = q->rq_qos;
  	q->rq_qos = rqos;
-	static_branch_inc(&block_rq_qos);
+	blk_queue_flag_set(QUEUE_FLAG_QOS_ENABLED, q);

One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?

Quoted 5 lines

What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.

I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:

We have following code paths from where we invoke
rq_qos_add() APIs with queue already frozen:

ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue -> blk_iocost_init()

    -> rq_qos_add()
Quoted 6 lines
Quoted 3 lines

queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()

The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.

Quoted 6 lines

Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?

Quoted 4 lines

But it is that in-tree code is doing, isn't it?

static branch is only evaluated iff at least one rqos is added.

In the current in-tree implementation, the static branch is evaluated only if at least one rq_qos is added.

Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch q->rq_qos, even though q->rq_qos would still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additional q->rq_qos fetch.

In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qos becomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.

With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that q->rq_qos is non-NULL. Thus, we either:
- Skip evaluating the static branch key (and q->rq_qos) entirely, or
- If the static branch key is enabled, also have a valid q->rq_qos.

Quoted 7 lines
Quoted 3 lines

In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but q->rq_qos is still NULL, incurring unnecessary fetch overhead in the I/O hotpath.

Yes, but the window is pretty small, so the extra overhead isn't something matters. More importantly it shows correct static_branch_*() usage, which is supposed to be called safely without subsystem lock.

I see your point about static_branch_*() usage being independent of the subsystem lock, but in this case that “small window” still sits directly in the I/O hotpath and will be hit by all in-flight requests during queue freeze. The current ordering is intentionally structured so that:

1. The branch is only enabled after q->rq_qos is guaranteed non-NULL.
2. Any hotpath evaluation of the static key implies a valid pointer dereference with no wasted cache miss.
Even if the window is short, we’d still pay for unnecessary q->rq_qos loads and possible cacheline misses for every inflight I/O during that period. In practice, that’s why this patch replaces the static key with a queue_flags bit: it’s lock- free to update, eliminates the deadlock risk, and preserves the “no penalty until active” property without depending on lock ordering subtlety.

Quoted 3 lines

Having said that, I’m not opposed to reworking the patch per your proposal if all agree the minor hotpath cost is acceptable, but wanted to make sure the trade-off is clear.

I'm good with this set, you already have my review :)

BTW, this set have another advantage that the flag is per disk, the static branch

can only optimize when all the disks in the system doesn't enable rq_qos, is one

disk enable it, all disks in the system that doesn't have rq_qos enabled will all suffer

from the additional cacheline load.

Thanks

Kuai

Quoted 5 lines

Thanks,
--Nilay

Reviewed-by on v3 3/3
#21 lore raw

Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock

In reply to Nilay Shroff
Quoted 27 lines · Nilay Shroff

A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").

That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].

To resolve this, replace the static key mechanism with q->queue_flags:
QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.

Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.

This change avoids the lockdep splat without introducing performance regressions.

Quoted 7 lines

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

It is hard to use static branch correctly in current case from lock viewpoint, and most distributions should enable at least one rqos, so static branch won't optimize for typical cases:

Thanks,
Ming

#22 lore raw
In reply to Nilay Shroff

Hi Jens,

Just a gentle ping on this patchset. There have been a couple of recent reports[1][2] highlighting this issue. Could you please consider pulling it?

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Thanks,
--Nilay

Quoted 95 lines · Nilay Shroff

Hi,

This patchset replaces the use of a static key in the I/O path (rq_qos_ xxx()) with an atomic queue flag (QUEUE_FLAG_QOS_ENABLED). This change is made to eliminate a potential deadlock introduced by the use of static keys in the blk-rq-qos infrastructure, as reported by lockdep during blktests block/005[1].

The original static key approach was introduced to avoid unnecessary dereferencing of q->rq_qos when no blk-rq-qos module (e.g., blk-wbt or blk-iolatency) is configured. While efficient, enabling a static key at runtime requires taking cpu_hotplug_lock and jump_label_mutex, which becomes problematic if the queue is already frozen — causing a reverse dependency on ->freeze_lock. This results in a lockdep splat indicating a potential deadlock.

To resolve this, we now gate q->rq_qos access with a q->queue_flags bitop (QUEUE_FLAG_QOS_ENABLED), avoiding the static key and the associated locking altogether.

I compared both static key and atomic bitop implementations using ftrace function graph tracer over ~50 invocations of rq_qos_issue() while ensuring blk-wbt/blk-iolatency were disabled (i.e., no QoS functionality). For easy comparision, I made rq_qos_issue() noinline. The comparision was made on PowerPC machine.

Static Key disabled (QoS is not configured):

5d0: 00 00 00 60     nop    # patched in by static key framework
5d4: 20 00 80 4e     blr    # return (branch to link register)

Only a nop and blr (branch to link register) are executed — very lightweight.

atomic bitop (QoS is not configured):

5d0: 20 00 23 e9     ld      r9,32(r3)     # load q->queue_flags
5d4: 00 80 29 71     andi.   r9,r9,32768   # check QUEUE_FLAG_QOS_ENABLED (bit 15)
5d8: 20 00 82 4d     beqlr                 # return if bit not set

This performs an ld and andi. before returning. Slightly more work, but q->queue_flags is typically hot in cache during I/O submission.

With Static Key (disabled):
Duration (us): min=0.668 max=0.816 avg≈0.750

With atomic bitop QUEUE_FLAG_QOS_ENABLED (bit not set):
Duration (us): min=0.684 max=0.834 avg≈0.759

As expected, both versions are almost similar in cost. The added latency from an extra ld and andi. is in the range of ~9ns.

There're three patches in the series:
- First patch is a minor optimization which skips evaluating q->rq_qos check in the re_qos_done_bio() as it's not needed.
- Second patch fixes a subtle issue in rq_qos_del() to ensure that we decrement the block_rq_qos static key in rq_qos_del() when a QoS policy is detached from the queue.
- Third patch replaces usage of block_rq_qos static key with atomic flag QUEUE_FLAG_QOS_ENABLED and thus helps break cpu_hotplug_lock depedency on freeze_lock and that eventually help to fix the lockdep splat.

As usual, feedback and review comments are welcome!

[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/

Changes from v2:
- Added a change to skip the q->rq_qos check in rq_qos_done_bio(). This is now part of the first patch in this series. (Yu Kuai)
- Added a separate patch to ensure block_rq_qos is decremented when detaching a QoS policy in rq_qos_del(). This is now the second patch in this series. (Yu Kuai)
- Folded the third patch from v2 into the new third patch, as patch ordering changed (the second patch from v2 is now the third patch in v3).

Link to v2: https://lore.kernel.org/all/[email protected]/

Changes from v1:
- For debugging I made rq_qos_issue() noinline in my local workspace, but then inadvertently it slipped through the patchset upstream. So reverted it and made rq_qos_issue() inline again as earlier.
- Added Reported-by and Closes tags in the first patch, which I obviously missed to add in the first version.

Link to v1: https://lore.kernel.org/all/[email protected]/

Nilay Shroff (3):
block: skip q->rq_qos check in rq_qos_done_bio()
block: decrement block_rq_qos static key in rq_qos_del()
block: avoid cpu_hotplug_lock depedency on freeze_lock

 block/blk-mq-debugfs.c |  1 +
 block/blk-rq-qos.c     |  8 +++----
 block/blk-rq-qos.h     | 48 +++++++++++++++++++++++++++---------------
 include/linux/blkdev.h |  1 +
 4 files changed, 37 insertions(+), 21 deletions(-)
maintainer
#23 lore raw
Quoted 7 lines · Nilay Shroff

Hi Jens,

Just a gentle ping on this patchset. There have been a couple of recent reports[1][2] highlighting this issue. Could you please consider pulling it?

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Was waiting for discussion to settle down, but looks like there's agreement to go with this approach. I've queued it up, thanks.

Signature · 1 line
Jens Axboe
maintainer Applied
#24 lore raw
In reply to Nilay Shroff
Quoted 14 lines · Nilay Shroff

This patchset replaces the use of a static key in the I/O path (rq_qos_ xxx()) with an atomic queue flag (QUEUE_FLAG_QOS_ENABLED). This change is made to eliminate a potential deadlock introduced by the use of static keys in the blk-rq-qos infrastructure, as reported by lockdep during blktests block/005[1].

The original static key approach was introduced to avoid unnecessary dereferencing of q->rq_qos when no blk-rq-qos module (e.g., blk-wbt or blk-iolatency) is configured. While efficient, enabling a static key at runtime requires taking cpu_hotplug_lock and jump_label_mutex, which becomes problematic if the queue is already frozen — causing a reverse dependency on ->freeze_lock. This results in a lockdep splat indicating a potential deadlock.

Quoted 1 line

[...]

Applied, thanks!

[1/3] block: skip q->rq_qos check in rq_qos_done_bio()

      commit: 275332877e2fa9d6efa7402b1e897f6c6ee695bb
[2/3] block: decrement block_rq_qos static key in rq_qos_del()
      commit: ade1beea1c27657712aa8f594226d461639382ff
[3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
      commit: 370ac285f23aecae40600851fb4a1a9e75e50973

Best regards,

Signature · 1 line
Jens Axboe