block: blk-rq-qos: replace static key with atomic bitop
Subsystems (2)
v3 cover block: blk-rq-qos: replace static key with atomic bitop
Hi,
This patchset replaces the use of a static key in the I/O path (rq_qos_ xxx()) with an atomic queue flag (QUEUE_FLAG_QOS_ENABLED). This change is made to eliminate a potential deadlock introduced by the use of static keys in the blk-rq-qos infrastructure, as reported by lockdep during blktests block/005[1].
The original static key approach was introduced to avoid unnecessary dereferencing of q->rq_qos when no blk-rq-qos module (e.g., blk-wbt or blk-iolatency) is configured. While efficient, enabling a static key at runtime requires taking cpu_hotplug_lock and jump_label_mutex, which becomes problematic if the queue is already frozen — causing a reverse dependency on ->freeze_lock. This results in a lockdep splat indicating a potential deadlock.
To resolve this, we now gate q->rq_qos access with a q->queue_flags bitop (QUEUE_FLAG_QOS_ENABLED), avoiding the static key and the associated locking altogether.
I compared both static key and atomic bitop implementations using ftrace function graph tracer over ~50 invocations of rq_qos_issue() while ensuring blk-wbt/blk-iolatency were disabled (i.e., no QoS functionality). For easy comparision, I made rq_qos_issue() noinline. The comparision was made on PowerPC machine.
Static Key disabled (QoS is not configured):
5d0: 00 00 00 60 nop # patched in by static key framework 5d4: 20 00 80 4e blr # return (branch to link register)
Only a nop and blr (branch to link register) are executed — very lightweight.
atomic bitop (QoS is not configured):
5d0: 20 00 23 e9 ld r9,32(r3) # load q->queue_flags 5d4: 00 80 29 71 andi. r9,r9,32768 # check QUEUE_FLAG_QOS_ENABLED (bit 15) 5d8: 20 00 82 4d beqlr # return if bit not set
This performs an ld and andi. before returning. Slightly more work, but q->queue_flags is typically hot in cache during I/O submission.
With Static Key (disabled):
Duration (us): min=0.668 max=0.816 avg≈0.750
With atomic bitop QUEUE_FLAG_QOS_ENABLED (bit not set):
Duration (us): min=0.684 max=0.834 avg≈0.759
As expected, both versions are almost similar in cost. The added latency from an extra ld and andi. is in the range of ~9ns.
There're three patches in the series:
- First patch is a minor optimization which skips evaluating q->rq_qos check in the re_qos_done_bio() as it's not needed.
- Second patch fixes a subtle issue in rq_qos_del() to ensure that we decrement the block_rq_qos static key in rq_qos_del() when a QoS policy is detached from the queue.
- Third patch replaces usage of block_rq_qos static key with atomic flag QUEUE_FLAG_QOS_ENABLED and thus helps break cpu_hotplug_lock depedency on freeze_lock and that eventually help to fix the lockdep splat.
As usual, feedback and review comments are welcome!
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Changes from v2:
- Added a change to skip the q->rq_qos check in rq_qos_done_bio(). This is now part of the first patch in this series. (Yu Kuai)
- Added a separate patch to ensure block_rq_qos is decremented when detaching a QoS policy in rq_qos_del(). This is now the second patch in this series. (Yu Kuai)
- Folded the third patch from v2 into the new third patch, as patch ordering changed (the second patch from v2 is now the third patch in v3).
Link to v2: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- For debugging I made rq_qos_issue() noinline in my local workspace, but then inadvertently it slipped through the patchset upstream. So reverted it and made rq_qos_issue() inline again as earlier.
- Added Reported-by and Closes tags in the first patch, which I obviously missed to add in the first version.
Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (3):
block: skip q->rq_qos check in rq_qos_done_bio()
block: decrement block_rq_qos static key in rq_qos_del()
block: avoid cpu_hotplug_lock depedency on freeze_lock
block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 8 +++---- block/blk-rq-qos.h | 48 +++++++++++++++++++++++++++--------------- include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 21 deletions(-)
Signature · 1 line
2.50.1
PATCH v3 1/3 block: skip q->rq_qos check in rq_qos_done_bio()
If a bio has BIO_QOS_THROTTLED or BIO_QOS_MERGED set, it implicitly guarantees that q->rq_qos is present. Avoid re-checking q->rq_qos in this case and call __rq_qos_done_bio() directly as a minor optimization.
--- block/blk-rq-qos.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
block/blk-rq-qos.h +8 -2
@@ -142,8 +142,14 @@ static inline void rq_qos_done_bio(struct bio *bio) bio->bi_bdev && (bio_flagged(bio, BIO_QOS_THROTTLED) || bio_flagged(bio, BIO_QOS_MERGED))) {struct request_queue *q =bdev_get_queue (bio->bi_bdev); -if (q->rq_qos) -__rq_qos_done_bio (q->rq_qos, bio); + +/* +* If a bio has BIO_QOS_xxx set, it implicitly implies that +* q->rq_qos is present. So, we skip re-checking q->rq_qos +* here as an extra optimization and directly call +* __rq_qos_done_bio(). +*/ +__rq_qos_done_bio (q->rq_qos, bio); } }
Signature · 1 line
2.50.1
Re: [PATCHv3 1/3] block: skip q->rq_qos check in rq_qos_done_bio()
Quoted 10 lines · Nilay Shroff
If a bio has
BIO_QOS_THROTTLEDorBIO_QOS_MERGEDset, it implicitly guarantees thatq->rq_qosis present. Avoid re-checkingq->rq_qosin this case and call__rq_qos_done_bio()directly as a minor optimization.Suggested-by Yu Kuai<[email protected]>Signed-off-by Nilay Shroff<[email protected]>--- block/blk-rq-qos.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
LGTM
Re: [PATCHv3 1/3] block: skip q->rq_qos check in rq_qos_done_bio()
In reply to Nilay ShroffQuoted 7 lines · Nilay Shroff
If a bio has
BIO_QOS_THROTTLEDorBIO_QOS_MERGEDset, it implicitly guarantees thatq->rq_qosis present. Avoid re-checkingq->rq_qosin this case and call__rq_qos_done_bio()directly as a minor optimization.Suggested-by Yu Kuai <[email protected]>Signed-off-by Nilay Shroff <[email protected]>
Thanks,
Ming
PATCH v3 2/3 block: decrement block_rq_qos static key in rq_qos_del()
In reply to Nilay Shroffrq_qos_add() increments the block_rq_qos static key when a QoS policy is attached. When a QoS policy is removed via rq_qos_del(), we must symmetrically decrement the static key. If this removal drops the last QoS policy from the queue (q->rq_qos becomes NULL), the static branch can be disabled and the jump label patched to a NOP, avoiding overhead on the hot path.
This change ensures rq_qos_add()/rq_qos_del() keep the block_rq_qos static key balanced and prevents leaving the branch permanently enabled after the last policy is removed.
--- block/blk-rq-qos.c | 1 + 1 file changed, 1 insertion(+)
block/blk-rq-qos.c +1 -0
@@ -374,6 +374,7 @@ void rq_qos_del(struct rq_qos *rqos)for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {if (*cur == rqos) { *cur = rqos->next; +static_branch_dec (&block_rq_qos);break ; } }
Signature · 1 line
2.50.1
Re: [PATCHv3 2/3] block: decrement block_rq_qos static key in rq_qos_del()
Hi,
Quoted 11 lines · Nilay Shroff
rq_qos_add()increments theblock_rq_qosstatic key when a QoS policy is attached. When a QoS policy is removed viarq_qos_del(), we must symmetrically decrement the static key. If this removal drops the last QoS policy from the queue (q->rq_qosbecomes NULL), the static branch can be disabled and the jump label patched to a NOP, avoiding overhead on the hot path.This change ensures rq_qos_add()/rq_qos_del() keep the
block_rq_qosstatic key balanced and prevents leaving the branch permanently enabled after the last policy is removed.
BTW, this problem is unlikely to happen in real users cases, because rq_qos_del is only called from error path, where activate cgroup policy failed with -ENOMEM.
Quoted 12 lines
Fixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Signed-off-by Nilay Shroff <[email protected]>--- block/blk-rq-qos.c | 1 + 1 file changed, 1 insertion(+)block/blk-rq-qos.c +0
-0
block/blk-rq-qos.c void rq_qos_del(struct rq_qos *rqos) · line 380
Quoted 7 lines
block/blk-rq-qos.c +1
-0@@ -374,6 +374,7 @@ void rq_qos_del(struct rq_qos *rqos)for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {if (*cur == rqos) { *cur = rqos->next; +static_branch_dec (&block_rq_qos);break ; } }
Thanks
Re: [PATCHv3 2/3] block: decrement block_rq_qos static key in rq_qos_del()
In reply to Nilay ShroffQuoted 7 lines · Nilay Shroff
rq_qos_add()increments theblock_rq_qosstatic key when a QoS policy is attached. When a QoS policy is removed viarq_qos_del(), we must symmetrically decrement the static key. If this removal drops the last QoS policy from the queue (q->rq_qosbecomes NULL), the static branch can be disabled and the jump label patched to a NOP, avoiding overhead on the hot path.
Quoted 6 lines
This change ensures rq_qos_add()/rq_qos_del() keep the
block_rq_qosstatic key balanced and prevents leaving the branch permanently enabled after the last policy is removed.Fixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Signed-off-by Nilay Shroff <[email protected]>
Thanks,
Ming
PATCH v3 3/3 block: avoid cpu_hotplug_lock depedency on freeze_lock
In reply to Nilay ShroffA recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the cpu_hotplug_lock dependency on ->freeze_lock. This dependency was introduced by commit 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key").
That change added a static key to avoid fetching q->rq_qos when neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetching q->rq_qos in the I/O hot path. However, enabling a static key at runtime requires acquiring both cpu_hotplug_lock and jump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency from cpu_hotplug_lock to ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].
To resolve this, replace the static key mechanism with q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessing q->rq_qos. If the flag is set, we proceed to fetch q->rq_qos; otherwise, the access is skipped.
Since q->queue_flags is commonly accessed in IO hotpath and resides in the first cacheline of struct request_queue, checking it imposes minimal overhead while eliminating the deadlock risk.
This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)
block/blk-mq-debugfs.c +1 -0
@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAME
block/blk-rq-qos.c +4 -5
@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'.
@@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }
@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);blk_mq_unfreeze_queue (q, memflags);
@@ -374,10 +372,11 @@ void rq_qos_del(struct rq_qos *rqos)for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {if (*cur == rqos) { *cur = rqos->next; -static_branch_dec (&block_rq_qos);break ; } } +if (!q->rq_qos) +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);blk_mq_unfreeze_queue (q, memflags);mutex_lock (&q->debugfs_mutex);
block/blk-rq-qos.h +31 -23
108 lines not on this page · raw message
include/linux/blkdev.h +1 -0
@@ -656,6 +656,7 @@ enum {QUEUE_FLAG_SQ_SCHED ,/* single queue style io dispatch */ QUEUE_FLAG_DISABLE_WBT_DEF ,/* for sched to disable/enable wbt */ QUEUE_FLAG_NO_ELV_SWITCH ,/* can't switch elevator any more */ +QUEUE_FLAG_QOS_ENABLED ,/* qos is enabled */ QUEUE_FLAG_MAX };
Signature · 1 line
2.50.1
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 34 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1]https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki<[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki<[email protected]>Signed-off-by Nilay Shroff<[email protected]>
Quoted 6 lines
--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)
LGTM
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
In reply to Nilay ShroffQuoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }
block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);
One stupid question: can we simply move static_branch_inc(&block_rq_qos) out of queue freeze in rq_qos_add()?
What matters is just the 1st static_branch_inc() which switches the counter from 0 to 1, when blk_mq_freeze_queue() guarantees that all in-progress code paths observe q->rq_qos as NULL. That means static_branch_inc(&block_rq_qos) needn't queue freeze protection.
Thanks,
Ming
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?
Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.
I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto rq_qos_add(), For instance:
We have following code paths from where we invokerq_qos_add() APIs with queue already frozen:
ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_add()
queue_wb_lat_store() => freezes queue
-> wbt_init()
-> rq_qos_add()
And yes we do have code path leading to rq_qos_exit() which freezes queue first:
__del_gendisk() => freezes queue
-> rq_qos_exit()
And similarly for rq_qos_delete():ioc_qos_write()
-> blkg_conf_open_bdev_frozen() => freezes queue
-> blk_iocost_init()
-> rq_qos_del()
So even if we move static_branch_inc() out of freeze queue in rq_qos_add(), it'd still cause the observed lockdep splat.
Thanks,
--Nilay
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 102 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
Quoted 8 lines
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_add()
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()
The above two shouldn't be hard to solve, such as, add helper rq_qos_prep_add() for increasing the static branch counter.
Quoted 13 lines
And yes we do have code path leading to
rq_qos_exit()which freezes queue first:
__del_gendisk()=> freezes queue
->rq_qos_exit()And similarly for
rq_qos_delete():ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_del()
Quoted 3 lines
So even if we move
static_branch_inc()out of freeze queue inrq_qos_add(), it'd still cause the observed lockdep splat.
rqos won't be called into for passthrough request, so rq_qos_exit() may be moved to end of __del_gendisk(), when the freeze lock map is released.
Thanks,
Ming
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_add()
Quoted 7 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch q->rq_qos (which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?
Thanks,
--Nilay
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_add()Quoted 7 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.
Quoted 6 lines
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch
q->rq_qos(which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?
But it is that in-tree code is doing, isn't it?
static branch is only evaluated iff at least one rqos is added.
Thanks,
Ming
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Hi,
Quoted 112 lines · Ming Lei
Quoted 111 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue ->blk_iocost_init()-> rq_qos_add()
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.
I thought about this, we'll need some return value to know if rq_qos is really added and I feel code will be much complex. We'll need at least two different APIs for cgroup based policy iocost/iolatency and pure rq_qos policy wbt.
Quoted 6 lines
Quoted 6 lines
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch
q->rq_qos(which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?
I don't quite understand, do you mean between the window that static branch counter is increased and queue is not freezed? I think this is not hot path.
Quoted 5 lines
But it is that in-tree code is doing, isn't it?
static branchis only evaluated iff at least one rqos is added.
And yes.
Thanks,
Kuai
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 115 lines · Yu Kuai
Hi,
Quoted 112 lines · Ming Lei
Quoted 111 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 106 lines · Nilay Shroff
Quoted 83 lines · Ming Lei
Quoted 73 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue ->blk_iocost_init()-> rq_qos_add()
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.I thought about this, we'll need some return value to know if
rq_qosis really added and I feel code will be much complex. We'll need at least two different APIs for cgroup based policy iocost/iolatency and purerq_qospolicy wbt.
Yes, but not too bad, such as:
block/blk-iocost.c +4 -1
@@ -3227,6 +3227,8 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,blkg_conf_init (&ctx, input); +rq_qos_prep_add (); + memflags =blkg_conf_open_bdev_frozen (&ctx);if (IS_ERR_VALUE (memflags)) { ret = memflags;
@@ -3344,7 +3346,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,if (enable)wbt_disable_default (disk);else -wbt_enable_default (disk); +wbt_enable_default (disk, false);blk_mq_unquiesce_queue (disk->queue);
@@ -3356,6 +3358,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, ret = -EINVAL ; err:blkg_conf_exit_frozen (&ctx, memflags); +rq_qos_prep_del ();return ret; }
block/blk-rq-qos.c +1 -2
@@ -319,7 +319,7 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); +rq_qos_prep_del (); }mutex_unlock (&q->rq_qos_mutex); }
@@ -346,7 +346,6 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos);blk_mq_unfreeze_queue (q, memflags);
block/blk-rq-qos.h +10 -0
@@ -179,4 +179,14 @@ static inline void rq_qos_queue_depth_changed(struct request_queue *q)void rq_qos_exit (struct request_queue *); +static inline void rq_qos_prep_add (void ) +{ +static_branch_inc (&block_rq_qos); +} + +static inline void rq_qos_prep_del (void ) +{ +static_branch_dec (&block_rq_qos); +} +#endif
block/blk-sysfs.c +7 -1
@@ -613,6 +613,7 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page,ssize_t ret;s64 val;unsigned int memflags; +bool rqos_added = false; ret =queue_var_store64 (&val, page);if (ret <0 )
@@ -620,6 +621,7 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page,if (val <-1 )return -EINVAL ; +rq_qos_prep_add (); memflags =blk_mq_freeze_queue (q); rqos =wbt_rq_qos (q);
@@ -627,6 +629,7 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page, ret =wbt_init (disk);if (ret) goto out; + rqos_added = true; } ret = count;
@@ -653,6 +656,9 @@ static ssize_t queue_wb_lat_store(struct gendisk *disk, const char *page, out:blk_mq_unfreeze_queue (q, memflags); +if (!rqos_added) +rq_qos_prep_del (); +return ret; }
@@ -903,7 +909,7 @@ int blk_register_queue(struct gendisk *disk)if (queue_is_mq (q))elevator_set_default (q); -wbt_enable_default (disk); +wbt_enable_default (disk, true);blk_queue_flag_set (QUEUE_FLAG_REGISTERED , q);
block/blk-wbt.c +13 -3
@@ -698,7 +698,7 @@ static void wbt_requeue(struct rq_qos *rqos, struct request *rq)/* * Enable wbt if defaults are configured that way */ -void wbt_enable_default (struct gendisk *disk) +void wbt_enable_default (struct gendisk *disk,bool track_rqos_add) {struct request_queue *q = disk->queue;struct rq_qos *rqos;
@@ -723,8 +723,18 @@ void wbt_enable_default(struct gendisk *disk)if (!blk_queue_registered (q))return ; -if (queue_is_mq (q) && enable) -wbt_init (disk); +if (queue_is_mq (q) && enable) { +if (track_rqos_add) { +int ret; + +rq_qos_prep_add (); + ret =wbt_init (disk); +if (ret) +rq_qos_prep_del (); + }else { +wbt_init (disk); + } + } }EXPORT_SYMBOL_GPL (wbt_enable_default);
block/blk-wbt.h +1 -1
@@ -6,7 +6,7 @@int wbt_init (struct gendisk *disk);void wbt_disable_default (struct gendisk *disk); -void wbt_enable_default (struct gendisk *disk); +void wbt_enable_default (struct gendisk *disk,bool track_rqos_add);u64 wbt_get_min_lat (struct request_queue *q);void wbt_set_min_lat (struct request_queue *q,u64 val);
block/elevator.c +1 -1
@@ -647,7 +647,7 @@ static int elevator_change_done(struct request_queue *q,blk_mq_free_sched_tags (ctx->old->et, q->tag_set);kobject_put (&ctx->old->kobj);if (enable_wbt) -wbt_enable_default (q->disk); +wbt_enable_default (q->disk, true); }if (ctx->new) { ret =elv_register_queue (q, ctx->new, !ctx->no_uevent);
block/genhd.c +1 -2
@@ -762,8 +762,6 @@ static void __del_gendisk(struct gendisk *disk)if (queue_is_mq (q))blk_mq_cancel_work_sync (q); -rq_qos_exit (q); -/* * If the disk does not own the queue, allow using passthrough requests * again. Else leave the queue frozen to fail all I/O.
@@ -775,6 +773,7 @@ static void __del_gendisk(struct gendisk *disk)if (start_drain)blk_unfreeze_release_lock (q); +rq_qos_exit (q); }static void disable_elv_switch (struct request_queue *q)
Thanks,
Ming
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Hi,
Quoted 150 lines · Ming Lei
Quoted 115 lines · Yu Kuai
Hi,
Quoted 112 lines · Ming Lei
Quoted 111 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 106 lines · Nilay Shroff
Quoted 83 lines · Ming Lei
Quoted 73 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue ->blk_iocost_init()-> rq_qos_add()
queue_wb_lat_store()=> freezes queue
->wbt_init()-> rq_qos_add()Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.I thought about this, we'll need some return value to know if
rq_qosis really added and I feel code will be much complex. We'll need at least two different APIs for cgroup based policy iocost/iolatency and purerq_qospolicy wbt.Yes, but not too bad, such as:
block/blk-iocost.c +3
-1@@ -3227,6 +3227,8 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,blkg_conf_init (&ctx, input); +rq_qos_prep_add (); + memflags =blkg_conf_open_bdev_frozen (&ctx);if (IS_ERR_VALUE (memflags)) { ret = memflags;@@ -3344,7 +3346,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,if (enable)wbt_disable_default (disk);else -wbt_enable_default (disk); +wbt_enable_default (disk, false);blk_mq_unquiesce_queue (disk->queue);
block/blk-iocost.c static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, · line 3363
Quoted 6 lines
block/blk-iocost.c +1
-0@@ -3356,6 +3358,7 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, ret = -EINVAL ; err:blkg_conf_exit_frozen (&ctx, memflags); +rq_qos_prep_del ();return ret; }
This is not enough for iocost:
1) ioc_qos_write() can be called with iocost already registered, we need to call rq_qos_prep_del() in the succeed branch as well.
2) ioc_cost_model_write() have to have the same treatment.
Quoted 31 lines
block/blk-rq-qos.c +1
-2@@ -319,7 +319,7 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); +rq_qos_prep_del (); }mutex_unlock (&q->rq_qos_mutex); }@@ -346,7 +346,6 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos);blk_mq_unfreeze_queue (q, memflags);block/blk-rq-qos.h +1
-0@@ -179,4 +179,14 @@ static inline void rq_qos_queue_depth_changed(struct request_queue *q)void rq_qos_exit (struct request_queue *); +static inline void rq_qos_prep_add (void )
block/blk-rq-qos.h static inline void rq_qos_queue_depth_changed(struct request_queue *q) · line 192
Quoted 10 lines
block/blk-rq-qos.h +9
-0+{ +static_branch_inc (&block_rq_qos); +} + +static inline void rq_qos_prep_del (void ) +{ +static_branch_dec (&block_rq_qos); +} +#endif
Wonder can we simplify this, we already have a disk level lock rq_qos_mutex, that is held before freeze queue for iocost and iolatency, and we can grab it before freeze queue for wbt as well, then we can just do the static_branch_inc/dec with rq_qos_mutex held easily since everything is serialized.
+static inline bool rq_qos_prep_add(struct request_queue *q, enum rq_qos_id id) +{
+ lockdep_assert_held(q->rq_qos_mutex);
+
+ if (!rq_qos_id(q, id)) {
+ static_branch_inc(&block_rq_qos);
+ return true;
+ }
+
+ return false;
+}
+
+/* paired with rq_qos_prep_add */
+static inline bool rq_qos_prep_del(struct request_queue *q, enum rq_qos_id id, bool prepared) +{
+ lockdep_assert_held(q->rq_qos_mutex);
+
+ if (prepared && !rq_qos_id(q, id))
+ static_branch_dec(&block_rq_qos);
+}
+
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
In reply to Ming LeiQuoted 126 lines · Ming Lei
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_add()Quoted 7 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.Quoted 6 lines
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch
q->rq_qos(which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?
Quoted 4 lines
But it is that in-tree code is doing, isn't it?
static branchis only evaluated iff at least one rqos is added.
In the current in-tree implementation, the static branch is evaluated only if at least one rq_qos is added.
Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch q->rq_qos, even though q->rq_qos would still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additional q->rq_qos fetch.
In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qos becomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.
With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that q->rq_qos is non-NULL. Thus, we either:
- Skip evaluating the static branch key (and q->rq_qos) entirely, or
- If the static branch key is enabled, also have a valid q->rq_qos.
In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but q->rq_qos is still NULL, incurring unnecessary fetch overhead in the I/O hotpath.
Thanks,
--Nilay
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 149 lines · Nilay Shroff
Quoted 126 lines · Ming Lei
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_add()Quoted 7 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.Quoted 6 lines
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch
q->rq_qos(which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?Quoted 4 lines
But it is that in-tree code is doing, isn't it?
static branchis only evaluated iff at least one rqos is added.In the current in-tree implementation, the static branch is evaluated only if at least one
rq_qosis added.Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch
q->rq_qos, even thoughq->rq_qoswould still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additionalq->rq_qosfetch.In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qosbecomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.
Quoted 8 lines
With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that
q->rq_qosis non-NULL. Thus, we either:
- Skip evaluating the static branch key (andq->rq_qos) entirely, or
- If the static branch key is enabled, also have a validq->rq_qos.In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but
q->rq_qosis still NULL, incurring unnecessary fetch overhead in the I/O hotpath.
Yes, but the window is pretty small, so the extra overhead isn't something matters. More importantly it shows correct static_branch_*() usage, which is supposed to be called safely without subsystem lock.
Thanks,
Ming
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Quoted 155 lines · Ming Lei
Quoted 154 lines · Nilay Shroff
Quoted 126 lines · Ming Lei
Quoted 118 lines · Nilay Shroff
Quoted 108 lines · Ming Lei
Quoted 107 lines · Nilay Shroff
Quoted 87 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue
->blk_iocost_init()
->rq_qos_add()Quoted 7 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.Quoted 6 lines
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch
q->rq_qos(which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?Quoted 4 lines
But it is that in-tree code is doing, isn't it?
static branchis only evaluated iff at least one rqos is added.In the current in-tree implementation, the static branch is evaluated only if at least one
rq_qosis added.Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch
q->rq_qos, even thoughq->rq_qoswould still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additionalq->rq_qosfetch.In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qosbecomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that
q->rq_qosis non-NULL. Thus, we either:
- Skip evaluating the static branch key (andq->rq_qos) entirely, or
- If the static branch key is enabled, also have a validq->rq_qos.
Quoted 8 lines
Quoted 3 lines
In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but
q->rq_qosis still NULL, incurring unnecessary fetch overhead in the I/O hotpath.Yes, but the window is pretty small, so the extra overhead isn't something matters. More importantly it shows correct static_branch_*() usage, which is supposed to be called safely without subsystem lock.
I see your point about static_branch_*() usage being independent of the subsystem lock, but in this case that “small window” still sits directly in the I/O hotpath and will be hit by all in-flight requests during queue freeze. The current ordering is intentionally structured so that:
1. The branch is only enabled after q->rq_qos is guaranteed non-NULL.
2. Any hotpath evaluation of the static key implies a valid pointer dereference with no wasted cache miss.
Even if the window is short, we’d still pay for unnecessary q->rq_qos loads and possible cacheline misses for every inflight I/O during that period. In practice, that’s why this patch replaces the static key with a queue_flags bit: it’s lock- free to update, eliminates the deadlock risk, and preserves the “no penalty until active” property without depending on lock ordering subtlety.
Having said that, I’m not opposed to reworking the patch per your proposal if all agree the minor hotpath cost is acceptable, but wanted to make sure the trade-off is clear.
Thanks,
--Nilay
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
Hi,
Quoted 172 lines · Nilay Shroff
Quoted 149 lines · Ming Lei
Quoted 148 lines · Nilay Shroff
Quoted 121 lines · Ming Lei
Quoted 114 lines · Nilay Shroff
Quoted 106 lines · Ming Lei
Quoted 105 lines · Nilay Shroff
Quoted 86 lines · Ming Lei
Quoted 77 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>--- block/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 9 ++++--- block/blk-rq-qos.h | 54 ++++++++++++++++++++++++------------------ include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 28 deletions(-)block/blk-mq-debugfs.c +1
-0@@ -95,6 +95,7 @@ static const char *const blk_queue_flag_name[] = {QUEUE_FLAG_NAME (SQ_SCHED ),QUEUE_FLAG_NAME (DISABLE_WBT_DEF ),QUEUE_FLAG_NAME (NO_ELV_SWITCH ), + QUEUE_FLAG_NAME(QOS_ENABLED), };#undef QUEUE_FLAG_NAMEblock/blk-rq-qos.c +1
-3@@ -2,8 +2,6 @@#include "blk-rq-qos.h" -__read_mostly DEFINE_STATIC_KEY_FALSE (block_rq_qos); -/* * Increment 'v', if 'v' is below 'below'. Returns true if we succeeded, * false if 'v' + 1 would be bigger than 'below'. @@ -319,8 +317,8 @@ void rq_qos_exit(struct request_queue *q)struct rq_qos *rqos = q->rq_qos; q->rq_qos = rqos->next; rqos->ops->exit (rqos); -static_branch_dec (&block_rq_qos); } +blk_queue_flag_clear (QUEUE_FLAG_QOS_ENABLED , q);mutex_unlock (&q->rq_qos_mutex); }block/blk-rq-qos.c int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, · line 347
Quoted 5 lines
block/blk-rq-qos.c +1
-1@@ -346,7 +344,7 @@ int rq_qos_add(struct rq_qos *rqos, struct gendisk *disk, enum rq_qos_id id, goto ebusy; rqos->next = q->rq_qos; q->rq_qos = rqos; -static_branch_inc (&block_rq_qos); +blk_queue_flag_set (QUEUE_FLAG_QOS_ENABLED , q);One stupid question: can we simply move
static_branch_inc(&block_rq_qos)out of queue freeze inrq_qos_add()?Quoted 5 lines
What matters is just the 1st
static_branch_inc()which switches the counter from 0 to 1, whenblk_mq_freeze_queue()guarantees that all in-progress code paths observeq->rq_qosas NULL. That meansstatic_branch_inc(&block_rq_qos)needn't queue freeze protection.I thought about it earlier but that won't work because we have code paths freezing queue before it reaches upto
rq_qos_add(), For instance:We have following code paths from where we invoke
rq_qos_add()APIs with queue already frozen:
ioc_qos_write()
->blkg_conf_open_bdev_frozen()=> freezes queue ->blk_iocost_init()-> rq_qos_add()Quoted 6 lines
Quoted 3 lines
queue_wb_lat_store()=> freezes queue
->wbt_init()
->rq_qos_add()The above two shouldn't be hard to solve, such as, add helper
rq_qos_prep_add()for increasing the static branch counter.Quoted 6 lines
Yes but then it means that IOs which would be in flight would take a hit in hotpath: In hotpath those IOs would evaluate static key branch to true and then fetch
q->rq_qos(which probably would not be in the first cacheline). So are we okay to take hat hit in IO hotpath?Quoted 4 lines
But it is that in-tree code is doing, isn't it?
static branchis only evaluated iff at least one rqos is added.In the current in-tree implementation, the static branch is evaluated only if at least one
rq_qosis added.Per you suggested change, we would increment the static branch key before freezing the queue (and before attaching the QoS policy to the request queue). This means that any I/O already in flight would see the static branch key as enabled and would proceed to fetch
q->rq_qos, even thoughq->rq_qoswould still be NULL at that point since the QoS policy hasn’t yet been attached. This results in a performance penalty due to the additionalq->rq_qosfetch.In contrast, the current tree avoids this penalty. The existing sequence is:
- Freeze the queue.
- Attach the QoS policy to the queue (q->rq_qosbecomes non-NULL).
- Increment the static branch key.
- Unfreeze the queue.With this ordering, if the hotpath finds the static branch key enabled, it is guaranteed that
q->rq_qosis non-NULL. Thus, we either:
- Skip evaluating the static branch key (andq->rq_qos) entirely, or
- If the static branch key is enabled, also have a validq->rq_qos.Quoted 7 lines
Quoted 3 lines
In summary, it appears that your proposed ordering introduces a window where the static branch key is enabled but
q->rq_qosis still NULL, incurring unnecessary fetch overhead in the I/O hotpath.Yes, but the window is pretty small, so the extra overhead isn't something matters. More importantly it shows correct static_branch_*() usage, which is supposed to be called safely without subsystem lock.
I see your point about static_branch_*() usage being independent of the subsystem lock, but in this case that “small window” still sits directly in the I/O hotpath and will be hit by all in-flight requests during queue freeze. The current ordering is intentionally structured so that:
1. The branch is only enabled after
q->rq_qosis guaranteed non-NULL.
2. Any hotpath evaluation of the static key implies a valid pointer dereference with no wasted cache miss.
Even if the window is short, we’d still pay for unnecessaryq->rq_qosloads and possible cacheline misses for every inflight I/O during that period. In practice, that’s why this patch replaces the static key with aqueue_flagsbit: it’s lock- free to update, eliminates the deadlock risk, and preserves the “no penalty until active” property without depending on lock ordering subtlety.
Quoted 3 lines
Having said that, I’m not opposed to reworking the patch per your proposal if all agree the minor hotpath cost is acceptable, but wanted to make sure the trade-off is clear.
I'm good with this set, you already have my review :)
BTW, this set have another advantage that the flag is per disk, the static branch
can only optimize when all the disks in the system doesn't enable rq_qos, is one
disk enable it, all disks in the system that doesn't have rq_qos enabled will all suffer
from the additional cacheline load.
Thanks
Kuai
Quoted 5 lines
Thanks,
--Nilay
Re: [PATCHv3 3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
In reply to Nilay ShroffQuoted 27 lines · Nilay Shroff
A recent lockdep[1] splat observed while running blktest block/005 reveals a potential deadlock caused by the
cpu_hotplug_lockdependency on ->freeze_lock. This dependency was introduced by commit033b667a823e("block: blk-rq-qos: guard rq-qos helpers by static key").That change added a static key to avoid fetching
q->rq_qoswhen neither blk-wbt nor blk-iolatency is configured. The static key dynamically patches kernel text to a NOP when disabled, eliminating overhead of fetchingq->rq_qosin the I/O hot path. However, enabling a static key at runtime requires acquiring bothcpu_hotplug_lockandjump_label_mutex. When this happens after the queue has already been frozen (i.e., while holding ->freeze_lock), it creates a locking dependency fromcpu_hotplug_lockto ->freeze_lock, which leads to a potential deadlock reported by lockdep [1].To resolve this, replace the static key mechanism with
q->queue_flags:QUEUE_FLAG_QOS_ENABLED. This flag is evaluated in the fast path before accessingq->rq_qos. If the flag is set, we proceed to fetchq->rq_qos; otherwise, the access is skipped.Since
q->queue_flagsis commonly accessed in IO hotpath and resides in the first cacheline of structrequest_queue, checking it imposes minimal overhead while eliminating the deadlock risk.This change avoids the lockdep splat without introducing performance regressions.
Quoted 7 lines
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Reported-by Shinichiro Kawasaki <[email protected]>Closes https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/ open hereFixes 033b667a823e ("block: blk-rq-qos: guard rq-qos helpers by static key")Tested-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Nilay Shroff <[email protected]>
It is hard to use static branch correctly in current case from lock viewpoint, and most distributions should enable at least one rqos, so static branch won't optimize for typical cases:
Thanks,
Ming
Hi Jens,
Just a gentle ping on this patchset. There have been a couple of recent reports[1][2] highlighting this issue. Could you please consider pulling it?
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Thanks,
--Nilay
Quoted 95 lines · Nilay Shroff
Hi,
This patchset replaces the use of a static key in the I/O path (
rq_qos_xxx()) with an atomic queue flag (QUEUE_FLAG_QOS_ENABLED). This change is made to eliminate a potential deadlock introduced by the use of static keys in the blk-rq-qos infrastructure, as reported by lockdep during blktests block/005[1].The original static key approach was introduced to avoid unnecessary dereferencing of
q->rq_qoswhen no blk-rq-qos module (e.g., blk-wbt or blk-iolatency) is configured. While efficient, enabling a static key at runtime requires takingcpu_hotplug_lockandjump_label_mutex, which becomes problematic if the queue is already frozen — causing a reverse dependency on ->freeze_lock. This results in a lockdep splat indicating a potential deadlock.To resolve this, we now gate
q->rq_qosaccess with aq->queue_flagsbitop (QUEUE_FLAG_QOS_ENABLED), avoiding the static key and the associated locking altogether.I compared both static key and atomic bitop implementations using ftrace function graph tracer over ~50 invocations of
rq_qos_issue()while ensuring blk-wbt/blk-iolatency were disabled (i.e., no QoS functionality). For easy comparision, I maderq_qos_issue()noinline. The comparision was made on PowerPC machine.Static Key disabled (QoS is not configured):
5d0: 00 00 00 60 nop # patched in by static key framework 5d4: 20 00 80 4e blr # return (branch to link register)Only a nop and blr (branch to link register) are executed — very lightweight.
atomic bitop (QoS is not configured):
5d0: 20 00 23 e9 ld r9,32(r3) # load q->queue_flags 5d4: 00 80 29 71 andi. r9,r9,32768 # check QUEUE_FLAG_QOS_ENABLED (bit 15) 5d8: 20 00 82 4d beqlr # return if bit not setThis performs an ld and andi. before returning. Slightly more work, but
q->queue_flagsis typically hot in cache during I/O submission.With Static Key (disabled):
Duration (us): min=0.668 max=0.816 avg≈0.750With atomic bitop
QUEUE_FLAG_QOS_ENABLED(bit not set):
Duration (us): min=0.684 max=0.834 avg≈0.759As expected, both versions are almost similar in cost. The added latency from an extra ld and andi. is in the range of ~9ns.
There're three patches in the series:
- First patch is a minor optimization which skips evaluatingq->rq_qoscheck in there_qos_done_bio()as it's not needed.
- Second patch fixes a subtle issue inrq_qos_del()to ensure that we decrement theblock_rq_qosstatic key inrq_qos_del()when a QoS policy is detached from the queue.
- Third patch replaces usage ofblock_rq_qosstatic key with atomic flagQUEUE_FLAG_QOS_ENABLEDand thus helps breakcpu_hotplug_lockdepedency onfreeze_lockand that eventually help to fix the lockdep splat.As usual, feedback and review comments are welcome!
[1] https://lore.kernel.org/linux-block/4fdm37so3o4xricdgfosgmohn63aa7wj3ua4e5vpihoamwg3ui@fq42f5q5t5ic/
Changes from v2:
- Added a change to skip theq->rq_qoscheck inrq_qos_done_bio(). This is now part of the first patch in this series. (Yu Kuai)
- Added a separate patch to ensureblock_rq_qosis decremented when detaching a QoS policy inrq_qos_del(). This is now the second patch in this series. (Yu Kuai)
- Folded the third patch from v2 into the new third patch, as patch ordering changed (the second patch from v2 is now the third patch in v3).Link to v2: https://lore.kernel.org/all/[email protected]/
Changes from v1:
- For debugging I maderq_qos_issue()noinline in my local workspace, but then inadvertently it slipped through the patchset upstream. So reverted it and maderq_qos_issue()inline again as earlier.
- Added Reported-by and Closes tags in the first patch, which I obviously missed to add in the first version.Link to v1: https://lore.kernel.org/all/[email protected]/
Nilay Shroff (3):
block: skipq->rq_qoscheck inrq_qos_done_bio()
block: decrementblock_rq_qosstatic key inrq_qos_del()
block: avoidcpu_hotplug_lockdepedency onfreeze_lockblock/blk-mq-debugfs.c | 1 + block/blk-rq-qos.c | 8 +++---- block/blk-rq-qos.h | 48 +++++++++++++++++++++++++++--------------- include/linux/blkdev.h | 1 + 4 files changed, 37 insertions(+), 21 deletions(-)
Jens Axboe
maintainerQuoted 7 lines · Nilay Shroff
Hi Jens,
Just a gentle ping on this patchset. There have been a couple of recent reports[1][2] highlighting this issue. Could you please consider pulling it?
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Was waiting for discussion to settle down, but looks like there's agreement to go with this approach. I've queued it up, thanks.
Signature · 1 line
Jens Axboe
Quoted 14 lines · Nilay Shroff
This patchset replaces the use of a static key in the I/O path (
rq_qos_xxx()) with an atomic queue flag (QUEUE_FLAG_QOS_ENABLED). This change is made to eliminate a potential deadlock introduced by the use of static keys in the blk-rq-qos infrastructure, as reported by lockdep during blktests block/005[1].The original static key approach was introduced to avoid unnecessary dereferencing of
q->rq_qoswhen no blk-rq-qos module (e.g., blk-wbt or blk-iolatency) is configured. While efficient, enabling a static key at runtime requires takingcpu_hotplug_lockandjump_label_mutex, which becomes problematic if the queue is already frozen — causing a reverse dependency on ->freeze_lock. This results in a lockdep splat indicating a potential deadlock.
Quoted 1 line
[...]
Applied, thanks!
[1/3] block: skip q->rq_qos check in rq_qos_done_bio()
commit: 275332877e2fa9d6efa7402b1e897f6c6ee695bb
[2/3] block: decrement block_rq_qos static key in rq_qos_del()
commit: ade1beea1c27657712aa8f594226d461639382ff
[3/3] block: avoid cpu_hotplug_lock depedency on freeze_lock
commit: 370ac285f23aecae40600851fb4a1a9e75e50973Best regards,
Signature · 1 line
Jens Axboe