blk-cgroup: fix blkg list and policy data races
Subsystems (4)
Yu Kuai
maintainercover blk-cgroup: fix blkg list and policy data races
This small series fixes races between blkg destruction, q->blkg_list iteration, and blkcg policy activation.
The first two patches serialize q->blkg_list walks in blkg_destroy_all() and BFQ writeback weight-raising teardown with blkcg_mutex. The next two patches close policy activation races with concurrent blkg destruction, including skipping blkgs that are already dying. The final patch factors the common policy data teardown loop.
This uses blkcg_mutex rather than extending queue_lock coverage because the races are about blkg list visibility and policy-data lifetime, not request-queue dispatch state. blkg_free_workfn() already uses blkcg_mutex to serialize policy-data freeing with policy deactivation and removes blkgs from q->blkg_list only after that teardown. Taking the same mutex around the remaining q->blkg_list walkers gives one sleepable serialization point for blkg lifetime, avoids adding more queue_lock nesting, and prepares the follow-up conversion that removes queue_lock from blkcg list protection entirely.
Yu Kuai (2):
blk-cgroup: protect q->blkg_list iteration in blkg_destroy_all() with
blkcg_mutex
bfq: protect q->blkg_list iteration in bfq_end_wr_async() with
blkcg_mutex
Zheng Qixing (3):
blk-cgroup: fix race between policy activation and blkg destruction
blk-cgroup: skip dying blkg in blkcg_activate_policy()
blk-cgroup: factor policy pd teardown loop into helper
block/bfq-cgroup.c | 3 ++- block/bfq-iosched.c | 6 +++++ block/blk-cgroup.c | 65 ++++++++++++++++++++++++--------------------- 3 files changed, 43 insertions(+), 31 deletions(-)
Signature · 1 line
2.51.0
Yu Kuai
maintainerPATCH 1/5 blk-cgroup: protect q->blkg_list iteration in blkg_destroy_all() with blkcg_mutex
blkg_destroy_all() iterates q->blkg_list without holding blkcg_mutex, which can race with blkg_free_workfn() that removes blkgs from the list while holding blkcg_mutex.
Add blkcg_mutex protection around the q->blkg_list iteration to prevent potential list corruption or use-after-free issues.
--- block/blk-cgroup.c | 3 +++ 1 file changed, 3 insertions(+)
block/blk-cgroup.c +3 -0
@@ -573,10 +573,11 @@ static void blkg_destroy_all(struct gendisk *disk)struct blkcg_gq *blkg;int count =BLKG_DESTROY_BATCH_SIZE ;int i; restart: +mutex_lock (&q->blkcg_mutex);spin_lock_irq (&q->queue_lock);list_for_each_entry (blkg, &q->blkg_list, q_node) {struct blkcg *blkcg = blkg->blkcg;if (hlist_unhashed (&blkg->blkcg_node))
@@ -591,10 +592,11 @@ static void blkg_destroy_all(struct gendisk *disk)* it when a batch of blkgs are destroyed. */ if (!(--count)) { count =BLKG_DESTROY_BATCH_SIZE ;spin_unlock_irq (&q->queue_lock); +mutex_unlock (&q->blkcg_mutex);cond_resched (); goto restart; } }
@@ -610,10 +612,11 @@ static void blkg_destroy_all(struct gendisk *disk)__clear_bit (pol->plid, q->blkcg_pols); } q->root_blkg =NULL ;spin_unlock_irq (&q->queue_lock); +mutex_unlock (&q->blkcg_mutex);wake_up_var (&q->root_blkg); }static void blkg_iostat_set (struct blkg_iostat *dst,struct blkg_iostat *src)
Signature · 1 line
2.51.0
Yu Kuai
maintainerPATCH 2/5 bfq: protect q->blkg_list iteration in bfq_end_wr_async() with blkcg_mutex
In reply to Yu Kuaibfq_end_wr_async() iterates q->blkg_list while only holding bfqd->lock, but not blkcg_mutex. This can race with blkg_free_workfn() that removes blkgs from the list while holding blkcg_mutex.
Add blkcg_mutex protection in bfq_end_wr() before taking bfqd->lock to ensure proper synchronization when iterating q->blkg_list.
--- block/bfq-cgroup.c | 3 ++- block/bfq-iosched.c | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
block/bfq-cgroup.c +2 -1
@@ -939,11 +939,12 @@ void bfq_end_wr_async(struct bfq_data *bfqd)struct blkcg_gq *blkg;list_for_each_entry (blkg, &bfqd->queue->blkg_list, q_node) {struct bfq_group *bfqg =blkg_to_bfqg (blkg); -bfq_end_wr_async_queues (bfqd, bfqg); +if (bfqg) +bfq_end_wr_async_queues (bfqd, bfqg); }bfq_end_wr_async_queues (bfqd, bfqd->root_group); }static int bfq_io_show_weight_legacy (struct seq_file *sf,void *v)
block/bfq-iosched.c +6 -0
@@ -2643,10 +2643,13 @@ void bfq_end_wr_async_queues(struct bfq_data *bfqd,static void bfq_end_wr (struct bfq_data *bfqd) {struct bfq_queue *bfqq;int i; +#ifdef CONFIG_BFQ_GROUP_IOSCHED +mutex_lock (&bfqd->queue->blkcg_mutex); +#endif spin_lock_irq (&bfqd->lock);for (i =0 ; i < bfqd->num_actuators; i++) {list_for_each_entry (bfqq, &bfqd->active_list[i], bfqq_list)bfq_bfqq_end_wr (bfqq);
@@ -2654,10 +2657,13 @@ static void bfq_end_wr(struct bfq_data *bfqd)list_for_each_entry (bfqq, &bfqd->idle_list, bfqq_list)bfq_bfqq_end_wr (bfqq);bfq_end_wr_async (bfqd);spin_unlock_irq (&bfqd->lock); +#ifdef CONFIG_BFQ_GROUP_IOSCHED +mutex_unlock (&bfqd->queue->blkcg_mutex); +#endif }static sector_t bfq_io_struct_pos (void *io_struct,bool request) {if (request)
Signature · 1 line
2.51.0
Re: [PATCH 2/5] bfq: protect q->blkg_list iteration in bfq_end_wr_async() with blkcg_mutex
Quoted 50 lines · Yu Kuai
bfq_end_wr_async()iteratesq->blkg_listwhile only holdingbfqd->lock, but notblkcg_mutex. This can race withblkg_free_workfn()that removes blkgs from the list while holdingblkcg_mutex.Add
blkcg_mutexprotection inbfq_end_wr()before takingbfqd->lockto ensure proper synchronization when iteratingq->blkg_list.Signed-off-by Yu Kuai <[email protected]>--- block/bfq-cgroup.c | 3 ++- block/bfq-iosched.c | 6 ++++++ 2 files changed, 8 insertions(+), 1 deletion(-)block/bfq-cgroup.c +2
-1@@ -939,11 +939,12 @@ void bfq_end_wr_async(struct bfq_data *bfqd)struct blkcg_gq *blkg;list_for_each_entry (blkg, &bfqd->queue->blkg_list, q_node) {struct bfq_group *bfqg =blkg_to_bfqg (blkg); -bfq_end_wr_async_queues (bfqd, bfqg); +if (bfqg) +bfq_end_wr_async_queues (bfqd, bfqg); }bfq_end_wr_async_queues (bfqd, bfqd->root_group); }static int bfq_io_show_weight_legacy (struct seq_file *sf,void *v)block/bfq-iosched.c +3
-0@@ -2643,10 +2643,13 @@ void bfq_end_wr_async_queues(struct bfq_data *bfqd,static void bfq_end_wr (struct bfq_data *bfqd) {struct bfq_queue *bfqq;int i; +#ifdef CONFIG_BFQ_GROUP_IOSCHED +mutex_lock (&bfqd->queue->blkcg_mutex); +#endif spin_lock_irq (&bfqd->lock);for (i =0 ; i < bfqd->num_actuators; i++) {list_for_each_entry (bfqq, &bfqd->active_list[i], bfqq_list)bfq_bfqq_end_wr (bfqq);
block/bfq-iosched.c static void bfq_end_wr(struct bfq_data *bfqd) · line 2665
Quoted 9 lines
block/bfq-iosched.c +3
-0@@ -2654,10 +2657,13 @@ static void bfq_end_wr(struct bfq_data *bfqd)list_for_each_entry (bfqq, &bfqd->idle_list, bfqq_list)bfq_bfqq_end_wr (bfqq);bfq_end_wr_async (bfqd);spin_unlock_irq (&bfqd->lock); +#ifdef CONFIG_BFQ_GROUP_IOSCHED +mutex_unlock (&bfqd->queue->blkcg_mutex); +#endif }
The above change protects the q->blkg_list iteration in bfq_end_wr_async() against list removal in blkg_free_workfn(). However the blkg insertion in blkg_create() still doesn't use q->blkcg_mutex and so list traversal in bfq_end_wr_async() may still race with blkg_create().
So I think we may also need to protect blkg insert in blkg_create() using q->blkcg_mutex.
Thanks,
--Nilay
yu kuai
maintainerRe: [PATCH 2/5] bfq: protect q->blkg_list iteration in bfq_end_wr_async() with blkcg_mutex
Hi,
Quoted 62 lines · Nilay Shroff
Quoted 46 lines · Yu Kuai
bfq_end_wr_async()iteratesq->blkg_listwhile only holdingbfqd->lock, but notblkcg_mutex. This can race withblkg_free_workfn()that removes blkgs from the list while holdingblkcg_mutex.Add
blkcg_mutexprotection inbfq_end_wr()before takingbfqd->lockto ensure proper synchronization when iteratingq->blkg_list.Signed-off-by Yu Kuai <[email protected]>---
block/bfq-cgroup.c | 3 ++-
block/bfq-iosched.c| 6 ++++++2 files changed, 8 insertions(+), 1 deletion(-)block/bfq-cgroup.c +2
-0@@ -939,11 +939,12 @@ void bfq_end_wr_async(struct bfq_data *bfqd) struct blkcg_gq *blkg; list_for_each_entry(blkg, &bfqd->queue->blkg_list, q_node) { struct bfq_group *bfqg = blkg_to_bfqg(blkg); - bfq_end_wr_async_queues(bfqd, bfqg); + if (bfqg) + bfq_end_wr_async_queues(bfqd, bfqg); } bfq_end_wr_async_queues(bfqd, bfqd->root_group); } static int bfq_io_show_weight_legacy(struct seq_file *sf, void *v)block/bfq-iosched.c +0
-0@@ -2643,10 +2643,13 @@ void bfq_end_wr_async_queues(struct bfq_data *bfqd, static void bfq_end_wr(struct bfq_data *bfqd) { struct bfq_queue *bfqq; int i;+#ifdef
CONFIG_BFQ_GROUP_IOSCHED
+mutex_lock(&bfqd->queue->blkcg_mutex);
+#endif
spin_lock_irq(&bfqd->lock);
for (i = 0; i <bfqd->num_actuators; i++) {list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list)bfq_bfqq_end_wr(bfqq);block/bfq-iosched.c static void bfq_end_wr(struct bfq_data *bfqd) · line 2664
Quoted 8 lines
block/bfq-iosched.c +3
-0@@ -2654,10 +2657,13 @@ static void bfq_end_wr(struct bfq_data *bfqd) list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list) bfq_bfqq_end_wr(bfqq); bfq_end_wr_async(bfqd); spin_unlock_irq(&bfqd->lock); +#ifdef CONFIG_BFQ_GROUP_IOSCHED + mutex_unlock(&bfqd->queue->blkcg_mutex); +#endif }The above change protects the
q->blkg_listiteration inbfq_end_wr_async()
against list removal inblkg_free_workfn(). However the blkg insertion inblkg_create()still doesn't useq->blkcg_mutexand so list traversal inbfq_end_wr_async()may still race withblkg_create().
Quoted 2 lines
So I think we may also need to protect blkg insert in
blkg_create()usingq->blkcg_mutex.
Yes, this is done in another huge patchset, because currently blkg_create() is protected by queue_lock and can be called under rcu, code refactor will be required.
I'll send the first set soon.
Quoted 3 lines
Thanks,
--Nilay
Signature · 2 lines
Thanks, Kuai
Yu Kuai
maintainerPATCH 3/5 blk-cgroup: fix race between policy activation and blkg destruction
In reply to Yu KuaiFrom: Zheng Qixing <[email protected]>
When switching an IO scheduler on a block device, blkcg_activate_policy() allocates blkg_policy_data (pd) for all blkgs attached to the queue. However, blkcg_activate_policy() may race with concurrent blkcg deletion, leading to use-after-free and memory leak issues.
The use-after-free occurs in the following race:
T1 (blkcg_activate_policy):
- Successfully allocates pd for blkg1 (loop0->queue, blkcgA)
- Fails to allocate pd for blkg2 (loop0->queue, blkcgB)
- Enters the enomem rollback path to release blkg1 resources
T2 (blkcg deletion):
- blkcgA is deleted concurrently
- blkg1 is freed via blkg_free_workfn()
- blkg1->pd is freed
T1 (continued):
- Rollback path accesses blkg1->pd->online after pd is freed
- Triggers use-after-free
In addition, blkg_free_workfn() frees pd before removing the blkg from q->blkg_list. This allows blkcg_activate_policy() to allocate a new pd for a blkg that is being destroyed, leaving the newly allocated pd unreachable when the blkg is finally freed.
Fix these races by extending blkcg_mutex coverage to serialize blkcg_activate_policy() rollback and blkg destruction, ensuring pd lifecycle is synchronized with blkg list visibility.
--- block/blk-cgroup.c | 3 +++ 1 file changed, 3 insertions(+)
block/blk-cgroup.c +3 -0
@@ -1612,10 +1612,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)if (WARN_ON_ONCE (!pol->pd_alloc_fn || !pol->pd_free_fn))return -EINVAL ;if (queue_is_mq (q)) memflags =blk_mq_freeze_queue (q); + +mutex_lock (&q->blkcg_mutex); retry:spin_lock_irq (&q->queue_lock);/* blkg_list is pushed at the head, reverse walk to initialize parents first */ list_for_each_entry_reverse (blkg, &q->blkg_list, q_node) {
@@ -1674,10 +1676,11 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)__set_bit (pol->plid, q->blkcg_pols); ret =0 ;spin_unlock_irq (&q->queue_lock); out: +mutex_unlock (&q->blkcg_mutex);if (queue_is_mq (q))blk_mq_unfreeze_queue (q, memflags);if (pinned_blkg)blkg_put (pinned_blkg);if (pd_prealloc)
Signature · 1 line
2.51.0
Yu Kuai
maintainerPATCH 4/5 blk-cgroup: skip dying blkg in blkcg_activate_policy()
In reply to Yu KuaiFrom: Zheng Qixing <[email protected]>
When switching IO schedulers on a block device, blkcg_activate_policy() can race with concurrent blkcg deletion, leading to a use-after-free in rcu_accelerate_cbs.
T1: T2:
blkg_destroy
kill(&blkg->refcnt) // blkg->refcnt=1->0
blkg_release // call_rcu(__blkg_release)
...
blkg_free_workfn
->pd_free_fn(pd)elv_iosched_storeelevator_switch
...
iterate blkg listblkg_get(blkg) // blkg->refcnt=0->1
list_del_init(&blkg->q_node)
blkg_put(pinned_blkg) // blkg->refcnt=1->0blkg_release // call_rcu againrcu_accelerate_cbs // uaf
Fix this by checking hlist_unhashed(&blkg->blkcg_node) before getting a reference to the blkg. This is the same check used in blkg_destroy() to detect if a blkg has already been destroyed. If the blkg is already unhashed, skip processing it since it's being destroyed.
--- block/blk-cgroup.c | 2 ++ 1 file changed, 2 insertions(+)
block/blk-cgroup.c +2 -0
@@ -1623,10 +1623,12 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)list_for_each_entry_reverse (blkg, &q->blkg_list, q_node) {struct blkg_policy_data *pd;if (blkg->pd[pol->plid])continue ; +if (hlist_unhashed (&blkg->blkcg_node)) +continue ;/* If prealloc matches, use it; otherwise try GFP_NOWAIT */ if (blkg == pinned_blkg) { pd = pd_prealloc; pd_prealloc =NULL ;
Signature · 1 line
2.51.0
Yu Kuai
maintainerPATCH 5/5 blk-cgroup: factor policy pd teardown loop into helper
In reply to Yu KuaiFrom: Zheng Qixing <[email protected]>
Move the teardown sequence which offlines and frees per-policy blkg_policy_data (pd) into a helper for readability.
No functional change intended.
--- block/blk-cgroup.c | 57 ++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 30 deletions(-)
block/blk-cgroup.c +27 -30
@@ -1575,10 +1575,35 @@ struct cgroup_subsys io_cgrp_subsys = { .depends_on = 1 << memory_cgrp_id,#endif };EXPORT_SYMBOL_GPL (io_cgrp_subsys); +/* +* Tear down per-blkg policy data for @pol on @q. +*/ +static void blkcg_policy_teardown_pds (struct request_queue *q, +const struct blkcg_policy *pol) +{ +struct blkcg_gq *blkg; + +list_for_each_entry (blkg, &q->blkg_list, q_node) { +struct blkcg *blkcg = blkg->blkcg; +struct blkg_policy_data *pd; + +spin_lock (&blkcg->lock); + pd = blkg->pd[pol->plid]; +if (pd) { +if (pd->online && pol->pd_offline_fn) + pol->pd_offline_fn (pd); + pd->online = false; + pol->pd_free_fn (pd); + blkg->pd[pol->plid] =NULL ; + } +spin_unlock (&blkcg->lock); + } +} +/** * blkcg_activate_policy - activate a blkcg policy on a gendisk * @disk: gendisk of interest * @pol: blkcg policy to activate *
@@ -1690,25 +1715,11 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)return ret; enomem:/* alloc failed, take down everything */ spin_lock_irq (&q->queue_lock); -list_for_each_entry (blkg, &q->blkg_list, q_node) { -struct blkcg *blkcg = blkg->blkcg; -struct blkg_policy_data *pd; - -spin_lock (&blkcg->lock); - pd = blkg->pd[pol->plid]; -if (pd) { -if (pd->online && pol->pd_offline_fn) - pol->pd_offline_fn (pd); - pd->online = false; - pol->pd_free_fn (pd); - blkg->pd[pol->plid] =NULL ; - } -spin_unlock (&blkcg->lock); - } +blkcg_policy_teardown_pds (q, pol);spin_unlock_irq (&q->queue_lock); ret = -ENOMEM ; goto out; }EXPORT_SYMBOL_GPL (blkcg_activate_policy);
@@ -1723,11 +1734,10 @@ EXPORT_SYMBOL_GPL(blkcg_activate_policy);*/ void blkcg_deactivate_policy (struct gendisk *disk,const struct blkcg_policy *pol) {struct request_queue *q = disk->queue; -struct blkcg_gq *blkg;unsigned int memflags;if (!blkcg_policy_enabled (q, pol))return ;
@@ -1736,24 +1746,11 @@ void blkcg_deactivate_policy(struct gendisk *disk,mutex_lock (&q->blkcg_mutex);spin_lock_irq (&q->queue_lock);__clear_bit (pol->plid, q->blkcg_pols); - -list_for_each_entry (blkg, &q->blkg_list, q_node) { -struct blkcg *blkcg = blkg->blkcg; - -spin_lock (&blkcg->lock); -if (blkg->pd[pol->plid]) { -if (blkg->pd[pol->plid]->online && pol->pd_offline_fn) - pol->pd_offline_fn (blkg->pd[pol->plid]); - pol->pd_free_fn (blkg->pd[pol->plid]); - blkg->pd[pol->plid] =NULL ; - } -spin_unlock (&blkcg->lock); - } - +blkcg_policy_teardown_pds (q, pol);spin_unlock_irq (&q->queue_lock);mutex_unlock (&q->blkcg_mutex);if (queue_is_mq (q))blk_mq_unfreeze_queue (q, memflags);
Signature · 1 line
2.51.0
Jens Axboe
maintainerQuoted 2 lines · Yu Kuai
This small series fixes races between blkg destruction,
q->blkg_listiteration, and blkcg policy activation.
In spam:
Authentication-Results: mx.google.com; spf=pass (google.com: domain of [email protected] designates 2600:3c04:e001:324:0:1991:8:25 as permitted sender) smtp.mailfrom="[email protected]"; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=QUARANTINE) header.from=fygo.io
if you don't get this sorted out, your messages will be missed as they end up in spam for most people.
Signature · 1 line
Jens Axboe
A message that is not in the archive (sent off-list or not yet received): [email protected]
Jens Axboe
maintainerQuoted 9 lines · yukuai
Hi,
? 2026/6/3 22:35, Jens Axboe ??:
Quoted 4 lines · Yu Kuai
Quoted 2 lines
This small series fixes races between blkg destruction,
q->blkg_listiteration, and blkcg policy activation.In spam: Authentication-Results: mx.google.com <mx.google.com>; spf=pass (google.com: <google.com:> domain of [email protected] designates 2600:3c04:e001:324:0:1991:8:25 as permitted sender) smtp.mailfrom="[email protected]"; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=QUARANTINE) header.from=fygo.io <header.from=fygo.io> if you don't get this sorted out, your messages will be missed as they end up in spam for most people.
Thanks again for the notice. I think I finally figured out why dmarc failed.
This email looks better!
Signature · 1 line
Jens Axboe
Quoted 10 lines · Yu Kuai
This small series fixes races between blkg destruction,
q->blkg_listiteration, and blkcg policy activation.The first two patches serialize
q->blkg_listwalks inblkg_destroy_all()and BFQ writeback weight-raising teardown withblkcg_mutex. The next two patches close policy activation races with concurrent blkg destruction, including skipping blkgs that are already dying. The final patch factors the common policy data teardown loop.
Quoted 9 lines
This uses
blkcg_mutexrather than extendingqueue_lockcoverage because the races are about blkg list visibility and policy-data lifetime, not request-queue dispatch state.blkg_free_workfn()already usesblkcg_mutexto serialize policy-data freeing with policy deactivation and removes blkgs fromq->blkg_listonly after that teardown. Taking the same mutex around the remainingq->blkg_listwalkers gives one sleepable serialization point for blkg lifetime, avoids adding morequeue_locknesting, and prepares the follow-up conversion that removesqueue_lockfrom blkcg list protection entirely.
LGTM.
I had noticed some time ago that blkcg_activate_policy() did not take q->blkcg_mutex, and this patchset nicely resolves my concern.
Best regards,
Yi
Quoted 20 lines
Yu Kuai (2):
blk-cgroup: protectq->blkg_listiteration inblkg_destroy_all()with
blkcg_mutex
bfq: protectq->blkg_listiteration inbfq_end_wr_async()with
blkcg_mutexZheng Qixing (3):
blk-cgroup: fix race between policy activation and blkg destruction
blk-cgroup: skip dying blkg inblkcg_activate_policy()
blk-cgroup: factor policy pd teardown loop into helperblock/bfq-cgroup.c | 3 ++- block/bfq-iosched.c | 6 +++++ block/blk-cgroup.c | 65 ++++++++++++++++++++++++--------------------- 3 files changed, 43 insertions(+), 31 deletions(-)--
2.51.0