Skip to contentSkip to conversation

blk-cgroup: Fix potential UAF & miscellaneous cleanup

PATCH Superseded Waiman Long 3 participants 12 messages Dec 11, 2022 – Dec 13, 2022

Subsystems (5)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover blk-cgroup: Fix potential UAF & miscellaneous cleanup

v2:
- Remove unnecessary rcu_read_{lock|unlock} from cgroup_rstat_css_cpu_flush() in patch 3.

It was found that blkcg_destroy_blkgs() may be called with all blkcg references gone. This may potentially cause user-after-free and so should be fixed. The last 2 patches are miscellaneous cleanups of commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()").

Waiman Long (3):
bdi, blk-cgroup: Fix potential UAF of blkcg
blk-cgroup: Don't flush a blkg if destroyed
blk-cgroup: Flush stats at blkgs destruction path

 block/blk-cgroup.c     | 26 ++++++++++++++++++++++++++
 include/linux/cgroup.h |  1 +
 kernel/cgroup/rstat.c  | 18 ++++++++++++++++++
 mm/backing-dev.c       |  8 ++++++--
 4 files changed, 51 insertions(+), 2 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

PATCH v2 1/3 bdi, blk-cgroup: Fix potential UAF of blkcg

Commit 59b57717fff8 ("blkcg: delay blkg destruction until after writeback has finished") delayed call to blkcg_destroy_blkgs() to cgwb_release_workfn(). However, it is done after a css_put() of blkcg which may be the final put that causes the blkcg to be freed as RCU read lock isn't held.

Another place where blkcg_destroy_blkgs() can be called indirectly via blkcg_unpin_online() is from the offline_css() function called from css_killed_work_fn(). Over there, the potentially final css_put() call is issued after offline_css().

By adding a css_tryget() into blkcg_destroy_blkgs() and warning its failure, the following stack trace was produced in a test system on bootup.

[   34.254240] RIP: 0010:blkcg_destroy_blkgs+0x16a/0x1a0
      :
[   34.339943] Call Trace:
[   34.344510]  blkcg_unpin_online+0x38/0x60
[   34.348523]  cgwb_release_workfn+0x6a/0x200
[   34.352708]  process_one_work+0x1e5/0x3b0
[   34.360758]  worker_thread+0x50/0x3a0
[   34.368447]  kthread+0xd9/0x100
[   34.376386]  ret_from_fork+0x22/0x30

This confirms that a potential UAF situation can really happen in cgwb_release_workfn().

Fix that by delaying the css_put() until after the blkcg_unpin_online() call. Also use css_tryget() in blkcg_destroy_blkgs() and issue a warning if css_tryget() fails.

The reproducing system can no longer produce a warning with this patch. All the runnable block/0* tests including block/027 were run successfully without failure.

Fixes 59b57717fff8 ("blkcg: delay blkg destruction until after writeback has finished")
---
 block/blk-cgroup.c | 8 ++++++++
 mm/backing-dev.c   | 8 ++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)
block/blk-cgroup.c +8 -0
index 1bb939d3b793..21cc88349f21 100644
@@ -1084,6 +1084,13 @@ struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css)
  */
 static void blkcg_destroy_blkgs(struct blkcg *blkcg)
 {
+	/*
+	 * blkcg_destroy_blkgs() shouldn't be called with all the blkcg
+	 * references gone.
+	 */
+	if (WARN_ON_ONCE(!css_tryget(&blkcg->css)))
+		return;
+
 	might_sleep();
 
 	spin_lock_irq(&blkcg->lock);
@@ -1110,6 +1117,7 @@ static void blkcg_destroy_blkgs(struct blkcg *blkcg)
 	}
 
 	spin_unlock_irq(&blkcg->lock);
+	css_put(&blkcg->css);
 }
 
 /**
mm/backing-dev.c +6 -2
index c30419a5e119..36f75b072325 100644
@@ -390,11 +390,15 @@ static void cgwb_release_workfn(struct work_struct *work)
 	wb_shutdown(wb);
 
 	css_put(wb->memcg_css);
-	css_put(wb->blkcg_css);
 	mutex_unlock(&wb->bdi->cgwb_release_mutex);
 
-	/* triggers blkg destruction if no online users left */
+	/*
+	 * Triggers blkg destruction if no online users left
+	 * The final blkcg css_put() has to be done after blkcg_unpin_online()
+	 * to avoid use-after-free.
+	 */
 	blkcg_unpin_online(wb->blkcg_css);
+	css_put(wb->blkcg_css);
 
 	fprop_local_destroy_percpu(&wb->memcg_completions);
 
Signature · 1 line
2.31.1
#4 lore raw

PATCH v2 2/3 blk-cgroup: Don't flush a blkg if destroyed

In reply to Waiman Long

Before commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()"), blkg's stats is only flushed if they are online. In addition, the stat flushing of blkgs in blkcg_rstat_flush() includes propagating the rstat data to its parent. However, if a blkg has been destroyed (offline), the validity of its parent may be questionable. For safety, revert back to the old behavior by ignoring offline blkg's.

---
 block/blk-cgroup.c | 6 ++++++
 1 file changed, 6 insertions(+)
block/blk-cgroup.c +6 -0
index 21cc88349f21..c466aef0d467 100644
@@ -885,6 +885,12 @@ static void blkcg_rstat_flush(struct cgroup_subsys_state *css, int cpu)
 
 		WRITE_ONCE(bisc->lqueued, false);
 
+		/* Don't flush its stats if blkg is offline */
+		if (unlikely(!blkg->online)) {
+			percpu_ref_put(&blkg->refcnt);
+			continue;
+		}
+
 		/* fetch the current per-cpu values */
 		do {
 			seq = u64_stats_fetch_begin(&bisc->sync);
Signature · 1 line
2.31.1
#9 lore raw

PATCH v2 3/3 blk-cgroup: Flush stats at blkgs destruction path

In reply to Waiman Long

As noted by Michal, the blkg_iostat_set's in the lockless list hold reference to blkg's to protect against their removal. Those blkg's hold reference to blkcg. When a cgroup is being destroyed, cgroup_rstat_flush() is only called at css_release_work_fn() which is called when the blkcg reference count reaches 0. This circular dependency will prevent blkcg from being freed until some other events cause cgroup_rstat_flush() to be called to flush out the pending blkcg stats.

To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() function to flush stats for a given css and cpu and call it at the blkgs destruction path, blkcg_destroy_blkgs(), whenever there are still some pending stats to be flushed. This will ensure that blkcg reference count can reach 0 ASAP.

---
 block/blk-cgroup.c     | 12 ++++++++++++
 include/linux/cgroup.h |  1 +
 kernel/cgroup/rstat.c  | 18 ++++++++++++++++++
 3 files changed, 31 insertions(+)
block/blk-cgroup.c +12 -0
index c466aef0d467..534f3baeb84a 100644
@@ -1090,6 +1090,8 @@ struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css)
  */
 static void blkcg_destroy_blkgs(struct blkcg *blkcg)
 {
+	int cpu;
+
 	/*
 	 * blkcg_destroy_blkgs() shouldn't be called with all the blkcg
 	 * references gone.
@@ -1099,6 +1101,16 @@ static void blkcg_destroy_blkgs(struct blkcg *blkcg)
 
 	might_sleep();
 
+	/*
+	 * Flush all the non-empty percpu lockless lists.
+	 */
+	for_each_possible_cpu(cpu) {
+		struct llist_head *lhead = per_cpu_ptr(blkcg->lhead, cpu);
+
+		if (!llist_empty(lhead))
+			cgroup_rstat_css_cpu_flush(&blkcg->css, cpu);
+	}
+
 	spin_lock_irq(&blkcg->lock);
 
 	while (!hlist_empty(&blkcg->blkg_list)) {
include/linux/cgroup.h +1 -0
index 528bd44b59e2..6c4e66b3fa84 100644
@@ -766,6 +766,7 @@ void cgroup_rstat_flush(struct cgroup *cgrp);
 void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp);
 void cgroup_rstat_flush_hold(struct cgroup *cgrp);
 void cgroup_rstat_flush_release(void);
+void cgroup_rstat_css_cpu_flush(struct cgroup_subsys_state *css, int cpu);
 
 /*
  * Basic resource stats.
kernel/cgroup/rstat.c +18 -0
index 793ecff29038..2e44be44351f 100644
@@ -281,6 +281,24 @@ void cgroup_rstat_flush_release(void)
 	spin_unlock_irq(&cgroup_rstat_lock);
 }
 
+/**
+ * cgroup_rstat_css_cpu_flush - flush stats for the given css and cpu
+ * @css: target css to be flush
+ * @cpu: the cpu that holds the stats to be flush
+ *
+ * A lightweight rstat flush operation for a given css and cpu.
+ * Only the cpu_lock is being held for mutual exclusion, the cgroup_rstat_lock
+ * isn't used.
+ */
+void cgroup_rstat_css_cpu_flush(struct cgroup_subsys_state *css, int cpu)
+{
+	raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu);
+
+	raw_spin_lock_irq(cpu_lock);
+	css->ss->css_rstat_flush(css, cpu);
+	raw_spin_unlock_irq(cpu_lock);
+}
+
 int cgroup_rstat_init(struct cgroup *cgrp)
 {
 	int cpu;
Signature · 1 line
2.31.1
maintainer
#5 lore raw

Re: [PATCH-block v2 2/3] blk-cgroup: Don't flush a blkg if destroyed

In reply to Waiman Long

Hello.

Quoted 2 lines · Waiman Long

Before commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()"), blkg's stats is only flushed if they are online.

I'm not sure I follow -- css_release_work_fn/cgroup_rstat_flush may be called on an offlined blkcg (offlined!=released). There's no invariant ensuring offlined blkcg won't be flushed. (There is only current situation when there is no reader of io data that'd need them flushed [1].)

Quoted 4 lines

In addition, the stat flushing of blkgs in blkcg_rstat_flush() includes propagating the rstat data to its parent. However, if a blkg has been destroyed (offline), the validity of its parent may be questionable.

Parents won't be freed (neither offlined) before children (see css_killed_work_fn). It should be regularly OK to pass data into a parent of an offlined blkcg.

Quoted 2 lines

For safety, revert back to the old behavior by ignoring offline blkg's.

I don't know if this is a good reasoning. If you argue that offlined children needn't be taken into parent's account, then I think it's more efficient to exclude the offlined blkcgs from update. (With the caveat I have in [1].)

Regards,
Michal

[1] https://lore.kernel.org/r/YqEfNgUc8jxlAq8D@blackbook/

#6 lore raw

Re: [PATCH-block v2 2/3] blk-cgroup: Don't flush a blkg if destroyed

Quoted 10 lines · Michal Koutný

Hello.

Quoted 2 lines · Waiman Long

Before commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()"), blkg's stats is only flushed if they are online.

I'm not sure I follow -- css_release_work_fn/cgroup_rstat_flush may be called on an offlined blkcg (offlined!=released). There's no invariant ensuring offlined blkcg won't be flushed. (There is only current situation when there is no reader of io data that'd need them flushed [1].)

The original cgroup_rstat_flush() iterates the list of blkg's in the blkg_list. A blkg will be removed from the list when it is offlined. This patch just reverts its behavior to that previous behavior.

Quoted 8 lines
Quoted 5 lines

In addition, the stat flushing of blkgs in blkcg_rstat_flush() includes propagating the rstat data to its parent. However, if a blkg has been destroyed (offline), the validity of its parent may be questionable.

Parents won't be freed (neither offlined) before children (see css_killed_work_fn). It should be regularly OK to pass data into a parent of an offlined blkcg.

I guess it is likely to be safe to flush an offline blkg. I am just being conservative in case there is any corner case where it may be a problem which I haven't foreseen.

Quoted 7 lines
Quoted 3 lines

For safety, revert back to the old behavior by ignoring offline blkg's.

I don't know if this is a good reasoning. If you argue that offlined children needn't be taken into parent's account, then I think it's more efficient to exclude the offlined blkcgs from update. (With the caveat I have in [1].)

It is possible that a blkg may be updated before it becomes offline, but the flush isn't done in time before that happens. The next patch will catch some of that.

Cheers,
Longman

maintainer
#2 lore raw

Re: [PATCH-block v2 1/3] bdi, blk-cgroup: Fix potential UAF of blkcg

In reply to Waiman Long

block/blk-cgroup.c struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css) · line 1092

Quoted 8 lines · Waiman Long

static void blkcg_destroy_blkgs(struct blkcg *blkcg) {

block/blk-cgroup.c +6 -0
+	/*
+	 * blkcg_destroy_blkgs() shouldn't be called with all the blkcg
+	 * references gone.
+	 */
+	if (WARN_ON_ONCE(!css_tryget(&blkcg->css)))
+		return;

Wouldn't it make more sense to use percpu_ref_is_zero()? It's not like the obtained extra reference does anything, right?

Thanks.

Signature · 1 line
tejun
maintainer
#7 lore raw

Re: [PATCH-block v2 2/3] blk-cgroup: Don't flush a blkg if destroyed

In reply to Michal Koutný
Quoted 21 lines · Michal Koutný

Hello.

Quoted 2 lines · Waiman Long

Before commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()"), blkg's stats is only flushed if they are online.

I'm not sure I follow -- css_release_work_fn/cgroup_rstat_flush may be called on an offlined blkcg (offlined!=released). There's no invariant ensuring offlined blkcg won't be flushed. (There is only current situation when there is no reader of io data that'd need them flushed [1].)

Quoted 4 lines

In addition, the stat flushing of blkgs in blkcg_rstat_flush() includes propagating the rstat data to its parent. However, if a blkg has been destroyed (offline), the validity of its parent may be questionable.

Parents won't be freed (neither offlined) before children (see css_killed_work_fn). It should be regularly OK to pass data into a parent of an offlined blkcg.

Quoted 7 lines
Quoted 2 lines

For safety, revert back to the old behavior by ignoring offline blkg's.

I don't know if this is a good reasoning. If you argue that offlined children needn't be taken into parent's account, then I think it's more efficient to exclude the offlined blkcgs from update. (With the caveat I have in [1].)

Yeah, I'm not sure about this patch either. Doesn't seem necessary.

Thanks.

Signature · 1 line
tejun
#3 lore raw

Re: [PATCH-block v2 1/3] bdi, blk-cgroup: Fix potential UAF of blkcg

In reply to Tejun Heo
Quoted 11 lines · Tejun Heo

block/blk-cgroup.c struct list_head *blkcg_get_cgwb_list(struct cgroup_subsys_state *css) · line 1092

Quoted 8 lines · Waiman Long
  static void blkcg_destroy_blkgs(struct blkcg *blkcg)
  {
block/blk-cgroup.c +6 -0
+	/*
+	 * blkcg_destroy_blkgs() shouldn't be called with all the blkcg
+	 * references gone.
+	 */
+	if (WARN_ON_ONCE(!css_tryget(&blkcg->css)))
+		return;

Wouldn't it make more sense to use percpu_ref_is_zero()? It's not like the obtained extra reference does anything, right?

Yes, that makes sense. Will incorporate the change in the next version.

Thanks,
Longman

maintainer Acked-by on v2 3/3
#10 lore raw

Re: [PATCH-block v2 3/3] blk-cgroup: Flush stats at blkgs destruction path

In reply to Waiman Long
Quoted 8 lines · Waiman Long

As noted by Michal, the blkg_iostat_set's in the lockless list hold reference to blkg's to protect against their removal. Those blkg's hold reference to blkcg. When a cgroup is being destroyed, cgroup_rstat_flush() is only called at css_release_work_fn() which is called when the blkcg reference count reaches 0. This circular dependency will prevent blkcg from being freed until some other events cause cgroup_rstat_flush() to be called to flush out the pending blkcg stats.

Quoted 7 lines

To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() function to flush stats for a given css and cpu and call it at the blkgs destruction path, blkcg_destroy_blkgs(), whenever there are still some pending stats to be flushed. This will ensure that blkcg reference count can reach 0 ASAP.

But a nit below

block/blk-cgroup.c static void blkcg_destroy_blkgs(struct blkcg *blkcg) · line 1106

Quoted 3 lines
block/blk-cgroup.c +3 -0
+	/*
+	 * Flush all the non-empty percpu lockless lists.
+	 */

Can you please explain the deadlock that's being avoided in the above comment? ie. it should say why this flush is necessary.

block/blk-cgroup.c static void blkcg_destroy_blkgs(struct blkcg *blkcg) · line 1112

Quoted 6 lines
block/blk-cgroup.c +6 -0
+	for_each_possible_cpu(cpu) {
+		struct llist_head *lhead = per_cpu_ptr(blkcg->lhead, cpu);
+
+		if (!llist_empty(lhead))
+			cgroup_rstat_css_cpu_flush(&blkcg->css, cpu);
+	}

Thanks.

Signature · 1 line
tejun
#11 lore raw

Re: [PATCH-block v2 3/3] blk-cgroup: Flush stats at blkgs destruction path

Quoted 18 lines · Tejun Heo
Quoted 8 lines · Waiman Long

As noted by Michal, the blkg_iostat_set's in the lockless list hold reference to blkg's to protect against their removal. Those blkg's hold reference to blkcg. When a cgroup is being destroyed, cgroup_rstat_flush() is only called at css_release_work_fn() which is called when the blkcg reference count reaches 0. This circular dependency will prevent blkcg from being freed until some other events cause cgroup_rstat_flush() to be called to flush out the pending blkcg stats.

Quoted 7 lines

To prevent this delayed blkcg removal, add a new cgroup_rstat_css_flush() function to flush stats for a given css and cpu and call it at the blkgs destruction path, blkcg_destroy_blkgs(), whenever there are still some pending stats to be flushed. This will ensure that blkcg reference count can reach 0 ASAP.

Quoted 7 lines

But a nit below

block/blk-cgroup.c static void blkcg_destroy_blkgs(struct blkcg *blkcg) · line 1106

Quoted 3 lines
block/blk-cgroup.c +3 -0
+	/*
+	 * Flush all the non-empty percpu lockless lists.
+	 */

Can you please explain the deadlock that's being avoided in the above comment? ie. it should say why this flush is necessary.

Sure. I will expand the comment to elaborate a bit more.

Cheers,
Longman

Quoted 7 lines
Quoted 7 lines
block/blk-cgroup.c +6 -0
+	for_each_possible_cpu(cpu) {
+		struct llist_head *lhead = per_cpu_ptr(blkcg->lhead, cpu);
+
+		if (!llist_empty(lhead))
+			cgroup_rstat_css_cpu_flush(&blkcg->css, cpu);
+	}
#8 lore raw

Re: [PATCH-block v2 2/3] blk-cgroup: Don't flush a blkg if destroyed

In reply to Tejun Heo
Quoted 20 lines · Tejun Heo
Quoted 19 lines · Michal Koutný

Hello.

Quoted 2 lines · Waiman Long

Before commit 3b8cc6298724 ("blk-cgroup: Optimize blkcg_rstat_flush()"), blkg's stats is only flushed if they are online.

I'm not sure I follow -- css_release_work_fn/cgroup_rstat_flush may be called on an offlined blkcg (offlined!=released). There's no invariant ensuring offlined blkcg won't be flushed. (There is only current situation when there is no reader of io data that'd need them flushed [1].)

Quoted 4 lines

In addition, the stat flushing of blkgs in blkcg_rstat_flush() includes propagating the rstat data to its parent. However, if a blkg has been destroyed (offline), the validity of its parent may be questionable.

Parents won't be freed (neither offlined) before children (see css_killed_work_fn). It should be regularly OK to pass data into a parent of an offlined blkcg.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

For safety, revert back to the old behavior by ignoring offline blkg's.

I don't know if this is a good reasoning. If you argue that offlined children needn't be taken into parent's account, then I think it's more efficient to exclude the offlined blkcgs from update. (With the caveat I have in [1].)

Yeah, I'm not sure about this patch either. Doesn't seem necessary.

I wrote this patch because I am not totally sure it is safe to flush offline blkgs. Since both you and Michal don't see any problem with it. I am fine taking it out.

Cheers,
Longman