Skip to contentSkip to conversation

bpf, cgroup: Don't wait for cgroup_mutex to read or free cgroup bpf state

PATCH Applied Shakeel Butt 4 participants 8 messages Nov 12 – Nov 14

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover bpf, cgroup: Don't wait for cgroup_mutex to read or free cgroup bpf state

In the Meta fleet, BPF_PROG_QUERY callers, link-info readers and the cgroup bpf release work often wait minutes for cgroup_mutex. The holder is almost always a cgroup mkdir or rmdir stuck on kernfs_rwsem. In one day, hung-task reports showed this on 1,075 hosts (query), 94 hosts (link info) and 458 hosts (release work).

Patch 3 gives the cgroup bpf state its own mutex, so these paths no longer take cgroup_mutex. Attach, detach and replace still take cgroup_mutex first, because they walk the descendants.

Patches 1 and 2 fix two old races. Attach and query get the cgroup from an fd and take the lock later, without checking whether the cgroup's bpf state was released in between. An effective query then reads freed memory (patch 1). Attach leaks the program, or with cgroup storage writes to a freed cgroup later (patch 2). Patch 3 makes the attach race easy to hit, so patch 2 must come before it. Both fixes have Fixes tags.

Testing, in an x86-64 VM with lockdep, PROVE_RCU and KASAN:

- The 41 cgroup bpf selftests pass. A 60-second stress of mkdir/rmdir, attach/detach, links, queries, fdinfo, storage map free and sockets in dying cgroups shows no lockdep or KASAN reports.
- Debug-only delays reproduce both races. Without the fixes, KASAN reports use-after-free in cgroup_bpf_prog_query() and, with patch 3, in cgroup_storage_map_free(), and BPF_PROG_ATTACH leaks the program. With the fixes, attach and query return -ENOENT.
- While a mkdir holds cgroup_mutex for 5 seconds, query and link info return in under a millisecond instead of 4.6 seconds. Attach and detach still wait.
- Builds with CGROUP_BPF=n and with BPF_LSM=n.

Shakeel Butt (3):
bpf, cgroup: Don't read freed effective arrays in BPF_PROG_QUERY bpf, cgroup: Don't attach to a removed cgroup
bpf, cgroup: Protect cgroup_bpf state with its own mutex

 include/linux/bpf-cgroup-defs.h |   1 +
 include/linux/bpf-cgroup.h      |   3 +
 kernel/bpf/cgroup.c             | 158 +++++++++++++++++++++++---------
 kernel/bpf/local_storage.c      |   8 +-
 4 files changed, 123 insertions(+), 47 deletions(-)

base-commit: 0fb18ce8a2e9de830433923cf65838933d615359

Signature · 1 line
2.53.0-Meta
#1 lore raw

PATCH 1/3 bpf, cgroup: Don't read freed effective arrays in BPF_PROG_QUERY

BPF_PROG_QUERY gets the cgroup from an fd before it takes cgroup_mutex. If the cgroup is removed in between and its bpf refcount drops to zero, cgroup_bpf_release() can run first. It frees the effective arrays but leaves the pointers, so a BPF_F_QUERY_EFFECTIVE query then reads freed memory.

The window is small because the release work also takes cgroup_mutex.

Check under the lock that the bpf refcount isn't zero, and return -ENOENT if it is, the error the fd lookup gives for a removed cgroup. The release work only runs after the count hits zero, and it takes the same lock.

Reproduced in a VM with a debug-only delay between the fd lookup and the
lock: KASAN reports a use-after-free in cgroup_bpf_prog_query(). With this patch, the query returns -ENOENT.

Fixes 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from cgroup itself")
---
 kernel/bpf/cgroup.c | 8 ++++++++
 1 file changed, 8 insertions(+)
kernel/bpf/cgroup.c +8 -0
index 57a785241d42..9258c2669fa8 100644
@@ -1446,6 +1446,14 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
 	if (effective_query && prog_attach_flags)
 		return -EINVAL;
 
+	/*
+	 * cgroup_bpf_release() frees the effective arrays once the bpf
+	 * refcount is zero.  It takes this lock too, so a nonzero count means
+	 * the arrays are still valid.
+	 */
+	if (percpu_ref_is_zero(&cgrp->bpf.refcnt))
+		return -ENOENT;
+
 	if (type == BPF_STRUCT_OPS) {
 		u32 type_id = attr->query.type_id;
 
Signature · 1 line
2.53.0-Meta
#2 lore raw

PATCH 2/3 bpf, cgroup: Don't attach to a removed cgroup

In reply to Shakeel Butt

Attach gets the cgroup from an fd before it takes cgroup_mutex. If the cgroup is removed in between and cgroup_bpf_release() runs first, attach adds a program that never runs:

- With BPF_PROG_ATTACH the program leaks, since nothing can detach it.
- With a link that uses cgroup storage, the storage stays on the freed cgroup's list, and freeing the map later writes to the freed cgroup.
- A cgroup struct_ops link is attached but never called.

The window is small today because the release work also waits for cgroup_mutex. The next patch stops that, which makes the race easy to hit.

Check cgroup_is_dead() under the lock and return -ENOENT, the error the fd lookup gives for a removed cgroup. rmdir marks the cgroup dead under cgroup_mutex before it starts the release, so the check is exact.

Reproduced in a VM with debug-only delays. With the next patch applied, a link attach queued behind a slow mkdir lands on the released cgroup, and freeing the map then hits a KASAN use-after-free in cgroup_storage_map_free(). BPF_PROG_ATTACH leaks the program. With this patch, attach returns -ENOENT.

Fixes 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf from cgroup itself")
Fixes 369d9dcd8fb8 ("bpf: Add infrastructure to support attaching struct_ops to cgroups")
---
 kernel/bpf/cgroup.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
kernel/bpf/cgroup.c +13 -0
index 9258c2669fa8..334237096759 100644
@@ -988,6 +988,13 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
 		/* replace_prog implies BPF_F_REPLACE, and vice versa */
 		return -EINVAL;
 
+	/*
+	 * The cgroup may have been removed since the caller looked it up.
+	 * Then its bpf state may be freed, and nothing would detach this.
+	 */
+	if (cgroup_is_dead(cgrp))
+		return -ENOENT;
+
 	atype = bpf_cgroup_atype_find(type, new_prog->aux->attach_btf_id);
 	if (atype < 0)
 		return -EINVAL;
@@ -3170,6 +3177,12 @@ int cgroup_bpf_struct_ops_attach(struct bpf_map *map, const union bpf_attr *attr
 
 	cgroup_lock();
 
+	/* See the cgroup_is_dead() check in __cgroup_bpf_attach() */
+	if (cgroup_is_dead(cgrp)) {
+		err = -ENOENT;
+		goto unlock;
+	}
+
 	if (attr->link_create.cgroup.expected_revision &&
 	    attr->link_create.cgroup.expected_revision != cgrp->bpf.revisions[atype]) {
 		err = -ESTALE;
Signature · 1 line
2.53.0-Meta
#3 lore raw

PATCH 3/3 bpf, cgroup: Protect cgroup_bpf state with its own mutex

In reply to Shakeel Butt

In the Meta fleet, tasks that only read or free cgroup bpf state often wait minutes for cgroup_mutex:

- In one day, hung-task reports (blocked over 120 seconds) showed this for BPF_PROG_QUERY on 1,075 hosts, the cgroup bpf release work on 458 hosts and link info on 94 hosts.
- The query callers are a container manager's health checks. Over 30 days, 6,700 hosts reported one hung. The typical wait is 11 minutes, the longest 75 minutes.
- The cgroup_mutex holder is almost always a cgroup mkdir or rmdir that is waiting for kernfs_rwsem.

Only attach, detach and replace need cgroup_mutex. They update the effective arrays of all descendants, so the hierarchy must not change.

Add cgroup_bpf_mutex for the bpf state. Attach, detach and replace take cgroup_mutex and then cgroup_bpf_mutex. Everything else (query, link info, the release work, dropping an LSM attach type, freeing a cgroup storage map) takes only cgroup_bpf_mutex.

cgroup_bpf_inherit() still runs at mkdir with only cgroup_mutex, so mkdir never waits for cgroup_bpf_mutex. It writes the new cgroup, which no one else can see yet. What it reads from the ancestors changes only under cgroup_mutex, and the release work can't run for them because they are online.

The release work no longer waits for cgroup_mutex, so attach must check that the cgroup is still alive. The previous patch does that.

With a test delay that makes a mkdir hold cgroup_mutex for 5 seconds, BPF_PROG_QUERY and link info now return in under a millisecond instead of 4.6 seconds. Attach and detach still wait.

---
 include/linux/bpf-cgroup-defs.h |   1 +
 include/linux/bpf-cgroup.h      |   3 +
 kernel/bpf/cgroup.c             | 137 +++++++++++++++++++++-----------
 kernel/bpf/local_storage.c      |   8 +-
 4 files changed, 102 insertions(+), 47 deletions(-)
include/linux/bpf-cgroup-defs.h +1 -0
index 0147b8bec973..671f27ba6a62 100644
@@ -53,6 +53,7 @@ enum cgroup_bpf_attach_type {
 	MAX_CGROUP_BPF_ATTACH_TYPE
 };
 
+/* Locking: see cgroup_bpf_mutex in kernel/bpf/cgroup.c */
 struct cgroup_bpf {
 	/* array of effective progs in this cgroup */
 	struct bpf_prog_array __rcu *effective[MAX_CGROUP_BPF_ATTACH_TYPE];
include/linux/bpf-cgroup.h +3 -0
index 02ef7f899b3d..dfa7a75507c9 100644
@@ -421,6 +421,9 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,
 	__ret;								       \
 })
 
+/* Take cgroup_mutex first if you need both.  See kernel/bpf/cgroup.c. */
+void cgroup_bpf_lock(void);
+void cgroup_bpf_unlock(void);
 int cgroup_bpf_prog_attach(const union bpf_attr *attr,
 			   enum bpf_prog_type ptype, struct bpf_prog *prog);
 int cgroup_bpf_prog_detach(const union bpf_attr *attr,
kernel/bpf/cgroup.c +92 -45

386 lines not on this page · raw message

kernel/bpf/local_storage.c +6 -2
index 83cd527a2542..2cfbddeb1c7e 100644
@@ -345,14 +345,18 @@ static void cgroup_storage_map_free(struct bpf_map *_map)
 	struct list_head *storages = &map->list;
 	struct bpf_cgroup_storage *storage, *stmp;
 
-	cgroup_lock();
+	/*
+	 * Attach and cgroup_bpf_release() change the same cgroup storage
+	 * lists under other maps' locks, so take cgroup_bpf_mutex.
+	 */
+	cgroup_bpf_lock();
 
 	list_for_each_entry_safe(storage, stmp, storages, list_map) {
 		bpf_cgroup_storage_unlink(storage);
 		bpf_cgroup_storage_free(storage);
 	}
 
-	cgroup_unlock();
+	cgroup_bpf_unlock();
 
 	WARN_ON(!RB_EMPTY_ROOT(&map->root));
 	WARN_ON(!list_empty(&map->list));
Signature · 1 line
2.53.0-Meta
#4 lore raw

Re: [PATCH bpf-next 3/3] bpf, cgroup: Protect cgroup_bpf state with its own mutex

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [Medium] bpf: lost error codes in…

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Medium] bpf: lost error codes in __cgroup_bpf_query() [1] --

commit 20190389f8922442abed742617dfa9e59bf26340
Author: Shakeel Butt <[email protected]>

bpf, cgroup: Protect cgroup_bpf state with its own mutex

This commit introduces a dedicated cgroup_bpf_mutex to protect cgroup BPF state so that operations not modifying the hierarchy do not have to contend on cgroup_mutex.

Quoted 4 lines
kernel/bpf/cgroup.c +0 -0
index 334237096759f..ae082ac86130b 100644

[ ... ]

kernel/bpf/cgroup.c static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · line 1571

Quoted 9 lines
kernel/bpf/cgroup.c +1 -1
@@ -1518,7 +1565,7 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
 	for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {
 		if (effective_query) {
 			effective = rcu_dereference_protected(cgrp->bpf.effective[atype],
-							      lockdep_is_held(&cgroup_mutex));
+							      cgroup_bpf_held());
 			cnt = min_t(int, bpf_cgroup_array_length(effective, atype), total_cnt);
 			ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype);
 		} else {

[Severity: Medium]
This problem wasn't introduced by this patch, but can assigning ret from bpf_cgroup_array_copy_to_user() drop an earlier -ENOSPC or -EFAULT error [1]?

Before the loop, __cgroup_bpf_query() sets ret = -ENOSPC when attr->query.prog_cnt < total_cnt:

kernel/bpf/cgroup.c:__cgroup_bpf_query() {
	...
	if (attr->query.prog_cnt < total_cnt) {
		total_cnt = attr->query.prog_cnt;
		ret = -ENOSPC;
	}

	for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {
		if (effective_query) {
			...
			ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype);
	...
}

When effective_query is set, bpf_cgroup_array_copy_to_user() overwrites ret with 0 when the truncated slice fits in the array, or a later iteration (for BPF_LSM_CGROUP) can overwrite an earlier -EFAULT, causing __cgroup_bpf_query() to return 0 on truncated or faulted queries.

[1] https://sashiko.dev/#/bug/linux-20442c38-bcc8-440c-9711-1977ea6cd04b

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#5 lore raw

Re: [PATCH bpf-next 3/3] bpf, cgroup: Protect cgroup_bpf state with its own mutex

Quoted 54 lines · On Thu, Nov 12, 2026 at 05:41:57PM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Medium] bpf: lost error codes in __cgroup_bpf_query() [1] --

commit 20190389f8922442abed742617dfa9e59bf26340
Author: Shakeel Butt <[email protected]>

bpf, cgroup: Protect cgroup_bpf state with its own mutex

This commit introduces a dedicated cgroup_bpf_mutex to protect cgroup BPF state so that operations not modifying the hierarchy do not have to contend on cgroup_mutex.

Quoted 4 lines
kernel/bpf/cgroup.c +0 -0
index 334237096759f..ae082ac86130b 100644

[ ... ]

kernel/bpf/cgroup.c static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · line 1571

Quoted 9 lines
kernel/bpf/cgroup.c +1 -1
@@ -1518,7 +1565,7 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
 	for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {
 		if (effective_query) {
 			effective = rcu_dereference_protected(cgrp->bpf.effective[atype],
-							      lockdep_is_held(&cgroup_mutex));
+							      cgroup_bpf_held());
 			cnt = min_t(int, bpf_cgroup_array_length(effective, atype), total_cnt);
 			ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype);
 		} else {

[Severity: Medium]
This problem wasn't introduced by this patch, but can assigning ret from bpf_cgroup_array_copy_to_user() drop an earlier -ENOSPC or -EFAULT error [1]?

Before the loop, __cgroup_bpf_query() sets ret = -ENOSPC when attr->query.prog_cnt < total_cnt:

kernel/bpf/cgroup.c:__cgroup_bpf_query() {
	...
	if (attr->query.prog_cnt < total_cnt) {
		total_cnt = attr->query.prog_cnt;
		ret = -ENOSPC;
	}

	for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {
		if (effective_query) {
			...
			ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype);
	...
}
Quoted 6 lines

When effective_query is set, bpf_cgroup_array_copy_to_user() overwrites ret with 0 when the truncated slice fits in the array, or a later iteration (for BPF_LSM_CGROUP) can overwrite an earlier -EFAULT, causing __cgroup_bpf_query() to return 0 on truncated or faulted queries.

[1] https://sashiko.dev/#/bug/linux-20442c38-bcc8-440c-9711-1977ea6cd04b

Seems like an independent bug. I will send the fix separately.

maintainer Acked-by on the v1 series
#6 lore raw
In reply to Shakeel Butt
Quoted 5 lines · Shakeel Butt

In the Meta fleet, BPF_PROG_QUERY callers, link-info readers and the cgroup bpf release work often wait minutes for cgroup_mutex. The holder is almost always a cgroup mkdir or rmdir stuck on kernfs_rwsem. In one day, hung-task reports showed this on 1,075 hosts (query), 94 hosts (link info) and 458 hosts (release work).

For the series,

Thanks.

--
tejun

#7 lore raw
In reply to Shakeel Butt
This series was applied to bpf/bpf.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf.git (master)
by Alexei Starovoitov <[email protected]>:

Quoted 11 lines · On Thu, 12 Nov 2026 09:27:12 -0800 you wrote:

In the Meta fleet, BPF_PROG_QUERY callers, link-info readers and the cgroup bpf release work often wait minutes for cgroup_mutex. The holder is almost always a cgroup mkdir or rmdir stuck on kernfs_rwsem. In one day, hung-task reports showed this on 1,075 hosts (query), 94 hosts (link info) and 458 hosts (release work).

Patch 3 gives the cgroup bpf state its own mutex, so these paths no longer take cgroup_mutex. Attach, detach and replace still take cgroup_mutex first, because they walk the descendants.

[...]

Here is the summary with links:
- [bpf-next,1/3] bpf, cgroup: Don't read freed effective arrays in BPF_PROG_QUERY

    https://git.kernel.org/bpf/bpf/c/90f4e698f9c1
  - [bpf-next,2/3] bpf, cgroup: Don't attach to a removed cgroup
    https://git.kernel.org/bpf/bpf/c/b2e007371909
  - [bpf-next,3/3] bpf, cgroup: Protect cgroup_bpf state with its own mutex
    https://git.kernel.org/bpf/bpf/c/40aec6bab02e

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html