bpf, cgroup: Don't wait for cgroup_mutex to read or free cgroup bpf state
cover bpf, cgroup: Don't wait for cgroup_mutex to read or free cgroup bpf state
In the Meta fleet, BPF_PROG_QUERY callers, link-info readers and the cgroup bpf release work often wait minutes for cgroup_mutex. The holder is almost always a cgroup mkdir or rmdir stuck on kernfs_rwsem. In one day, hung-task reports showed this on 1,075 hosts (query), 94 hosts (link info) and 458 hosts (release work).
Patch 3 gives the cgroup bpf state its own mutex, so these paths no longer take cgroup_mutex. Attach, detach and replace still take cgroup_mutex first, because they walk the descendants.
Patches 1 and 2 fix two old races. Attach and query get the cgroup from an fd and take the lock later, without checking whether the cgroup's bpf state was released in between. An effective query then reads freed memory (patch 1). Attach leaks the program, or with cgroup storage writes to a freed cgroup later (patch 2). Patch 3 makes the attach race easy to hit, so patch 2 must come before it. Both fixes have Fixes tags.
Testing, in an x86-64 VM with lockdep, PROVE_RCU and KASAN:
- The 41 cgroup bpf selftests pass. A 60-second stress of mkdir/rmdir, attach/detach, links, queries, fdinfo, storage map free and sockets in dying cgroups shows no lockdep or KASAN reports.
- Debug-only delays reproduce both races. Without the fixes, KASAN reports use-after-free in cgroup_bpf_prog_query() and, with patch 3, in cgroup_storage_map_free(), and BPF_PROG_ATTACH leaks the program. With the fixes, attach and query return -ENOENT.
- While a mkdir holds cgroup_mutex for 5 seconds, query and link info return in under a millisecond instead of 4.6 seconds. Attach and detach still wait.
- Builds with CGROUP_BPF=n and with BPF_LSM=n.
Shakeel Butt (3):
bpf, cgroup: Don't read freed effective arrays in BPF_PROG_QUERY bpf, cgroup: Don't attach to a removed cgroup
bpf, cgroup: Protect cgroup_bpf state with its own mutex
include/linux/bpf-cgroup-defs.h | 1 + include/linux/bpf-cgroup.h | 3 + kernel/bpf/cgroup.c | 158 +++++++++++++++++++++++--------- kernel/bpf/local_storage.c | 8 +- 4 files changed, 123 insertions(+), 47 deletions(-)
base-commit: 0fb18ce8a2e9de830433923cf65838933d615359
Signature · 1 line
2.53.0-Meta
PATCH 1/3 bpf, cgroup: Don't read freed effective arrays in BPF_PROG_QUERY
BPF_PROG_QUERY gets the cgroup from an fd before it takes cgroup_mutex. If the cgroup is removed in between and its bpf refcount drops to zero, cgroup_bpf_release() can run first. It frees the effective arrays but leaves the pointers, so a BPF_F_QUERY_EFFECTIVE query then reads freed memory.
The window is small because the release work also takes cgroup_mutex.
Check under the lock that the bpf refcount isn't zero, and return -ENOENT if it is, the error the fd lookup gives for a removed cgroup. The release work only runs after the count hits zero, and it takes the same lock.
Reproduced in a VM with a debug-only delay between the fd lookup and the
lock: KASAN reports a use-after-free in cgroup_bpf_prog_query(). With this patch, the query returns -ENOENT.
--- kernel/bpf/cgroup.c | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/bpf/cgroup.c +8 -0
@@ -1446,6 +1446,14 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,if (effective_query && prog_attach_flags)return -EINVAL ; +/* +* cgroup_bpf_release() frees the effective arrays once the bpf +* refcount is zero. It takes this lock too, so a nonzero count means +* the arrays are still valid. +*/ +if (percpu_ref_is_zero (&cgrp->bpf.refcnt)) +return -ENOENT ; +if (type ==BPF_STRUCT_OPS ) {u32 type_id = attr->query.type_id;
Signature · 1 line
2.53.0-Meta
PATCH 2/3 bpf, cgroup: Don't attach to a removed cgroup
In reply to Shakeel ButtAttach gets the cgroup from an fd before it takes cgroup_mutex. If the cgroup is removed in between and cgroup_bpf_release() runs first, attach adds a program that never runs:
- With BPF_PROG_ATTACH the program leaks, since nothing can detach it.
- With a link that uses cgroup storage, the storage stays on the freed cgroup's list, and freeing the map later writes to the freed cgroup.
- A cgroup struct_ops link is attached but never called.
The window is small today because the release work also waits for cgroup_mutex. The next patch stops that, which makes the race easy to hit.
Check cgroup_is_dead() under the lock and return -ENOENT, the error the fd lookup gives for a removed cgroup. rmdir marks the cgroup dead under cgroup_mutex before it starts the release, so the check is exact.
Reproduced in a VM with debug-only delays. With the next patch applied, a link attach queued behind a slow mkdir lands on the released cgroup, and freeing the map then hits a KASAN use-after-free in cgroup_storage_map_free(). BPF_PROG_ATTACH leaks the program. With this patch, attach returns -ENOENT.
--- kernel/bpf/cgroup.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/bpf/cgroup.c +13 -0
@@ -988,6 +988,13 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,/* replace_prog implies BPF_F_REPLACE, and vice versa */ return -EINVAL ; +/* +* The cgroup may have been removed since the caller looked it up. +* Then its bpf state may be freed, and nothing would detach this. +*/ +if (cgroup_is_dead (cgrp)) +return -ENOENT ; + atype =bpf_cgroup_atype_find (type, new_prog->aux->attach_btf_id);if (atype <0 )return -EINVAL ;
@@ -3170,6 +3177,12 @@ int cgroup_bpf_struct_ops_attach(struct bpf_map *map, const union bpf_attr *attrcgroup_lock (); +/* See the cgroup_is_dead() check in __cgroup_bpf_attach() */ +if (cgroup_is_dead (cgrp)) { + err = -ENOENT ; + goto unlock; + } +if (attr->link_create.cgroup.expected_revision && attr->link_create.cgroup.expected_revision != cgrp->bpf.revisions[atype]) { err = -ESTALE ;
Signature · 1 line
2.53.0-Meta
PATCH 3/3 bpf, cgroup: Protect cgroup_bpf state with its own mutex
In reply to Shakeel ButtIn the Meta fleet, tasks that only read or free cgroup bpf state often wait minutes for cgroup_mutex:
- In one day, hung-task reports (blocked over 120 seconds) showed this for BPF_PROG_QUERY on 1,075 hosts, the cgroup bpf release work on 458 hosts and link info on 94 hosts.
- The query callers are a container manager's health checks. Over 30 days, 6,700 hosts reported one hung. The typical wait is 11 minutes, the longest 75 minutes.
- The cgroup_mutex holder is almost always a cgroup mkdir or rmdir that is waiting for kernfs_rwsem.
Only attach, detach and replace need cgroup_mutex. They update the effective arrays of all descendants, so the hierarchy must not change.
Add cgroup_bpf_mutex for the bpf state. Attach, detach and replace take cgroup_mutex and then cgroup_bpf_mutex. Everything else (query, link info, the release work, dropping an LSM attach type, freeing a cgroup storage map) takes only cgroup_bpf_mutex.
cgroup_bpf_inherit() still runs at mkdir with only cgroup_mutex, so mkdir never waits for cgroup_bpf_mutex. It writes the new cgroup, which no one else can see yet. What it reads from the ancestors changes only under cgroup_mutex, and the release work can't run for them because they are online.
The release work no longer waits for cgroup_mutex, so attach must check that the cgroup is still alive. The previous patch does that.
With a test delay that makes a mkdir hold cgroup_mutex for 5 seconds, BPF_PROG_QUERY and link info now return in under a millisecond instead of 4.6 seconds. Attach and detach still wait.
--- include/linux/bpf-cgroup-defs.h | 1 + include/linux/bpf-cgroup.h | 3 + kernel/bpf/cgroup.c | 137 +++++++++++++++++++++----------- kernel/bpf/local_storage.c | 8 +- 4 files changed, 102 insertions(+), 47 deletions(-)
include/linux/bpf-cgroup-defs.h +1 -0
@@ -53,6 +53,7 @@ enum cgroup_bpf_attach_type {MAX_CGROUP_BPF_ATTACH_TYPE }; +/* Locking: see cgroup_bpf_mutex in kernel/bpf/cgroup.c */ struct cgroup_bpf {/* array of effective progs in this cgroup */ struct bpf_prog_array __rcu *effective[MAX_CGROUP_BPF_ATTACH_TYPE ];
include/linux/bpf-cgroup.h +3 -0
@@ -421,6 +421,9 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk, __ret; \ }) +/* Take cgroup_mutex first if you need both. See kernel/bpf/cgroup.c. */ +void cgroup_bpf_lock (void ); +void cgroup_bpf_unlock (void );int cgroup_bpf_prog_attach (const union bpf_attr *attr,enum bpf_prog_type ptype,struct bpf_prog *prog); int cgroup_bpf_prog_detach(const union bpf_attr *attr,
kernel/bpf/cgroup.c +92 -45
386 lines not on this page · raw message
kernel/bpf/local_storage.c +6 -2
@@ -345,14 +345,18 @@ static void cgroup_storage_map_free(struct bpf_map *_map)struct list_head *storages = &map->list;struct bpf_cgroup_storage *storage, *stmp; -cgroup_lock (); +/* +* Attach and cgroup_bpf_release() change the same cgroup storage +* lists under other maps' locks, so take cgroup_bpf_mutex. +*/ +cgroup_bpf_lock ();list_for_each_entry_safe (storage, stmp, storages, list_map) {bpf_cgroup_storage_unlink (storage);bpf_cgroup_storage_free (storage); } -cgroup_unlock (); +cgroup_bpf_unlock ();WARN_ON (!RB_EMPTY_ROOT (&map->root));WARN_ON (!list_empty (&map->list));
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH bpf-next 3/3] bpf, cgroup: Protect cgroup_bpf state with its own mutex
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [Medium] bpf: lost error codes in…
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Medium] bpf: lost error codes in __cgroup_bpf_query() [1] --
commit 20190389f8922442abed742617dfa9e59bf26340
Author: Shakeel Butt <[email protected]>
bpf, cgroup: Protect cgroup_bpf state with its own mutex
This commit introduces a dedicated cgroup_bpf_mutex to protect cgroup BPF state so that operations not modifying the hierarchy do not have to contend on cgroup_mutex.
Quoted 4 lines
kernel/bpf/cgroup.c +0
-0
[ ... ]
kernel/bpf/cgroup.c static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · line 1571
Quoted 9 lines
kernel/bpf/cgroup.c +1
-1@@ -1518,7 +1565,7 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {if (effective_query) { effective =rcu_dereference_protected (cgrp->bpf.effective[atype], -lockdep_is_held (&cgroup_mutex)); +cgroup_bpf_held ()); cnt =min_t (int,bpf_cgroup_array_length (effective, atype), total_cnt); ret =bpf_cgroup_array_copy_to_user (effective, prog_ids, cnt, atype); }else {
[Severity: Medium]
This problem wasn't introduced by this patch, but can assigning ret from bpf_cgroup_array_copy_to_user() drop an earlier -ENOSPC or -EFAULT error [1]?
Before the loop, __cgroup_bpf_query() sets ret = -ENOSPC when attr->query.prog_cnt < total_cnt:
kernel/bpf/cgroup.c:__cgroup_bpf_query() {
...
if (attr->query.prog_cnt < total_cnt) {
total_cnt = attr->query.prog_cnt;
ret = -ENOSPC;
}
for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {
if (effective_query) {
...
ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype);
...
}When effective_query is set, bpf_cgroup_array_copy_to_user() overwrites ret with 0 when the truncated slice fits in the array, or a later iteration (for BPF_LSM_CGROUP) can overwrite an earlier -EFAULT, causing __cgroup_bpf_query() to return 0 on truncated or faulted queries.
[1] https://sashiko.dev/#/bug/linux-20442c38-bcc8-440c-9711-1977ea6cd04b
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH bpf-next 3/3] bpf, cgroup: Protect cgroup_bpf state with its own mutex
Quoted 54 lines · On Thu, Nov 12, 2026 at 05:41:57PM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Medium] bpf: lost error codes in__cgroup_bpf_query()[1] --commit
20190389f8922442abed742617dfa9e59bf26340
Author: Shakeel Butt <[email protected]>bpf, cgroup: Protect
cgroup_bpfstate with its own mutexThis commit introduces a dedicated
cgroup_bpf_mutexto protect cgroup BPF state so that operations not modifying the hierarchy do not have to contend oncgroup_mutex.Quoted 4 lines
kernel/bpf/cgroup.c +0
-0[ ... ]
kernel/bpf/cgroup.c static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · line 1571
Quoted 9 lines
kernel/bpf/cgroup.c +1
-1@@ -1518,7 +1565,7 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,for (atype = from_atype; atype <= to_atype && total_cnt; atype++) {if (effective_query) { effective =rcu_dereference_protected (cgrp->bpf.effective[atype], -lockdep_is_held (&cgroup_mutex)); +cgroup_bpf_held ()); cnt =min_t (int,bpf_cgroup_array_length (effective, atype), total_cnt); ret =bpf_cgroup_array_copy_to_user (effective, prog_ids, cnt, atype); }else {[Severity: Medium]
This problem wasn't introduced by this patch, but can assigning ret frombpf_cgroup_array_copy_to_user()drop an earlier-ENOSPCor-EFAULTerror [1]?Before the loop,
__cgroup_bpf_query()sets ret =-ENOSPCwhenattr->query.prog_cnt<total_cnt:kernel/bpf/cgroup.c:__cgroup_bpf_query() { ... if (attr->query.prog_cnt < total_cnt) { total_cnt = attr->query.prog_cnt; ret = -ENOSPC; } for (atype = from_atype; atype <= to_atype && total_cnt; atype++) { if (effective_query) { ... ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype); ... }
Quoted 6 lines
When
effective_queryis set,bpf_cgroup_array_copy_to_user()overwrites ret with 0 when the truncated slice fits in the array, or a later iteration (forBPF_LSM_CGROUP) can overwrite an earlier-EFAULT, causing__cgroup_bpf_query()to return 0 on truncated or faulted queries.[1] https://sashiko.dev/#/bug/linux-20442c38-bcc8-440c-9711-1977ea6cd04b
Seems like an independent bug. I will send the fix separately.
Quoted 5 lines · Shakeel Butt
In the Meta fleet,
BPF_PROG_QUERYcallers, link-info readers and the cgroup bpf release work often wait minutes forcgroup_mutex. The holder is almost always a cgroup mkdir or rmdir stuck onkernfs_rwsem. In one day, hung-task reports showed this on 1,075 hosts (query), 94 hosts (link info) and 458 hosts (release work).
For the series,
Thanks.
--
tejun
This series was applied to bpf/bpf.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf.git (master)
by Alexei Starovoitov <[email protected]>:
Quoted 11 lines · On Thu, 12 Nov 2026 09:27:12 -0800 you wrote:
In the Meta fleet,
BPF_PROG_QUERYcallers, link-info readers and the cgroup bpf release work often wait minutes forcgroup_mutex. The holder is almost always a cgroup mkdir or rmdir stuck onkernfs_rwsem. In one day, hung-task reports showed this on 1,075 hosts (query), 94 hosts (link info) and 458 hosts (release work).Patch 3 gives the cgroup bpf state its own mutex, so these paths no longer take
cgroup_mutex. Attach, detach and replace still takecgroup_mutexfirst, because they walk the descendants.[...]
Here is the summary with links:
- [bpf-next,1/3] bpf, cgroup: Don't read freed effective arrays in BPF_PROG_QUERY
https://git.kernel.org/bpf/bpf/c/90f4e698f9c1 - [bpf-next,2/3] bpf, cgroup: Don't attach to a removed cgroup https://git.kernel.org/bpf/bpf/c/b2e007371909 - [bpf-next,3/3] bpf, cgroup: Protect cgroup_bpf state with its own mutex https://git.kernel.org/bpf/bpf/c/40aec6bab02e
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html