Skip to contentSkip to conversation

mm: BPF OOM

PATCH In review Roman Gushchin 11 participants 65 messages Jan 27 – Apr 29

Subsystems (11)
lore
Share & tools
lore

Roman Gushchin

author maintainer
#0 lore raw

v3 cover mm: BPF OOM

This patchset adds an ability to customize the out of memory handling using bpf.

It focuses on two parts:
1) OOM handling policy,
2) PSI-based OOM invocation.

The idea to use bpf for customizing the OOM handling is not new, but unlike the previous proposal [1], which augmented the existing task ranking policy, this one tries to be as generic as possible and leverage the full power of the modern bpf.

It provides a generic interface which is called before the existing OOM killer code and allows implementing any policy, e.g. picking a victim task or memory cgroup or potentially even releasing memory in other ways, e.g. deleting tmpfs files (the last one might require some additional but relatively simple changes).

The past attempt to implement memory-cgroup aware policy [2] showed that there are multiple opinions on what the best policy is. As it's highly workload-dependent and specific to a concrete way of organizing workloads, the structure of the cgroup tree etc, a customizable bpf-based implementation is preferable over an in-kernel implementation with a dozen of sysctls.

The second part is related to the fundamental question on when to declare the OOM event. It's a trade-off between the risk of unnecessary OOM kills and associated work losses and the risk of infinite trashing and effective soft lockups. In the last few years several PSI-based userspace solutions were developed (e.g. OOMd [3] or systemd-OOMd [4]). The common idea was to use userspace daemons to implement custom OOM logic as well as rely on PSI monitoring to avoid stalls. In this scenario the userspace daemon was supposed to handle the majority of OOMs, while the in-kernel OOM killer worked as the last resort measure to guarantee that the system would never deadlock on the memory. But this approach creates additional infrastructure
churn: userspace OOM daemon is a separate entity which needs to be deployed, updated, monitored. A completely different pipeline needs to be built to monitor both types of OOM events and collect associated logs. A userspace daemon is more restricted in terms on what data is available to it. Implementing a daemon which can work reliably under a heavy memory pressure in the system is also tricky.

This patchset includes the code, tests and many ideas from the patchset of JP Kobryn, which implemented bpf kfuncs to provide a faster method to access memcg data [5].

[1]: https://lwn.net/ml/linux-kernel/[email protected]/
[2]: https://lore.kernel.org/lkml/[email protected]/
[3]: https://github.com/facebookincubator/oomd
[4]: https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html
[5]: https://lkml.org/lkml/2025/10/15/1554

---

v3:
1) Replaced bpf_psi struct ops with a tracepoint in psi_avgs_work() (Tejun H.)
2) Updated bpf_oom struct ops:
- removed bpf_oom_ctx, passing bpf_struct_ops_link instead (by Alexei S.)
- removed handle_cgroup_offline callback.
3) Updated kfuncs:
- bpf_out_of_memory() dropped constraint_text argument (by Michal H.)
- bpf_oom_kill_process() added check for OOM_SCORE_ADJ_MIN.
4) Libbpf: updated bpf_map__attach_struct_ops_opts to use target_fd. (by Alexei S.)

v2:
1) A single bpf_oom can be attached system-wide and a single bpf_oom per memcg. (by Alexei Starovoitov)
2) Initial support for attaching struct ops to cgroups (Martin KaFai Lau, Andrii Nakryiko and others)
3) bpf memcontrol kfuncs enhancements and tests (co-developed by JP Kobryn)
4) Many mall-ish fixes and cleanups (suggested by Andrew Morton, Suren Baghdasaryan, Andrii Nakryiko and Kumar Kartikeya Dwivedi)
5) bpf_out_of_memory() is taking u64 flags instead of bool wait_on_oom_lock (suggested by Kumar Kartikeya Dwivedi)
6) bpf_get_mem_cgroup() got KF_RCU flag (suggested by Kumar Kartikeya Dwivedi)
7) cgroup online and offline callbacks for bpf_psi, cgroup offline for bpf_oom

v1:
1) Both OOM and PSI parts are now implemented using bpf struct ops, providing a path the future extensions (suggested by Kumar Kartikeya Dwivedi, Song Liu and Matt Bobrowski)
2) It's possible to create PSI triggers from BPF, no need for an additional userspace agent. (suggested by Suren Baghdasaryan)
Also there is now a callback for the cgroup release event.
3) Added an ability to block on oom_lock instead of bailing out (suggested by Michal Hocko)
4) Added bpf_task_is_oom_victim (suggested by Michal Hocko)
5) PSI callbacks are scheduled using a separate workqueue (suggested by Suren Baghdasaryan)

RFC:
https://lwn.net/ml/all/[email protected]/

JP Kobryn (1):
bpf: selftests: add config for psi

Roman Gushchin (16):
bpf: move bpf_struct_ops_link into bpf.h
bpf: allow attaching struct_ops to cgroups
libbpf: fix return value on memory allocation failure
libbpf: introduce bpf_map__attach_struct_ops_opts()
bpf: mark struct oom_control's memcg field as TRUSTED_OR_NULL
mm: define mem_cgroup_get_from_ino() outside of CONFIG_SHRINKER_DEBUG
mm: introduce BPF OOM struct ops
mm: introduce bpf_oom_kill_process() bpf kfunc
mm: introduce bpf_out_of_memory() BPF kfunc
mm: introduce bpf_task_is_oom_victim() kfunc
bpf: selftests: introduce read_cgroup_file() helper
bpf: selftests: BPF OOM struct ops test
sched: psi: add a trace point to psi_avgs_work()
sched: psi: add cgroup_id field to psi_group structure
bpf: allow calling bpf_out_of_memory() from a PSI tracepoint
bpf: selftests: PSI struct ops test

 MAINTAINERS                                   |   2 +
 include/linux/bpf-cgroup-defs.h               |   6 +
 include/linux/bpf-cgroup.h                    |  16 ++
 include/linux/bpf.h                           |  10 +
 include/linux/bpf_oom.h                       |  46 ++++
 include/linux/memcontrol.h                    |   4 +-
 include/linux/oom.h                           |  13 +
 include/linux/psi_types.h                     |   4 +
 include/trace/events/psi.h                    |  27 ++
 include/uapi/linux/bpf.h                      |   3 +
 kernel/bpf/bpf_struct_ops.c                   |  77 +++++-
 kernel/bpf/cgroup.c                           |  46 ++++
 kernel/bpf/verifier.c                         |   5 +
 kernel/sched/psi.c                            |   7 +
 mm/Makefile                                   |   2 +-
 mm/bpf_oom.c                                  | 192 +++++++++++++
 mm/memcontrol.c                               |   2 -
 mm/oom_kill.c                                 | 202 ++++++++++++++
 tools/include/uapi/linux/bpf.h                |   1 +
 tools/lib/bpf/libbpf.c                        |  22 +-
 tools/lib/bpf/libbpf.h                        |  14 +
 tools/lib/bpf/libbpf.map                      |   1 +
 tools/testing/selftests/bpf/cgroup_helpers.c  |  45 +++
 tools/testing/selftests/bpf/cgroup_helpers.h  |   3 +
 tools/testing/selftests/bpf/config            |   1 +
 .../selftests/bpf/prog_tests/test_oom.c       | 256 ++++++++++++++++++
 .../selftests/bpf/prog_tests/test_psi.c       | 225 +++++++++++++++
 tools/testing/selftests/bpf/progs/test_oom.c  | 111 ++++++++
 tools/testing/selftests/bpf/progs/test_psi.c  |  90 ++++++
 29 files changed, 1412 insertions(+), 21 deletions(-)
 create mode 100644 include/linux/bpf_oom.h
 create mode 100644 include/trace/events/psi.h
 create mode 100644 mm/bpf_oom.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_oom.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_psi.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_oom.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_psi.c
Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH v3 1/17 bpf: move bpf_struct_ops_link into bpf.h

Move struct bpf_struct_ops_link's definition into bpf.h, where other custom bpf links definitions are.

It's necessary to access its members from outside of generic bpf_struct_ops implementation, which will be done by following patches in the series.

---
 include/linux/bpf.h         | 6 ++++++
 kernel/bpf/bpf_struct_ops.c | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/bpf.h +6 -0
index 4427c6e98331..899dd911dc82 100644
@@ -1891,6 +1891,12 @@ struct bpf_raw_tp_link {
 	u64 cookie;
 };
 
+struct bpf_struct_ops_link {
+	struct bpf_link link;
+	struct bpf_map __rcu *map;
+	wait_queue_head_t wait_hup;
+};
+
 struct bpf_link_primer {
 	struct bpf_link *link;
 	struct file *file;
kernel/bpf/bpf_struct_ops.c +0 -6
index c43346cb3d76..de01cf3025b3 100644
@@ -55,12 +55,6 @@ struct bpf_struct_ops_map {
 	struct bpf_struct_ops_value kvalue;
 };
 
-struct bpf_struct_ops_link {
-	struct bpf_link link;
-	struct bpf_map __rcu *map;
-	wait_queue_head_t wait_hup;
-};
-
 static DEFINE_MUTEX(update_mutex);
 
 #define VALUE_PREFIX "bpf_struct_ops_"
Signature · 1 line
2.52.0
Acked-by on v3 1/17
#2 lore raw

Re: [PATCH bpf-next v3 01/17] bpf: move bpf_struct_ops_link into bpf.h

Quoted 9 lines · Roman Gushchin

Move struct bpf_struct_ops_link's definition into bpf.h, where other custom bpf links definitions are.

It's necessary to access its members from outside of generic bpf_struct_ops implementation, which will be done by following patches in the series.

Feel free to add:

Quoted 43 lines
---
 include/linux/bpf.h         | 6 ++++++
 kernel/bpf/bpf_struct_ops.c | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/bpf.h +6 -0
index 4427c6e98331..899dd911dc82 100644
@@ -1891,6 +1891,12 @@ struct bpf_raw_tp_link {
        u64 cookie;
 };
 
+struct bpf_struct_ops_link {
+       struct bpf_link link;
+       struct bpf_map __rcu *map;
+       wait_queue_head_t wait_hup;
+};
+
 struct bpf_link_primer {
        struct bpf_link *link;
        struct file *file;
kernel/bpf/bpf_struct_ops.c +0 -7
index c43346cb3d76..de01cf3025b3 100644
@@ -55,12 +55,6 @@ struct bpf_struct_ops_map {
        struct bpf_struct_ops_value kvalue;
 };
 
-struct bpf_struct_ops_link {
-       struct bpf_link link;
-       struct bpf_map __rcu *map;
-       wait_queue_head_t wait_hup;
-};
-
 static DEFINE_MUTEX(update_mutex);
 
 #define VALUE_PREFIX "bpf_struct_ops_"
--

2.52.0

Signature · 2 lines
Regards
Yafang
Acked-by on v3 1/17
#3 lore raw

Re: [PATCH bpf-next v3 01/17] bpf: move bpf_struct_ops_link into bpf.h

In reply to Roman Gushchin
Quoted 6 lines · Roman Gushchin

Move struct bpf_struct_ops_link's definition into bpf.h, where other custom bpf links definitions are.

It's necessary to access its members from outside of generic bpf_struct_ops implementation, which will be done by following patches in the series.

Looks OK to me:

Quoted 43 lines
---
 include/linux/bpf.h         | 6 ++++++
 kernel/bpf/bpf_struct_ops.c | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/bpf.h +6 -0
index 4427c6e98331..899dd911dc82 100644
@@ -1891,6 +1891,12 @@ struct bpf_raw_tp_link {
 	u64 cookie;
 };
 
+struct bpf_struct_ops_link {
+	struct bpf_link link;
+	struct bpf_map __rcu *map;
+	wait_queue_head_t wait_hup;
+};
+
 struct bpf_link_primer {
 	struct bpf_link *link;
 	struct file *file;
kernel/bpf/bpf_struct_ops.c +0 -6
index c43346cb3d76..de01cf3025b3 100644
@@ -55,12 +55,6 @@ struct bpf_struct_ops_map {
 	struct bpf_struct_ops_value kvalue;
 };
 
-struct bpf_struct_ops_link {
-	struct bpf_link link;
-	struct bpf_map __rcu *map;
-	wait_queue_head_t wait_hup;
-};
-
 static DEFINE_MUTEX(update_mutex);
 
 #define VALUE_PREFIX "bpf_struct_ops_"
Signature · 1 line
2.52.0
author maintainer
#4 lore raw

PATCH v3 2/17 bpf: allow attaching struct_ops to cgroups

In reply to Roman Gushchin

Introduce an ability to attach bpf struct_ops'es to cgroups.

From user's standpoint it works in the following way:
a user passes a BPF_F_CGROUP_FD flag and specifies the target cgroup fd while creating a struct_ops link. As the result, the bpf struct_ops link will be created and attached to a cgroup.

The cgroup.bpf structure maintains a list of attached struct ops links. If the cgroup is getting deleted, attached struct ops'es are getting auto-detached and the userspace program gets a notification.

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

---
 include/linux/bpf-cgroup-defs.h |  3 ++
 include/linux/bpf-cgroup.h      | 16 +++++++++
 include/linux/bpf.h             |  3 ++
 include/uapi/linux/bpf.h        |  3 ++
 kernel/bpf/bpf_struct_ops.c     | 59 ++++++++++++++++++++++++++++++---
 kernel/bpf/cgroup.c             | 46 +++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h  |  1 +
 7 files changed, 127 insertions(+), 4 deletions(-)
include/linux/bpf-cgroup-defs.h +3 -0
index c9e6b26abab6..6c5e37190dad 100644
@@ -71,6 +71,9 @@ struct cgroup_bpf {
 	/* temp storage for effective prog array used by prog_attach/detach */
 	struct bpf_prog_array *inactive;
 
+	/* list of bpf struct ops links */
+	struct list_head struct_ops_links;
+
 	/* reference counter used to detach bpf programs after cgroup removal */
 	struct percpu_ref refcnt;
 
include/linux/bpf-cgroup.h +16 -0
index 2f535331f926..a6c327257006 100644
@@ -423,6 +423,11 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
 int cgroup_bpf_prog_query(const union bpf_attr *attr,
 			  union bpf_attr __user *uattr);
 
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+				 struct bpf_struct_ops_link *link);
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+				  struct bpf_struct_ops_link *link);
+
 const struct bpf_func_proto *
 cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
 #else
@@ -451,6 +456,17 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
 	return -EINVAL;
 }
 
+static inline int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+					       struct bpf_struct_ops_link *link)
+{
+	return -EINVAL;
+}
+
+static inline void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+						struct bpf_struct_ops_link *link)
+{
+}
+
 static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
 					union bpf_attr __user *uattr)
 {
include/linux/bpf.h +3 -0
index 899dd911dc82..391888eb257c 100644
@@ -1894,6 +1894,9 @@ struct bpf_raw_tp_link {
 struct bpf_struct_ops_link {
 	struct bpf_link link;
 	struct bpf_map __rcu *map;
+	struct cgroup *cgroup;
+	bool cgroup_removed;
+	struct list_head list;
 	wait_queue_head_t wait_hup;
 };
 
include/uapi/linux/bpf.h +3 -0
index 44e7dbc278e3..28544e8af1cd 100644
@@ -1237,6 +1237,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_CGROUP_FD		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
@@ -6775,6 +6776,8 @@ struct bpf_link_info {
 		} xdp;
 		struct {
 			__u32 map_id;
+			__u32 :32;
+			__u64 cgroup_id;
 		} struct_ops;
 		struct {
 			__u32 pf;
kernel/bpf/bpf_struct_ops.c +55 -4

135 lines not on this page · raw message

kernel/bpf/cgroup.c +46 -0

79 lines not on this page · raw message

tools/include/uapi/linux/bpf.h +1 -0
index 3ca7d76e05f0..d5492e60744a 100644
@@ -1237,6 +1237,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_CGROUP_FD		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
Signature · 1 line
2.52.0
#5 lore raw

Re: [PATCH bpf-next v3 02/17] bpf: allow attaching struct_ops to cgroups

Can this code access st_link after it has been freed?
Quoted 2 lines
kernel/bpf/cgroup.c +0 -0
index 69988af44b37..7b1903be6f69 100644

[ ... ]

Quoted 10 lines
kernel/bpf/cgroup.c +2 -0
@@ -307,12 +308,23 @@ static void cgroup_bpf_release(struct work_struct *work)
 					       bpf.release_work);
 	struct bpf_prog_array *old_array;
 	struct list_head *storages = &cgrp->bpf.storages;
+	struct bpf_struct_ops_link *st_link, *st_tmp;
 	struct bpf_cgroup_storage *storage, *stmp;
+	LIST_HEAD(st_links);
 
 	unsigned int atype;
 

kernel/bpf/cgroup.c static void cgroup_bpf_release(struct work_struct *work) · line 326

Quoted 10 lines
kernel/bpf/cgroup.c +8 -0
 	cgroup_lock();
 
+	list_splice_init(&cgrp->bpf.struct_ops_links, &st_links);
+	list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+		st_link->cgroup = NULL;
+		st_link->cgroup_removed = true;
+		cgroup_put(cgrp);
+		if (IS_ERR(bpf_link_inc_not_zero(&st_link->link)))
+			list_del(&st_link->list);
+	}

Can this code access st_link after it has been freed?

Once st_link->cgroup is set to NULL, if another CPU concurrently closes the link FD, bpf_struct_ops_map_link_dealloc() will see st_link->cgroup as NULL, skip the cgroup_bpf_detach_struct_ops() call, and proceed to kfree(st_link). Then the bpf_link_inc_not_zero() call here would access freed memory.

The race window is between setting st_link->cgroup = NULL and calling bpf_link_inc_not_zero(). During this window:

CPU 0 (cgroup_bpf_release) CPU 1 (close link FD)
--------------------------- ---------------------
st_link->cgroup = NULL

                                 bpf_link_put_direct()
                                   bpf_link_free()
                                     bpf_struct_ops_map_link_dealloc()
                                       if (st_link->cgroup) // false
                                       kfree(st_link) // freed!

bpf_link_inc_not_zero() // use-after-free

Would it be safer to call bpf_link_inc_not_zero() before modifying st_link fields? That way, if the link is already at refcount zero, the loop would skip it; otherwise, the held reference would prevent the concurrent free.

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21382620878

AI-authorship-score: low
AI-authorship-explanation: The commit is authored by a known kernel maintainer with a consistent, human-like technical writing style and appropriate kernel development patterns.
issues-found: 1
issue-severity-score: medium
issue-severity-explanation: Use-after-free race condition in cgroup_bpf_release() that can cause kernel crash when link FD is closed concurrently with cgroup deletion.

#6 lore raw

Re: [PATCH bpf-next v3 02/17] bpf: allow attaching struct_ops to cgroups

In reply to Roman Gushchin
Quoted 5 lines · Roman Gushchin

Introduce an ability to attach bpf struct_ops'es to cgroups.

From user's standpoint it works in the following way:
a user passes a BPF_F_CGROUP_FD flag and specifies the target cgroup

Since both fdinfo and link_info show the cgroup ID, why not use BPF_F_CGROUP_ID for better alignment?

Quoted 78 lines

fd while creating a struct_ops link. As the result, the bpf struct_ops link will be created and attached to a cgroup.

The cgroup.bpf structure maintains a list of attached struct ops links. If the cgroup is getting deleted, attached struct ops'es are getting auto-detached and the userspace program gets a notification.

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

---
 include/linux/bpf-cgroup-defs.h |  3 ++
 include/linux/bpf-cgroup.h      | 16 +++++++++
 include/linux/bpf.h             |  3 ++
 include/uapi/linux/bpf.h        |  3 ++
 kernel/bpf/bpf_struct_ops.c     | 59 ++++++++++++++++++++++++++++++---
 kernel/bpf/cgroup.c             | 46 +++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h  |  1 +
 7 files changed, 127 insertions(+), 4 deletions(-)
include/linux/bpf-cgroup-defs.h +3 -0
index c9e6b26abab6..6c5e37190dad 100644
@@ -71,6 +71,9 @@ struct cgroup_bpf {
        /* temp storage for effective prog array used by prog_attach/detach */
        struct bpf_prog_array *inactive;
 
+       /* list of bpf struct ops links */
+       struct list_head struct_ops_links;
+
        /* reference counter used to detach bpf programs after cgroup removal */
        struct percpu_ref refcnt;
 
include/linux/bpf-cgroup.h +16 -0
index 2f535331f926..a6c327257006 100644
@@ -423,6 +423,11 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
 int cgroup_bpf_prog_query(const union bpf_attr *attr,
                          union bpf_attr __user *uattr);
 
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+                                struct bpf_struct_ops_link *link);
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+                                 struct bpf_struct_ops_link *link);
+
 const struct bpf_func_proto *
 cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
 #else
@@ -451,6 +456,17 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
        return -EINVAL;
 }
 
+static inline int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+                                              struct bpf_struct_ops_link *link)
+{
+       return -EINVAL;
+}
+
+static inline void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+                                               struct bpf_struct_ops_link *link)
+{
+}
+
 static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
                                        union bpf_attr __user *uattr)
 {
include/linux/bpf.h +0 -0
index 899dd911dc82..391888eb257c 100644

include/linux/bpf.h struct bpf_raw_tp_link { · line 1899

Quoted 6 lines
include/linux/bpf.h +3 -0
@@ -1894,6 +1894,9 @@ struct bpf_raw_tp_link {
 struct bpf_struct_ops_link {
        struct bpf_link link;
        struct bpf_map __rcu *map;
+       struct cgroup *cgroup;
+       bool cgroup_removed;
+       struct list_head list;

We may need to support other structs in the future. Could we implement a more generic solution, such as:

int type; // cgroup, task, etc void *private; // ptr to type-specific data

Signature · 2 lines
Regards
Yafang
#7 lore raw

Re: [PATCH bpf-next v3 02/17] bpf: allow attaching struct_ops to cgroups

In reply to Roman Gushchin

Hi Roman,

Quoted 3 lines · Roman Gushchin

Introduce an ability to attach bpf struct_ops'es to cgroups.

[snip]

kernel/bpf/bpf_struct_ops.c static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link) · line 1228

Quoted 10 lines
 struct bpf_struct_ops_value {
        struct bpf_struct_ops_common_value common;
kernel/bpf/bpf_struct_ops.c +4 -0
@@ -1220,6 +1222,10 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
                st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
                bpf_map_put(&st_map->map);
        }
+
+       if (st_link->cgroup)
+               cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+

I was worried about concurrency with cgroup ops until I saw cgroup_bpf_detach_struct_ops() takes cgroup_lock() internally (since you take it inline sometimes below I falsely assumed it wasn't present). In any case, I'm wondering why you need to pass in the cgroup pointer to cgroup_bpf_detach_struct_ops() at all, rather than just the link?

Quoted 17 lines
kernel/bpf/bpf_struct_ops.c +4 -0
@@ -1357,8 +1386,12 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
        struct bpf_link_primer link_primer;
        struct bpf_struct_ops_map *st_map;
        struct bpf_map *map;
+       struct cgroup *cgrp;
        int err;
 
+       if (attr->link_create.flags & ~BPF_F_CGROUP_FD)
+               return -EINVAL;
+
        map = bpf_map_get(attr->link_create.map_fd);
        if (IS_ERR(map))
                return PTR_ERR(map);
@@ -1378,11 +1411,26 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
        bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_map_lops, NULL,
                      attr->link_create.attach_type);
 

kernel/bpf/bpf_struct_ops.c int bpf_struct_ops_link_create(union bpf_attr *attr) · line 1423

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10 -0
+       init_waitqueue_head(&link->wait_hup);
+
+       if (attr->link_create.flags & BPF_F_CGROUP_FD) {
+               cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
+               if (IS_ERR(cgrp)) {
+                       err = PTR_ERR(cgrp);
+                       goto err_out;
+               }
+               link->cgroup = cgrp;
+               err = cgroup_bpf_attach_struct_ops(cgrp, link);

We have to be careful at this point. cgroup release could now occur concurrently which would clear link->cgroup. Maybe worth a comment here since this is a bit subtle.

Quoted 6 lines
kernel/bpf/bpf_struct_ops.c +6 -0
+               if (err) {
+                       cgroup_put(cgrp);
+                       link->cgroup = NULL;
+                       goto err_out;
+               }
+       }
author maintainer
#8 lore raw

Re: [PATCH bpf-next v3 02/17] bpf: allow attaching struct_ops to cgroups

Quoted 7 lines · Josh Don

Hi Roman,

Quoted 3 lines · Roman Gushchin

Introduce an ability to attach bpf struct_ops'es to cgroups.

[snip]

Quoted 10 lines

kernel/bpf/bpf_struct_ops.c static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link) · line 1228

Quoted 10 lines
 struct bpf_struct_ops_value {
        struct bpf_struct_ops_common_value common;
kernel/bpf/bpf_struct_ops.c +4 -0
@@ -1220,6 +1222,10 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
                st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
                bpf_map_put(&st_map->map);
        }
+
+       if (st_link->cgroup)
+               cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+

Hi Josh!

Quoted 7 lines

I was worried about concurrency with cgroup ops until I saw cgroup_bpf_detach_struct_ops() takes cgroup_lock() internally (since you take it inline sometimes below I falsely assumed it wasn't present). In any case, I'm wondering why you need to pass in the cgroup pointer to cgroup_bpf_detach_struct_ops() at all, rather than just the link?

Sure, good point.

Quoted 17 lines
Quoted 17 lines
kernel/bpf/bpf_struct_ops.c +4 -0
@@ -1357,8 +1386,12 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
        struct bpf_link_primer link_primer;
        struct bpf_struct_ops_map *st_map;
        struct bpf_map *map;
+       struct cgroup *cgrp;
        int err;
 
+       if (attr->link_create.flags & ~BPF_F_CGROUP_FD)
+               return -EINVAL;
+
        map = bpf_map_get(attr->link_create.map_fd);
        if (IS_ERR(map))
                return PTR_ERR(map);
@@ -1378,11 +1411,26 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
        bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_map_lops, NULL,
                      attr->link_create.attach_type);
 
Quoted 14 lines

kernel/bpf/bpf_struct_ops.c int bpf_struct_ops_link_create(union bpf_attr *attr) · line 1423

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +10 -0
+       init_waitqueue_head(&link->wait_hup);
+
+       if (attr->link_create.flags & BPF_F_CGROUP_FD) {
+               cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
+               if (IS_ERR(cgrp)) {
+                       err = PTR_ERR(cgrp);
+                       goto err_out;
+               }
+               link->cgroup = cgrp;
+               err = cgroup_bpf_attach_struct_ops(cgrp, link);

We have to be careful at this point. cgroup release could now occur concurrently which would clear link->cgroup. Maybe worth a comment here since this is a bit subtle.

Agree, will add.

Thanks!

#9 lore raw

Re: [PATCH bpf-next v3 02/17] bpf: allow attaching struct_ops to cgroups

In reply to Roman Gushchin
Quoted 137 lines · Roman Gushchin

Introduce an ability to attach bpf struct_ops'es to cgroups.

From user's standpoint it works in the following way:
a user passes a BPF_F_CGROUP_FD flag and specifies the target cgroup fd while creating a struct_ops link. As the result, the bpf struct_ops link will be created and attached to a cgroup.

The cgroup.bpf structure maintains a list of attached struct ops links. If the cgroup is getting deleted, attached struct ops'es are getting auto-detached and the userspace program gets a notification.

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

---
 include/linux/bpf-cgroup-defs.h |  3 ++
 include/linux/bpf-cgroup.h      | 16 +++++++++
 include/linux/bpf.h             |  3 ++
 include/uapi/linux/bpf.h        |  3 ++
 kernel/bpf/bpf_struct_ops.c     | 59 ++++++++++++++++++++++++++++++---
 kernel/bpf/cgroup.c             | 46 +++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h  |  1 +
 7 files changed, 127 insertions(+), 4 deletions(-)
include/linux/bpf-cgroup-defs.h +3 -0
index c9e6b26abab6..6c5e37190dad 100644
@@ -71,6 +71,9 @@ struct cgroup_bpf {
 	/* temp storage for effective prog array used by prog_attach/detach */
 	struct bpf_prog_array *inactive;
 
+	/* list of bpf struct ops links */
+	struct list_head struct_ops_links;
+
 	/* reference counter used to detach bpf programs after cgroup removal */
 	struct percpu_ref refcnt;
 
include/linux/bpf-cgroup.h +16 -0
index 2f535331f926..a6c327257006 100644
@@ -423,6 +423,11 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
 int cgroup_bpf_prog_query(const union bpf_attr *attr,
 			  union bpf_attr __user *uattr);
 
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+				 struct bpf_struct_ops_link *link);
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+				  struct bpf_struct_ops_link *link);
+
 const struct bpf_func_proto *
 cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
 #else
@@ -451,6 +456,17 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
 	return -EINVAL;
 }
 
+static inline int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+					       struct bpf_struct_ops_link *link)
+{
+	return -EINVAL;
+}
+
+static inline void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+						struct bpf_struct_ops_link *link)
+{
+}
+
 static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
 					union bpf_attr __user *uattr)
 {
include/linux/bpf.h +3 -0
index 899dd911dc82..391888eb257c 100644
@@ -1894,6 +1894,9 @@ struct bpf_raw_tp_link {
 struct bpf_struct_ops_link {
 	struct bpf_link link;
 	struct bpf_map __rcu *map;
+	struct cgroup *cgroup;
+	bool cgroup_removed;
+	struct list_head list;
 	wait_queue_head_t wait_hup;
 };
 
include/uapi/linux/bpf.h +3 -0
index 44e7dbc278e3..28544e8af1cd 100644
@@ -1237,6 +1237,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_CGROUP_FD		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
@@ -6775,6 +6776,8 @@ struct bpf_link_info {
 		} xdp;
 		struct {
 			__u32 map_id;
+			__u32 :32;
+			__u64 cgroup_id;
 		} struct_ops;
 		struct {
 			__u32 pf;
kernel/bpf/bpf_struct_ops.c +6 -0
index de01cf3025b3..2e361e22cfa0 100644
@@ -13,6 +13,8 @@
 #include <linux/btf_ids.h>
 #include <linux/rcupdate_wait.h>
 #include <linux/poll.h>
+#include <linux/bpf-cgroup.h>
+#include <linux/cgroup.h>
 
 struct bpf_struct_ops_value {
 	struct bpf_struct_ops_common_value common;
@@ -1220,6 +1222,10 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
 		st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
 		bpf_map_put(&st_map->map);
 	}
+
+	if (st_link->cgroup)
+		cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+
 	kfree(st_link);
 }
 

kernel/bpf/bpf_struct_ops.c static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link, · line 1237

Quoted 4 lines
kernel/bpf/bpf_struct_ops.c +1 -0
@@ -1228,6 +1234,7 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
 {
 	struct bpf_struct_ops_link *st_link;
 	struct bpf_map *map;
+	u64 cgrp_id = 0;

Assigning 0 to cgrp_id would technically be incorrect, right? Like, cgroup_id() for !CONFIG_CGROUPS default to returning 1, and for CONFIG_CGROUPS the ID allocation is done via the idr_alloc_cyclic() API using a range between 1 and INT_MAX. Perhaps here it serves as a valid sentinel value? Is that the rationale?

In general, shouldn't all the cgroup related logic within this source file be protected by a CONFIG_CGROUPS ifdef? For example, both cgroup_get_from_fd() and cgroup_put() lack stubs when building with !CONFIG_CGROUPS.

Quoted 4 lines
kernel/bpf/bpf_struct_ops.c +0 -0
 	st_link = container_of(link, struct bpf_struct_ops_link, link);
 	rcu_read_lock();
@@ -1235,6 +1242,14 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
 	if (map)

kernel/bpf/bpf_struct_ops.c static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link, · line 1252

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +8 -0
 		seq_printf(seq, "map_id:\t%d\n", map->id);
 	rcu_read_unlock();
+
+	cgroup_lock();
+	if (st_link->cgroup)
+		cgrp_id = cgroup_id(st_link->cgroup);
+	cgroup_unlock();
+
+	if (cgrp_id)
+		seq_printf(seq, "cgroup_id:\t%llu\n", cgrp_id);

Probably could introduce a simple inline helper for the cgroup_lock()/cgroup_id()/cgroup_unlock() dance that's going on in here and bpf_struct_ops_map_link_fill_link_info() below.

Quoted 12 lines
kernel/bpf/bpf_struct_ops.c +1 -0
 }
 
 static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
@@ -1242,6 +1257,7 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
 {
 	struct bpf_struct_ops_link *st_link;
 	struct bpf_map *map;
+	u64 cgrp_id = 0;
 
 	st_link = container_of(link, struct bpf_struct_ops_link, link);
 	rcu_read_lock();

kernel/bpf/bpf_struct_ops.c static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link, · line 1274

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +7 -0
@@ -1249,6 +1265,13 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
 	if (map)
 		info->struct_ops.map_id = map->id;
 	rcu_read_unlock();
+
+	cgroup_lock();
+	if (st_link->cgroup)
+		cgrp_id = cgroup_id(st_link->cgroup);
+	cgroup_unlock();
+
+	info->struct_ops.cgroup_id = cgrp_id;

As mentioned above a simple inline helper could simply yield the following here:

...

	  info->struct_ops.cgroup_id = bpf_struct_ops_lin_cgroup_id();

...

Quoted 24 lines
kernel/bpf/bpf_struct_ops.c +6 -0
 	return 0;
 }
@@ -1327,6 +1350,9 @@ static int bpf_struct_ops_map_link_detach(struct bpf_link *link)
 
 	mutex_unlock(&update_mutex);
 
+	if (st_link->cgroup)
+		cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+
 	wake_up_interruptible_poll(&st_link->wait_hup, EPOLLHUP);
 
 	return 0;
@@ -1339,6 +1365,9 @@ static __poll_t bpf_struct_ops_map_link_poll(struct file *file,
 
 	poll_wait(file, &st_link->wait_hup, pts);
 
+	if (st_link->cgroup_removed)
+		return EPOLLHUP;
+
 	return rcu_access_pointer(st_link->map) ? 0 : EPOLLHUP;
 }
 

kernel/bpf/bpf_struct_ops.c int bpf_struct_ops_link_create(union bpf_attr *attr) · line 1394

Quoted 9 lines
kernel/bpf/bpf_struct_ops.c +4 -0
@@ -1357,8 +1386,12 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 	struct bpf_link_primer link_primer;
 	struct bpf_struct_ops_map *st_map;
 	struct bpf_map *map;
+	struct cgroup *cgrp;
 	int err;
 
+	if (attr->link_create.flags & ~BPF_F_CGROUP_FD)
+		return -EINVAL;
+

BPF_F_CGROUP_FD is dependent on the cgroup subsystem, therefore it probably makes some sense to only accept BPF_F_CGROUP_FD when CONFIG_BPF_CGROUP is enabled, otherwise -EOPNOTSUPP?

I'd also probably rewrite this such that we do:

...

	struct cgroup *cgrp = NULL;
	...
	if (attr->link_create.flags & ~BPF_F_CGROUP_FD) {
#if IS_ENABLED(CONFIG_CGROUP_BPF)
	cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
	if (IS_ERR(cgrp))
		return PTR_ERR(cgrp);
#else
	return -EOPNOTSUPP;
#endif
	}
...
	if (cgrp) {
		link->cgroup = cgrp;
		if (cgroup_bpf_attach_struct_ops(cgrp, link)) {
		   cgroup_put(cgrp);
		   goto err_out;
		}
	}

IMO the code is cleaner and reads better too.

Quoted 110 lines
kernel/bpf/bpf_struct_ops.c +22 -4
 	map = bpf_map_get(attr->link_create.map_fd);
 	if (IS_ERR(map))
 		return PTR_ERR(map);
@@ -1378,11 +1411,26 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 	bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_map_lops, NULL,
 		      attr->link_create.attach_type);
 
+	init_waitqueue_head(&link->wait_hup);
+
+	if (attr->link_create.flags & BPF_F_CGROUP_FD) {
+		cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
+		if (IS_ERR(cgrp)) {
+			err = PTR_ERR(cgrp);
+			goto err_out;
+		}
+		link->cgroup = cgrp;
+		err = cgroup_bpf_attach_struct_ops(cgrp, link);
+		if (err) {
+			cgroup_put(cgrp);
+			link->cgroup = NULL;
+			goto err_out;
+		}
+	}
+
 	err = bpf_link_prime(&link->link, &link_primer);
 	if (err)
-		goto err_out;
-
-	init_waitqueue_head(&link->wait_hup);
+		goto err_put_cgroup;
 
 	/* Hold the update_mutex such that the subsystem cannot
 	 * do link->ops->detach() before the link is fully initialized.
@@ -1393,13 +1441,16 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 		mutex_unlock(&update_mutex);
 		bpf_link_cleanup(&link_primer);
 		link = NULL;
-		goto err_out;
+		goto err_put_cgroup;
 	}
 	RCU_INIT_POINTER(link->map, map);
 	mutex_unlock(&update_mutex);
 
 	return bpf_link_settle(&link_primer);
 
+err_put_cgroup:
+	if (link && link->cgroup)
+		cgroup_bpf_detach_struct_ops(link->cgroup, link);
 err_out:
 	bpf_map_put(map);
 	kfree(link);
kernel/bpf/cgroup.c +18 -0
index 69988af44b37..7b1903be6f69 100644
@@ -16,6 +16,7 @@
 #include <linux/bpf-cgroup.h>
 #include <linux/bpf_lsm.h>
 #include <linux/bpf_verifier.h>
+#include <linux/poll.h>
 #include <net/sock.h>
 #include <net/bpf_sk_storage.h>
 
@@ -307,12 +308,23 @@ static void cgroup_bpf_release(struct work_struct *work)
 					       bpf.release_work);
 	struct bpf_prog_array *old_array;
 	struct list_head *storages = &cgrp->bpf.storages;
+	struct bpf_struct_ops_link *st_link, *st_tmp;
 	struct bpf_cgroup_storage *storage, *stmp;
+	LIST_HEAD(st_links);
 
 	unsigned int atype;
 
 	cgroup_lock();
 
+	list_splice_init(&cgrp->bpf.struct_ops_links, &st_links);
+	list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+		st_link->cgroup = NULL;
+		st_link->cgroup_removed = true;
+		cgroup_put(cgrp);
+		if (IS_ERR(bpf_link_inc_not_zero(&st_link->link)))
+			list_del(&st_link->list);
+	}
+
 	for (atype = 0; atype < ARRAY_SIZE(cgrp->bpf.progs); atype++) {
 		struct hlist_head *progs = &cgrp->bpf.progs[atype];
 		struct bpf_prog_list *pl;
@@ -346,6 +358,11 @@ static void cgroup_bpf_release(struct work_struct *work)
 
 	cgroup_unlock();
 
+	list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+		st_link->link.ops->detach(&st_link->link);
+		bpf_link_put(&st_link->link);
+	}
+
 	for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
 		cgroup_bpf_put(p);
 
@@ -525,6 +542,7 @@ static int cgroup_bpf_inherit(struct cgroup *cgrp)
 		INIT_HLIST_HEAD(&cgrp->bpf.progs[i]);
 
 	INIT_LIST_HEAD(&cgrp->bpf.storages);
+	INIT_LIST_HEAD(&cgrp->bpf.struct_ops_links);
 
 	for (i = 0; i < NR; i++)
 		if (compute_effective_progs(cgrp, i, &arrays[i]))
@@ -2759,3 +2777,31 @@ cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 		return NULL;
 	}

kernel/bpf/cgroup.c cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 2788

Quoted 10 lines
kernel/bpf/cgroup.c +9 -0
 }
+
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+				 struct bpf_struct_ops_link *link)
+{
+	int ret = 0;
+
+	cgroup_lock();
+	if (percpu_ref_is_zero(&cgrp->bpf.refcnt)) {
+		ret = -EBUSY;

If the cgroup is dying, then perhaps -EINVAL would be more appropriate here, no? I'd argue that -EBUSY implies a temporary or transient state.

Quoted 9 lines
kernel/bpf/cgroup.c +9 -0
+		goto out;
+	}
+	list_add_tail(&link->list, &cgrp->bpf.struct_ops_links);
+out:
+	cgroup_unlock();
+	return ret;
+}
+
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,

kernel/bpf/cgroup.c cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 2807

Quoted 10 lines
kernel/bpf/cgroup.c +10 -0
+				  struct bpf_struct_ops_link *link)
+{
+	cgroup_lock();
+	if (link->cgroup == cgrp) {
+		list_del(&link->list);
+		link->cgroup = NULL;
+		cgroup_put(cgrp);
+	}
+	cgroup_unlock();
+}

Within cgroup_bpf_attach_struct_ops() and cgroup_bpf_detach_struct_ops() the cgrp pointer appears to be superfluous? Both should probably only operate on link->cgroup instead? A !link->cgroup when calling either should be considered as -EINVAL.

Quoted 15 lines
tools/include/uapi/linux/bpf.h +1 -0
index 3ca7d76e05f0..d5492e60744a 100644
@@ -1237,6 +1237,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_CGROUP_FD		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
Signature · 1 line
2.52.0
author maintainer
#10 lore raw

Re: [PATCH bpf-next v3 02/17] bpf: allow attaching struct_ops to cgroups

Quoted 137 lines · Matt Bobrowski
Quoted 136 lines · Roman Gushchin

Introduce an ability to attach bpf struct_ops'es to cgroups.

From user's standpoint it works in the following way:
a user passes a BPF_F_CGROUP_FD flag and specifies the target cgroup fd while creating a struct_ops link. As the result, the bpf struct_ops link will be created and attached to a cgroup.

The cgroup.bpf structure maintains a list of attached struct ops links. If the cgroup is getting deleted, attached struct ops'es are getting auto-detached and the userspace program gets a notification.

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

---
 include/linux/bpf-cgroup-defs.h |  3 ++
 include/linux/bpf-cgroup.h      | 16 +++++++++
 include/linux/bpf.h             |  3 ++
 include/uapi/linux/bpf.h        |  3 ++
 kernel/bpf/bpf_struct_ops.c     | 59 ++++++++++++++++++++++++++++++---
 kernel/bpf/cgroup.c             | 46 +++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h  |  1 +
 7 files changed, 127 insertions(+), 4 deletions(-)
include/linux/bpf-cgroup-defs.h +3 -0
index c9e6b26abab6..6c5e37190dad 100644
@@ -71,6 +71,9 @@ struct cgroup_bpf {
 	/* temp storage for effective prog array used by prog_attach/detach */
 	struct bpf_prog_array *inactive;
 
+	/* list of bpf struct ops links */
+	struct list_head struct_ops_links;
+
 	/* reference counter used to detach bpf programs after cgroup removal */
 	struct percpu_ref refcnt;
 
include/linux/bpf-cgroup.h +16 -0
index 2f535331f926..a6c327257006 100644
@@ -423,6 +423,11 @@ int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
 int cgroup_bpf_prog_query(const union bpf_attr *attr,
 			  union bpf_attr __user *uattr);
 
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+				 struct bpf_struct_ops_link *link);
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+				  struct bpf_struct_ops_link *link);
+
 const struct bpf_func_proto *
 cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog);
 #else
@@ -451,6 +456,17 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
 	return -EINVAL;
 }
 
+static inline int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+					       struct bpf_struct_ops_link *link)
+{
+	return -EINVAL;
+}
+
+static inline void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+						struct bpf_struct_ops_link *link)
+{
+}
+
 static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
 					union bpf_attr __user *uattr)
 {
include/linux/bpf.h +3 -0
index 899dd911dc82..391888eb257c 100644
@@ -1894,6 +1894,9 @@ struct bpf_raw_tp_link {
 struct bpf_struct_ops_link {
 	struct bpf_link link;
 	struct bpf_map __rcu *map;
+	struct cgroup *cgroup;
+	bool cgroup_removed;
+	struct list_head list;
 	wait_queue_head_t wait_hup;
 };
 
include/uapi/linux/bpf.h +3 -0
index 44e7dbc278e3..28544e8af1cd 100644
@@ -1237,6 +1237,7 @@ enum bpf_perf_event_type {
 #define BPF_F_AFTER		(1U << 4)
 #define BPF_F_ID		(1U << 5)
 #define BPF_F_PREORDER		(1U << 6)
+#define BPF_F_CGROUP_FD		(1U << 7)
 #define BPF_F_LINK		BPF_F_LINK /* 1 << 13 */
 
 /* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
@@ -6775,6 +6776,8 @@ struct bpf_link_info {
 		} xdp;
 		struct {
 			__u32 map_id;
+			__u32 :32;
+			__u64 cgroup_id;
 		} struct_ops;
 		struct {
 			__u32 pf;
kernel/bpf/bpf_struct_ops.c +6 -0
index de01cf3025b3..2e361e22cfa0 100644
@@ -13,6 +13,8 @@
 #include <linux/btf_ids.h>
 #include <linux/rcupdate_wait.h>
 #include <linux/poll.h>
+#include <linux/bpf-cgroup.h>
+#include <linux/cgroup.h>
 
 struct bpf_struct_ops_value {
 	struct bpf_struct_ops_common_value common;
@@ -1220,6 +1222,10 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
 		st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
 		bpf_map_put(&st_map->map);
 	}
+
+	if (st_link->cgroup)
+		cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+
 	kfree(st_link);
 }
 
Quoted 11 lines

kernel/bpf/bpf_struct_ops.c static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link, · line 1237

Quoted 5 lines
kernel/bpf/bpf_struct_ops.c +1 -0
@@ -1228,6 +1234,7 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
 {
 	struct bpf_struct_ops_link *st_link;
 	struct bpf_map *map;
+	u64 cgrp_id = 0;

Assigning 0 to cgrp_id would technically be incorrect, right? Like, cgroup_id() for !CONFIG_CGROUPS default to returning 1, and for CONFIG_CGROUPS the ID allocation is done via the idr_alloc_cyclic() API using a range between 1 and INT_MAX. Perhaps here it serves as a valid sentinel value? Is that the rationale?

Yes. Idk, maybe (u64)-1 works better here, I don't have a strong opinion. Realistically I doubt there are too many bpf users with !CONFIG_CGROUPS. Alexei even suggested in the past to make CONFIG_MEMCG mandatory, which implies CONFIG_CGROUPS.

Quoted 15 lines

In general, shouldn't all the cgroup related logic within this source file be protected by a CONFIG_CGROUPS ifdef? For example, both cgroup_get_from_fd() and cgroup_put() lack stubs when building with !CONFIG_CGROUPS.

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +4 -0
 	st_link = container_of(link, struct bpf_struct_ops_link, link);
 	rcu_read_lock();
@@ -1235,6 +1242,14 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
 	if (map)
 		seq_printf(seq, "map_id:\t%d\n", map->id);
 	rcu_read_unlock();
+
+	cgroup_lock();
+	if (st_link->cgroup)
+		cgrp_id = cgroup_id(st_link->cgroup);
Quoted 8 lines

kernel/bpf/bpf_struct_ops.c static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link, · line 1252

Quoted 4 lines
kernel/bpf/bpf_struct_ops.c +4 -0
+	cgroup_unlock();
+
+	if (cgrp_id)
+		seq_printf(seq, "cgroup_id:\t%llu\n", cgrp_id);

Probably could introduce a simple inline helper for the cgroup_lock()/cgroup_id()/cgroup_unlock() dance that's going on in here and bpf_struct_ops_map_link_fill_link_info() below.

I'll try, thanks!

Quoted 210 lines
Quoted 13 lines
kernel/bpf/bpf_struct_ops.c +1 -0
 }
 
 static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
@@ -1242,6 +1257,7 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
 {
 	struct bpf_struct_ops_link *st_link;
 	struct bpf_map *map;
+	u64 cgrp_id = 0;
 
 	st_link = container_of(link, struct bpf_struct_ops_link, link);
 	rcu_read_lock();

kernel/bpf/bpf_struct_ops.c static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link, · line 1274

Quoted 10 lines
kernel/bpf/bpf_struct_ops.c +7 -0
@@ -1249,6 +1265,13 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
 	if (map)
 		info->struct_ops.map_id = map->id;
 	rcu_read_unlock();
+
+	cgroup_lock();
+	if (st_link->cgroup)
+		cgrp_id = cgroup_id(st_link->cgroup);
+	cgroup_unlock();
+
+	info->struct_ops.cgroup_id = cgrp_id;

As mentioned above a simple inline helper could simply yield the following here:

...

	  info->struct_ops.cgroup_id = bpf_struct_ops_lin_cgroup_id();

...

Quoted 24 lines
kernel/bpf/bpf_struct_ops.c +6 -0
 	return 0;
 }
@@ -1327,6 +1350,9 @@ static int bpf_struct_ops_map_link_detach(struct bpf_link *link)
 
 	mutex_unlock(&update_mutex);
 
+	if (st_link->cgroup)
+		cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
+
 	wake_up_interruptible_poll(&st_link->wait_hup, EPOLLHUP);
 
 	return 0;
@@ -1339,6 +1365,9 @@ static __poll_t bpf_struct_ops_map_link_poll(struct file *file,
 
 	poll_wait(file, &st_link->wait_hup, pts);
 
+	if (st_link->cgroup_removed)
+		return EPOLLHUP;
+
 	return rcu_access_pointer(st_link->map) ? 0 : EPOLLHUP;
 }
 

kernel/bpf/bpf_struct_ops.c int bpf_struct_ops_link_create(union bpf_attr *attr) · line 1394

Quoted 9 lines
kernel/bpf/bpf_struct_ops.c +4 -0
@@ -1357,8 +1386,12 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 	struct bpf_link_primer link_primer;
 	struct bpf_struct_ops_map *st_map;
 	struct bpf_map *map;
+	struct cgroup *cgrp;
 	int err;
 
+	if (attr->link_create.flags & ~BPF_F_CGROUP_FD)
+		return -EINVAL;
+

BPF_F_CGROUP_FD is dependent on the cgroup subsystem, therefore it probably makes some sense to only accept BPF_F_CGROUP_FD when CONFIG_BPF_CGROUP is enabled, otherwise -EOPNOTSUPP?

I'd also probably rewrite this such that we do:

...

	struct cgroup *cgrp = NULL;
	...
	if (attr->link_create.flags & ~BPF_F_CGROUP_FD) {
#if IS_ENABLED(CONFIG_CGROUP_BPF)
	cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
	if (IS_ERR(cgrp))
		return PTR_ERR(cgrp);
#else
	return -EOPNOTSUPP;
#endif
	}
...
	if (cgrp) {
		link->cgroup = cgrp;
		if (cgroup_bpf_attach_struct_ops(cgrp, link)) {
		   cgroup_put(cgrp);
		   goto err_out;
		}
	}

IMO the code is cleaner and reads better too.

Quoted 116 lines
kernel/bpf/bpf_struct_ops.c +22 -4
 	map = bpf_map_get(attr->link_create.map_fd);
 	if (IS_ERR(map))
 		return PTR_ERR(map);
@@ -1378,11 +1411,26 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 	bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_map_lops, NULL,
 		      attr->link_create.attach_type);
 
+	init_waitqueue_head(&link->wait_hup);
+
+	if (attr->link_create.flags & BPF_F_CGROUP_FD) {
+		cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
+		if (IS_ERR(cgrp)) {
+			err = PTR_ERR(cgrp);
+			goto err_out;
+		}
+		link->cgroup = cgrp;
+		err = cgroup_bpf_attach_struct_ops(cgrp, link);
+		if (err) {
+			cgroup_put(cgrp);
+			link->cgroup = NULL;
+			goto err_out;
+		}
+	}
+
 	err = bpf_link_prime(&link->link, &link_primer);
 	if (err)
-		goto err_out;
-
-	init_waitqueue_head(&link->wait_hup);
+		goto err_put_cgroup;
 
 	/* Hold the update_mutex such that the subsystem cannot
 	 * do link->ops->detach() before the link is fully initialized.
@@ -1393,13 +1441,16 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
 		mutex_unlock(&update_mutex);
 		bpf_link_cleanup(&link_primer);
 		link = NULL;
-		goto err_out;
+		goto err_put_cgroup;
 	}
 	RCU_INIT_POINTER(link->map, map);
 	mutex_unlock(&update_mutex);
 
 	return bpf_link_settle(&link_primer);
 
+err_put_cgroup:
+	if (link && link->cgroup)
+		cgroup_bpf_detach_struct_ops(link->cgroup, link);
 err_out:
 	bpf_map_put(map);
 	kfree(link);
kernel/bpf/cgroup.c +23 -0
index 69988af44b37..7b1903be6f69 100644
@@ -16,6 +16,7 @@
 #include <linux/bpf-cgroup.h>
 #include <linux/bpf_lsm.h>
 #include <linux/bpf_verifier.h>
+#include <linux/poll.h>
 #include <net/sock.h>
 #include <net/bpf_sk_storage.h>
 
@@ -307,12 +308,23 @@ static void cgroup_bpf_release(struct work_struct *work)
 					       bpf.release_work);
 	struct bpf_prog_array *old_array;
 	struct list_head *storages = &cgrp->bpf.storages;
+	struct bpf_struct_ops_link *st_link, *st_tmp;
 	struct bpf_cgroup_storage *storage, *stmp;
+	LIST_HEAD(st_links);
 
 	unsigned int atype;
 
 	cgroup_lock();
 
+	list_splice_init(&cgrp->bpf.struct_ops_links, &st_links);
+	list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+		st_link->cgroup = NULL;
+		st_link->cgroup_removed = true;
+		cgroup_put(cgrp);
+		if (IS_ERR(bpf_link_inc_not_zero(&st_link->link)))
+			list_del(&st_link->list);
+	}
+
 	for (atype = 0; atype < ARRAY_SIZE(cgrp->bpf.progs); atype++) {
 		struct hlist_head *progs = &cgrp->bpf.progs[atype];
 		struct bpf_prog_list *pl;
@@ -346,6 +358,11 @@ static void cgroup_bpf_release(struct work_struct *work)
 
 	cgroup_unlock();
 
+	list_for_each_entry_safe(st_link, st_tmp, &st_links, list) {
+		st_link->link.ops->detach(&st_link->link);
+		bpf_link_put(&st_link->link);
+	}
+
 	for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
 		cgroup_bpf_put(p);
 
@@ -525,6 +542,7 @@ static int cgroup_bpf_inherit(struct cgroup *cgrp)
 		INIT_HLIST_HEAD(&cgrp->bpf.progs[i]);
 
 	INIT_LIST_HEAD(&cgrp->bpf.storages);
+	INIT_LIST_HEAD(&cgrp->bpf.struct_ops_links);
 
 	for (i = 0; i < NR; i++)
 		if (compute_effective_progs(cgrp, i, &arrays[i]))
@@ -2759,3 +2777,31 @@ cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 		return NULL;
 	}
 }
+
+int cgroup_bpf_attach_struct_ops(struct cgroup *cgrp,
+				 struct bpf_struct_ops_link *link)
+{
+	int ret = 0;
Quoted 8 lines

kernel/bpf/cgroup.c cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 2788

Quoted 4 lines
kernel/bpf/cgroup.c +4 -0
+
+	cgroup_lock();
+	if (percpu_ref_is_zero(&cgrp->bpf.refcnt)) {
+		ret = -EBUSY;

If the cgroup is dying, then perhaps -EINVAL would be more appropriate here, no? I'd argue that -EBUSY implies a temporary or transient state.

Idk, I thought about it and settled on -EBUSY to highlight the transient nature of the issue. ENOENT is another option. I don't really think EINVAL is the best choice here.

Quoted 18 lines
Quoted 18 lines
kernel/bpf/cgroup.c +17 -0
+		goto out;
+	}
+	list_add_tail(&link->list, &cgrp->bpf.struct_ops_links);
+out:
+	cgroup_unlock();
+	return ret;
+}
+
+void cgroup_bpf_detach_struct_ops(struct cgroup *cgrp,
+				  struct bpf_struct_ops_link *link)
+{
+	cgroup_lock();
+	if (link->cgroup == cgrp) {
+		list_del(&link->list);
+		link->cgroup = NULL;
+		cgroup_put(cgrp);
+	}
Quoted 8 lines

kernel/bpf/cgroup.c cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · line 2807

Quoted 2 lines
kernel/bpf/cgroup.c +2 -0
+	cgroup_unlock();
+}

Within cgroup_bpf_attach_struct_ops() and cgroup_bpf_detach_struct_ops() the cgrp pointer appears to be superfluous? Both should probably only operate on link->cgroup instead? A !link->cgroup when calling either should be considered as -EINVAL.

Ack.

Thank you for the review!

author maintainer
#11 lore raw

PATCH v3 3/17 libbpf: fix return value on memory allocation failure

In reply to Roman Gushchin

bpf_map__attach_struct_ops() returns -EINVAL instead of -ENOMEM on the memory allocation failure. Fix it.

Fixes 590a00888250 ("bpf: libbpf: Add STRUCT_OPS support")
---
 tools/lib/bpf/libbpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -1
index 0c8bf0b5cce4..46d2762f5993 100644
@@ -13480,7 +13480,7 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
 
 	link = calloc(1, sizeof(*link));
 	if (!link)
-		return libbpf_err_ptr(-EINVAL);
+		return libbpf_err_ptr(-ENOMEM);
 
 	/* kern_vdata should be prepared during the loading phase. */
 	err = bpf_map_update_elem(map->fd, &zero, map->st_ops->kern_vdata, 0);
Signature · 1 line
2.52.0
Acked-by on v3 3/17
#12 lore raw

Re: [PATCH bpf-next v3 03/17] libbpf: fix return value on memory allocation failure

Quoted 6 lines · Roman Gushchin

bpf_map__attach_struct_ops() returns -EINVAL instead of -ENOMEM on the memory allocation failure. Fix it.

Fixes 590a00888250 ("bpf: libbpf: Add STRUCT_OPS support")
Quoted 21 lines
---
 tools/lib/bpf/libbpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -2
index 0c8bf0b5cce4..46d2762f5993 100644
@@ -13480,7 +13480,7 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
 
        link = calloc(1, sizeof(*link));
        if (!link)
-               return libbpf_err_ptr(-EINVAL);
+               return libbpf_err_ptr(-ENOMEM);
 
        /* kern_vdata should be prepared during the loading phase. */
        err = bpf_map_update_elem(map->fd, &zero, map->st_ops->kern_vdata, 0);
--

2.52.0

Signature · 2 lines
Regards
Yafang
author maintainer
#13 lore raw

PATCH v3 4/17 libbpf: introduce bpf_map__attach_struct_ops_opts()

In reply to Roman Gushchin

Introduce bpf_map__attach_struct_ops_opts(), an extended version of bpf_map__attach_struct_ops(), which takes additional struct bpf_struct_ops_opts argument.

This allows to pass a target_fd argument and the BPF_F_CGROUP_FD flag and attach the struct ops to a cgroup as a result.

---
 tools/lib/bpf/libbpf.c   | 20 +++++++++++++++++---
 tools/lib/bpf/libbpf.h   | 14 ++++++++++++++
 tools/lib/bpf/libbpf.map |  1 +
 3 files changed, 32 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +17 -3
index 46d2762f5993..9ba67089bf9d 100644
@@ -13462,11 +13462,18 @@ static int bpf_link__detach_struct_ops(struct bpf_link *link)
 	return close(link->fd);
 }
 
-struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
+struct bpf_link *bpf_map__attach_struct_ops_opts(const struct bpf_map *map,
+						 const struct bpf_struct_ops_opts *opts)
 {
+	DECLARE_LIBBPF_OPTS(bpf_link_create_opts, link_opts);
 	struct bpf_link_struct_ops *link;
+	int err, fd, target_fd;
 	__u32 zero = 0;
-	int err, fd;
+
+	if (!OPTS_VALID(opts, bpf_struct_ops_opts)) {
+		pr_warn("map '%s': invalid opts\n", map->name);
+		return libbpf_err_ptr(-EINVAL);
+	}
 
 	if (!bpf_map__is_struct_ops(map)) {
 		pr_warn("map '%s': can't attach non-struct_ops map\n", map->name);
@@ -13503,7 +13510,9 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
 		return &link->link;
 	}
 
-	fd = bpf_link_create(map->fd, 0, BPF_STRUCT_OPS, NULL);
+	link_opts.flags = OPTS_GET(opts, flags, 0);
+	target_fd = OPTS_GET(opts, target_fd, 0);
+	fd = bpf_link_create(map->fd, target_fd, BPF_STRUCT_OPS, &link_opts);
 	if (fd < 0) {
 		free(link);
 		return libbpf_err_ptr(fd);
@@ -13515,6 +13524,11 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
 	return &link->link;
 }
 
+struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
+{
+	return bpf_map__attach_struct_ops_opts(map, NULL);
+}
+
 /*
  * Swap the back struct_ops of a link with a new struct_ops map.
  */
tools/lib/bpf/libbpf.h +14 -0
index dfc37a615578..2c28cf80e7fe 100644
@@ -920,6 +920,20 @@ bpf_program__attach_cgroup_opts(const struct bpf_program *prog, int cgroup_fd,
 struct bpf_map;
 
 LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map);
+
+struct bpf_struct_ops_opts {
+	/* size of this struct, for forward/backward compatibility */
+	size_t sz;
+	__u32 flags;
+	__u32 target_fd;
+	__u64 expected_revision;
+	size_t :0;
+};
+#define bpf_struct_ops_opts__last_field expected_revision
+
+LIBBPF_API struct bpf_link *
+bpf_map__attach_struct_ops_opts(const struct bpf_map *map,
+				const struct bpf_struct_ops_opts *opts);
 LIBBPF_API int bpf_link__update_map(struct bpf_link *link, const struct bpf_map *map);
 
 struct bpf_iter_attach_opts {
tools/lib/bpf/libbpf.map +1 -0
index d18fbcea7578..4779190c97b6 100644
@@ -454,4 +454,5 @@ LIBBPF_1.7.0 {
 		bpf_prog_assoc_struct_ops;
 		bpf_program__assoc_struct_ops;
 		btf__permute;
+		bpf_map__attach_struct_ops_opts;
 } LIBBPF_1.6.0;
Signature · 1 line
2.52.0
#14 lore raw

Re: [PATCH bpf-next v3 04/17] libbpf: introduce bpf_map__attach_struct_ops_opts()

The code reads flags and target_fd from opts, but the expected_revision field declared in struct bpf_struct_ops_opts is never read. Looking at similar opts structures (bpf_cgroup_opts, bpf_tcx_opts…
Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 46d2762f5993..9ba67089bf9d 100644

[ ... ]

tools/lib/bpf/libbpf.c struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map) · line 13515

Quoted 8 lines
tools/lib/bpf/libbpf.c +3 -1
@@ -13503,7 +13510,9 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map)
 		return &link->link;
 	}
 
-	fd = bpf_link_create(map->fd, 0, BPF_STRUCT_OPS, NULL);
+	link_opts.flags = OPTS_GET(opts, flags, 0);
+	target_fd = OPTS_GET(opts, target_fd, 0);
+	fd = bpf_link_create(map->fd, target_fd, BPF_STRUCT_OPS, &link_opts);

The code reads flags and target_fd from opts, but the expected_revision field declared in struct bpf_struct_ops_opts is never read. Looking at similar opts structures (bpf_cgroup_opts, bpf_tcx_opts, bpf_netkit_opts), they all pass expected_revision to bpf_link_create_opts. For example:

    link_create_opts.cgroup.expected_revision = OPTS_GET(opts, expected_revision, 0);

However, bpf_link_create_opts has no struct_ops sub-struct to receive expected_revision, and the kernel's bpf_attr link_create union has no struct_ops.expected_revision field either.

Is expected_revision in bpf_struct_ops_opts intentionally unused? If so, should the field be removed from the struct to avoid user confusion? If users set expected_revision expecting it to be honored, the value is silently ignored.

[ ... ]

Quoted 5 lines
tools/lib/bpf/libbpf.h +0 -0
index dfc37a615578..2c28cf80e7fe 100644

tools/lib/bpf/libbpf.h bpf_program__attach_cgroup_opts(const struct bpf_program *prog, int cgroup_fd, · line 929

Quoted 10 lines
tools/lib/bpf/libbpf.h +7 -0
@@ -920,6 +920,20 @@ bpf_program__attach_cgroup_opts(const struct bpf_program *prog, int cgroup_fd,
 struct bpf_map;
 
 LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map);
+
+struct bpf_struct_ops_opts {
+	/* size of this struct, for forward/backward compatibility */
+	size_t sz;
+	__u32 flags;
+	__u32 target_fd;
+	__u64 expected_revision;
                 ^^^^^^^^^^^^^^^^^

The expected_revision field is declared here, matching the pattern used in bpf_cgroup_opts, bpf_tcx_opts, and bpf_netkit_opts. However, unlike those structures where expected_revision is read and passed through, this field has no corresponding code path in bpf_map__attach_struct_ops_opts().

tools/lib/bpf/libbpf.h bpf_program__attach_cgroup_opts(const struct bpf_program *prog, int cgroup_fd, · line 932

Quoted 3 lines
tools/lib/bpf/libbpf.h +3 -0
+	size_t :0;
+};
+#define bpf_struct_ops_opts__last_field expected_revision

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21382620878

AI-authorship-score: low
AI-authorship-explanation: The commit follows kernel conventions and existing code patterns, with an oversight (unused expected_revision field) more typical of human development than AI-generated code.
issues-found: 1
issue-severity-score: low
issue-severity-explanation: The expected_revision field in bpf_struct_ops_opts is declared but never used, causing silent data loss if users set this field expecting it to be honored.

author maintainer
#15 lore raw

PATCH v3 5/17 bpf: mark struct oom_control's memcg field as TRUSTED_OR_NULL

In reply to Roman Gushchin

Struct oom_control is used to describe the OOM context. It's memcg field defines the scope of OOM: it's NULL for global OOMs and a valid memcg pointer for memcg-scoped OOMs. Teach bpf verifier to recognize it as trusted or NULL pointer. It will provide the bpf OOM handler a trusted memcg pointer, which for example is required for iterating the memcg's subtree.

---
 kernel/bpf/verifier.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
index c2f2650db9fd..cca36edb460d 100644
@@ -7242,6 +7242,10 @@ BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct) {
 	struct file *vm_file;
 };
 
+BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct oom_control) {
+	struct mem_cgroup *memcg;
+};
+
 static bool type_is_rcu(struct bpf_verifier_env *env,
 			struct bpf_reg_state *reg,
 			const char *field_name, u32 btf_id)
@@ -7284,6 +7288,7 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env,
 	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct socket));
 	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct dentry));
 	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct));
+	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct oom_control));
 
 	return btf_nested_type_is_trusted(&env->log, reg, field_name, btf_id,
 					  "__safe_trusted_or_null");
Signature · 1 line
2.52.0
Acked-by on v3 5/17
#16 lore raw

Re: [PATCH bpf-next v3 05/17] bpf: mark struct oom_control's memcg field as TRUSTED_OR_NULL

Quoted 10 lines · Roman Gushchin

Struct oom_control is used to describe the OOM context. It's memcg field defines the scope of OOM: it's NULL for global OOMs and a valid memcg pointer for memcg-scoped OOMs. Teach bpf verifier to recognize it as trusted or NULL pointer. It will provide the bpf OOM handler a trusted memcg pointer, which for example is required for iterating the memcg's subtree.

Quoted 31 lines
---
 kernel/bpf/verifier.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -1
index c2f2650db9fd..cca36edb460d 100644
@@ -7242,6 +7242,10 @@ BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct) {
        struct file *vm_file;
 };
 
+BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct oom_control) {
+       struct mem_cgroup *memcg;
+};
+
 static bool type_is_rcu(struct bpf_verifier_env *env,
                        struct bpf_reg_state *reg,
                        const char *field_name, u32 btf_id)
@@ -7284,6 +7288,7 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env,
        BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct socket));
        BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct dentry));
        BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct));
+       BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct oom_control));
 
        return btf_nested_type_is_trusted(&env->log, reg, field_name, btf_id,
                                          "__safe_trusted_or_null");
--

2.52.0

Signature · 2 lines
Regards
Yafang
Acked-by on v3 5/17
#17 lore raw

Re: [PATCH bpf-next v3 05/17] bpf: mark struct oom_control's memcg field as TRUSTED_OR_NULL

In reply to Roman Gushchin
Quoted 6 lines · Roman Gushchin

Struct oom_control is used to describe the OOM context. It's memcg field defines the scope of OOM: it's NULL for global OOMs and a valid memcg pointer for memcg-scoped OOMs. Teach bpf verifier to recognize it as trusted or NULL pointer. It will provide the bpf OOM handler a trusted memcg pointer, which for example is required for iterating the memcg's subtree.

This is fine. Feel free to add:

Quoted 32 lines
---
 kernel/bpf/verifier.c | 5 +++++
 1 file changed, 5 insertions(+)
kernel/bpf/verifier.c +5 -0
index c2f2650db9fd..cca36edb460d 100644
@@ -7242,6 +7242,10 @@ BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct) {
 	struct file *vm_file;
 };
 
+BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct oom_control) {
+	struct mem_cgroup *memcg;
+};
+
 static bool type_is_rcu(struct bpf_verifier_env *env,
 			struct bpf_reg_state *reg,
 			const char *field_name, u32 btf_id)
@@ -7284,6 +7288,7 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env,
 	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct socket));
 	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct dentry));
 	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct vm_area_struct));
+	BTF_TYPE_EMIT(BTF_TYPE_SAFE_TRUSTED_OR_NULL(struct oom_control));
 
 	return btf_nested_type_is_trusted(&env->log, reg, field_name, btf_id,
 					  "__safe_trusted_or_null");
Signature · 1 line
2.52.0
author maintainer
#18 lore raw

PATCH v3 6/17 mm: define mem_cgroup_get_from_ino() outside of CONFIG_SHRINKER_DEBUG

In reply to Roman Gushchin

mem_cgroup_get_from_ino() can be reused by the BPF OOM implementation, but currently depends on CONFIG_SHRINKER_DEBUG. Remove this dependency.

---
 include/linux/memcontrol.h | 4 ++--
 mm/memcontrol.c            | 2 --
 2 files changed, 2 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +2 -2
index 229ac9835adb..f3b8c71870d8 100644
@@ -833,9 +833,9 @@ static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
 	return memcg ? cgroup_ino(memcg->css.cgroup) : 0;
 }
+#endif
 
 struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino);
-#endif
 
 static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m)
 {
@@ -1298,12 +1298,12 @@ static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
 	return 0;
 }
+#endif
 
 static inline struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino)
 {
 	return NULL;
 }
-#endif
 
 static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m)
 {
mm/memcontrol.c +0 -2
index 3808845bc8cc..1f74fce27677 100644
@@ -3658,7 +3658,6 @@ struct mem_cgroup *mem_cgroup_from_id(unsigned short id)
 	return xa_load(&mem_cgroup_ids, id);
 }
 
-#ifdef CONFIG_SHRINKER_DEBUG
 struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino)
 {
 	struct cgroup *cgrp;
@@ -3679,7 +3678,6 @@ struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino)
 
 	return memcg;
 }
-#endif
 
 static void free_mem_cgroup_per_node_info(struct mem_cgroup_per_node *pn)
 {
Signature · 1 line
2.52.0
#19 lore raw

Re: [PATCH bpf-next v3 06/17] mm: define mem_cgroup_get_from_ino() outside of CONFIG_SHRINKER_DEBUG

Quoted 27 lines · Roman Gushchin

mem_cgroup_get_from_ino() can be reused by the BPF OOM implementation, but currently depends on CONFIG_SHRINKER_DEBUG. Remove this dependency.

---
 include/linux/memcontrol.h | 4 ++--
 mm/memcontrol.c            | 2 --
 2 files changed, 2 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +1 -1
index 229ac9835adb..f3b8c71870d8 100644
@@ -833,9 +833,9 @@ static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
        return memcg ? cgroup_ino(memcg->css.cgroup) : 0;
 }
+#endif
 
 struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino);
-#endif
 
 static inline struct mem_cgroup *mem_cgroup_from_seq(struct seq_file *m)
 {

include/linux/memcontrol.h static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg) · line 1301

Quoted 4 lines
include/linux/memcontrol.h +1 -0
@@ -1298,12 +1298,12 @@ static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
        return 0;
 }
+#endif

Given that mem_cgroup_ino() pairs with mem_cgroup_get_from_ino(), should we also define mem_cgroup_ino() outside CONFIG_SHRINKER_DEBUG?

Quoted 6 lines
include/linux/memcontrol.h +0 -1
 static inline struct mem_cgroup *mem_cgroup_get_from_ino(unsigned long ino)
 {
        return NULL;
 }
-#endif
Signature · 2 lines
Regards
Yafang
maintainer
#20 lore raw

Re: [PATCH bpf-next v3 06/17] mm: define mem_cgroup_get_from_ino() outside of CONFIG_SHRINKER_DEBUG

In reply to Roman Gushchin
Quoted 5 lines · Roman Gushchin

mem_cgroup_get_from_ino() can be reused by the BPF OOM implementation, but currently depends on CONFIG_SHRINKER_DEBUG. Remove this dependency.

This code has been changed in the mm-tree and you can directly use mem_cgroup_get_from_id() after changes in the mm-tree.

#21 lore raw

Re: [PATCH bpf-next v3 06/17] mm: define mem_cgroup_get_from_ino() outside of CONFIG_SHRINKER_DEBUG

In reply to Roman Gushchin
Quoted 6 lines

mem_cgroup_get_from_ino() can be reused by the BPF OOM implementation, but currently depends on CONFIG_SHRINKER_DEBUG. Remove this dependency.

After dropping this commit, I applied the remaining patches to the
current bpf-next and ran the selftests.
Everything passed, though the following warning was observed during
the build:
make -j 30 -C tools/testing/selftests/bpf
...
GEN /home/teawater/kernel/linux-ebpf/tools/testing/selftests/bpf/tools/build/libbpf/libbpf.pc
Warning: Kernel ABI header at 'tools/include/uapi/linux/bpf.h' differs from latest version at 'include/uapi/linux/bpf.h'
CC /home/teawater/kernel/linux-ebpf/tools/testing/selftests/bpf/tools/build/libbpf/sharedobjs/libbpf.o

Best,
Hui

Quoted 13 lines
---
 include/linux/memcontrol.h | 4 ++--
 mm/memcontrol.c | 2 --
 2 files changed, 2 insertions(+), 4 deletions(-)
include/linux/memcontrol.h +0 -0
index 229ac9835adb..f3b8c71870d8 100644
@@ -833,9 +833,9 @@ static inline unsigned long mem_cgroup_ino(struct mem_cgroup *memcg)
 {
 return memcg ? cgroup_ino(memcg->css.cgroup) : 0;
 }
author maintainer
#22 lore raw

PATCH v3 7/17 mm: introduce BPF OOM struct ops

In reply to Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

It's possible to load one bpf_oom_ops for the system and one bpf_oom_ops for every memory cgroup. In case of a memcg OOM, the cgroup tree is traversed from the OOM'ing memcg up to the root and corresponding BPF OOM handlers are executed until some memory is freed. If no memory is freed, the kernel OOM killer is invoked.

The struct ops provides the bpf_handle_out_of_memory() callback, which expected to return 1 if it was able to free some memory and 0 otherwise. If 1 is returned, the kernel also checks the bpf_memory_freed field of the oom_control structure, which is expected to be set by kfuncs suitable for releasing memory (which will be introduced later in the patch series). If both are set, OOM is considered handled, otherwise the next OOM handler in the chain is executed: e.g. BPF OOM attached to the parent cgroup or the kernel OOM killer.

The bpf_handle_out_of_memory() callback program is sleepable to allow using iterators, e.g. cgroup iterators. The callback receives struct oom_control as an argument, so it can determine the scope of the OOM
event: if this is a memcg-wide or system-wide OOM. It also receives bpf_struct_ops_link as the second argument, so it can detect the cgroup level at which this specific instance is attached.

The bpf_handle_out_of_memory() callback is executed just before the kernel victim task selection algorithm, so all heuristics and sysctls like panic on oom, sysctl_oom_kill_allocating_task and sysctl_oom_kill_allocating_task are respected.

The struct ops has the name field, which allows to define a custom name for the implemented policy. It's printed in the OOM report in the oom_handler=<name> format only if a bpf handler is invoked.

---
 MAINTAINERS                     |   2 +
 include/linux/bpf-cgroup-defs.h |   3 +
 include/linux/bpf.h             |   1 +
 include/linux/bpf_oom.h         |  46 ++++++++
 include/linux/oom.h             |   8 ++
 kernel/bpf/bpf_struct_ops.c     |  12 +-
 mm/Makefile                     |   2 +-
 mm/bpf_oom.c                    | 192 ++++++++++++++++++++++++++++++++
 mm/oom_kill.c                   |  19 ++++
 9 files changed, 282 insertions(+), 3 deletions(-)
 create mode 100644 include/linux/bpf_oom.h
 create mode 100644 mm/bpf_oom.c
MAINTAINERS +2 -0
index 491d567f7dc8..53465570c1e5 100644
@@ -4807,7 +4807,9 @@ M:	Shakeel Butt <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Maintained
+F:	include/linux/bpf_oom.h
 F:	mm/bpf_memcontrol.c
+F:	mm/bpf_oom.c
 
 BPF [MISC]
 L:	[email protected]
include/linux/bpf-cgroup-defs.h +3 -0
index 6c5e37190dad..52395834ce13 100644
@@ -74,6 +74,9 @@ struct cgroup_bpf {
 	/* list of bpf struct ops links */
 	struct list_head struct_ops_links;
 
+	/* BPF OOM struct ops link */
+	struct bpf_struct_ops_link __rcu *bpf_oom_link;
+
 	/* reference counter used to detach bpf programs after cgroup removal */
 	struct percpu_ref refcnt;
 
include/linux/bpf.h +1 -0
index 391888eb257c..a5cee5a657b0 100644
@@ -3944,6 +3944,7 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog)
 int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char **filep,
 			   const char **linep, int *nump);
 struct bpf_prog *bpf_prog_find_from_stack(void);
+void *bpf_struct_ops_data(struct bpf_map *map);
 
 int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog);
 int bpf_insn_array_ready(struct bpf_map *map);
include/linux/bpf_oom.h new file +46 -0

46 lines not on this page · raw message

include/linux/oom.h +8 -0
index 7b02bc1d0a7e..c2dce336bcb4 100644
@@ -51,6 +51,14 @@ struct oom_control {
 
 	/* Used to print the constraint info. */
 	enum oom_constraint constraint;
+
+#ifdef CONFIG_BPF_SYSCALL
+	/* Used by the bpf oom implementation to mark the forward progress */
+	bool bpf_memory_freed;
+
+	/* Handler name */
+	const char *bpf_handler_name;
+#endif
 };
 
 extern struct mutex oom_lock;
kernel/bpf/bpf_struct_ops.c +10 -2
index 2e361e22cfa0..6285a6d56b98 100644
@@ -1009,7 +1009,7 @@ static void bpf_struct_ops_map_free(struct bpf_map *map)
 	 * in the tramopline image to finish before releasing
 	 * the trampoline image.
 	 */
-	synchronize_rcu_mult(call_rcu, call_rcu_tasks);
+	synchronize_rcu_mult(call_rcu, call_rcu_tasks, call_rcu_tasks_trace);
 
 	__bpf_struct_ops_map_free(map);
 }
@@ -1226,7 +1226,8 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
 	if (st_link->cgroup)
 		cgroup_bpf_detach_struct_ops(st_link->cgroup, st_link);
 
-	kfree(st_link);
+	synchronize_rcu_tasks_trace();
+	kfree_rcu(st_link, link.rcu);
 }
 
 static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
@@ -1535,3 +1536,10 @@ void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struct bpf_map *map
 
 	info->btf_vmlinux_id = btf_obj_id(st_map->btf);
 }
+
+void *bpf_struct_ops_data(struct bpf_map *map)
+{
+	struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
+
+	return &st_map->kvalue.data;
+}
mm/Makefile +1 -1
index bf46fe31dc14..e939525ba01b 100644
@@ -107,7 +107,7 @@ ifdef CONFIG_SWAP
 obj-$(CONFIG_MEMCG) += swap_cgroup.o
 endif
 ifdef CONFIG_BPF_SYSCALL
-obj-$(CONFIG_MEMCG) += bpf_memcontrol.o
+obj-$(CONFIG_MEMCG) += bpf_memcontrol.o bpf_oom.o
 endif
 obj-$(CONFIG_CGROUP_HUGETLB) += hugetlb_cgroup.o
 obj-$(CONFIG_GUP_TEST) += gup_test.o
mm/bpf_oom.c new file +192 -0

192 lines not on this page · raw message

mm/oom_kill.c +19 -0
index 5eb11fbba704..44bbcf033804 100644
@@ -45,6 +45,7 @@
 #include <linux/mmu_notifier.h>
 #include <linux/cred.h>
 #include <linux/nmi.h>
+#include <linux/bpf_oom.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -246,6 +247,15 @@ static const char * const oom_constraint_text[] = {
 	[CONSTRAINT_MEMCG] = "CONSTRAINT_MEMCG",
 };
 
+static const char *oom_handler_name(struct oom_control *oc)
+{
+#ifdef CONFIG_BPF_SYSCALL
+	if (oc->bpf_handler_name)
+		return oc->bpf_handler_name;
+#endif
+	return NULL;
+}
+
 /*
  * Determine the type of allocation constraint.
  */
@@ -461,6 +471,8 @@ static void dump_header(struct oom_control *oc)
 	pr_warn("%s invoked oom-killer: gfp_mask=%#x(%pGg), order=%d, oom_score_adj=%hd\n",
 		current->comm, oc->gfp_mask, &oc->gfp_mask, oc->order,
 			current->signal->oom_score_adj);
+	if (oom_handler_name(oc))
+		pr_warn("oom bpf handler: %s\n", oom_handler_name(oc));
 	if (!IS_ENABLED(CONFIG_COMPACTION) && oc->order)
 		pr_warn("COMPACTION is disabled!!!\n");
 
@@ -1168,6 +1180,13 @@ bool out_of_memory(struct oom_control *oc)
 		return true;
 	}
 
+	/*
+	 * Let bpf handle the OOM first. If it was able to free up some memory,
+	 * bail out. Otherwise fall back to the kernel OOM killer.
+	 */
+	if (bpf_handle_oom(oc))
+		return true;
+
 	select_bad_process(oc);
 	/* Found nothing?!?! */
 	if (!oc->chosen) {
Signature · 1 line
2.52.0
maintainer
#23 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 9 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

It's possible to load one bpf_oom_ops for the system and one bpf_oom_ops for every memory cgroup. In case of a memcg OOM, the cgroup tree is traversed from the OOM'ing memcg up to the root and corresponding BPF OOM handlers are executed until some memory is freed. If no memory is freed, the kernel OOM killer is invoked.

Quoted 8 lines

The struct ops provides the bpf_handle_out_of_memory() callback, which expected to return 1 if it was able to free some memory and 0 otherwise. If 1 is returned, the kernel also checks the bpf_memory_freed field of the oom_control structure, which is expected to be set by kfuncs suitable for releasing memory (which will be introduced later in the patch series). If both are set, OOM is considered handled, otherwise the next OOM handler in the chain is executed: e.g. BPF OOM attached to the parent cgroup or the kernel OOM killer.

I still find this dual reporting a bit confusing. I can see your intention in having a pre-defined "releasers" of the memory to trust BPF handlers more but they do have access to oc->bpf_memory_freed so they can manipulate it. Therefore an additional level of protection is rather weak.

It is also not really clear to me how this works while there is OOM victim on the way out. (i.e. tsk_is_oom_victim() -> abort case). This will result in no killing therefore no bpf_memory_freed, right? Handler itself should consider its work done. How exactly is this handled.

Also is there any way to handle the oom by increasing the memcg limit? I do not see a callback for that.

Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#24 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 19 lines · Michal Hocko
Quoted 9 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

It's possible to load one bpf_oom_ops for the system and one bpf_oom_ops for every memory cgroup. In case of a memcg OOM, the cgroup tree is traversed from the OOM'ing memcg up to the root and corresponding BPF OOM handlers are executed until some memory is freed. If no memory is freed, the kernel OOM killer is invoked.

Quoted 8 lines

The struct ops provides the bpf_handle_out_of_memory() callback, which expected to return 1 if it was able to free some memory and 0 otherwise. If 1 is returned, the kernel also checks the bpf_memory_freed field of the oom_control structure, which is expected to be set by kfuncs suitable for releasing memory (which will be introduced later in the patch series). If both are set, OOM is considered handled, otherwise the next OOM handler in the chain is executed: e.g. BPF OOM attached to the parent cgroup or the kernel OOM killer.

Quoted 5 lines

I still find this dual reporting a bit confusing. I can see your intention in having a pre-defined "releasers" of the memory to trust BPF handlers more but they do have access to oc->bpf_memory_freed so they can manipulate it. Therefore an additional level of protection is rather weak.

No, they can't. They have only a read-only access.

Quoted 4 lines

It is also not really clear to me how this works while there is OOM victim on the way out. (i.e. tsk_is_oom_victim() -> abort case). This will result in no killing therefore no bpf_memory_freed, right? Handler itself should consider its work done. How exactly is this handled.

It's a good question, I see your point...
Basically we want to give a handler an option to exit with "I promise, some memory will be freed soon" without doing anything destructive. But keeping it save at the same time.

I don't have a perfect answer out of my head, maybe some sort of a rate-limiter/counter might work? E.g. a handler can promise this N times before the kernel kicks in? Any ideas?

Quoted 2 lines

Also is there any way to handle the oom by increasing the memcg limit? I do not see a callback for that.

There is no kfunc yet, but it's a good idea (which we accidentally discussed few days ago). I'll implement it.

Thank you!

maintainer
#25 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 21 lines · Roman Gushchin
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

It's possible to load one bpf_oom_ops for the system and one bpf_oom_ops for every memory cgroup. In case of a memcg OOM, the cgroup tree is traversed from the OOM'ing memcg up to the root and corresponding BPF OOM handlers are executed until some memory is freed. If no memory is freed, the kernel OOM killer is invoked.

The struct ops provides the bpf_handle_out_of_memory() callback, which expected to return 1 if it was able to free some memory and 0 otherwise. If 1 is returned, the kernel also checks the bpf_memory_freed field of the oom_control structure, which is expected to be set by kfuncs suitable for releasing memory (which will be introduced later in the patch series). If both are set, OOM is considered handled, otherwise the next OOM handler in the chain is executed: e.g. BPF OOM attached to the parent cgroup or the kernel OOM killer.

Quoted 7 lines
Quoted 5 lines

I still find this dual reporting a bit confusing. I can see your intention in having a pre-defined "releasers" of the memory to trust BPF handlers more but they do have access to oc->bpf_memory_freed so they can manipulate it. Therefore an additional level of protection is rather weak.

No, they can't. They have only a read-only access.

Could you explain this a bit more. This must be some BPF magic because they are getting a standard pointer to oom_control.

Quoted 9 lines
Quoted 4 lines

It is also not really clear to me how this works while there is OOM victim on the way out. (i.e. tsk_is_oom_victim() -> abort case). This will result in no killing therefore no bpf_memory_freed, right? Handler itself should consider its work done. How exactly is this handled.

It's a good question, I see your point...
Basically we want to give a handler an option to exit with "I promise, some memory will be freed soon" without doing anything destructive. But keeping it save at the same time.

Yes, something like OOM_BACKOFF, OOM_PROCESSED, OOM_FAILED.

Quoted 3 lines

I don't have a perfect answer out of my head, maybe some sort of a rate-limiter/counter might work? E.g. a handler can promise this N times before the kernel kicks in? Any ideas?

Counters usually do not work very well for async operations. In this case there is oom_repaer and/or task exit to finish the oom operation. The former is bound and guaranteed to make a forward progress but there is no time frame to assume when that happens as it depends on how many tasks might be queued (usually a single one but this is not something to rely on because of concurrent ooms in memcgs and also multiple tasks could be killed at the same time).

Another complication is that there are multiple levels of OOM to track (global, NUMA, memcg) so any watchdog would have to be aware of that as well. I am really wondering whether we really need to be so careful with handlers. It is not like you would allow any random oom handler to be loaded, right? Would it make sense to start without this protection and converge to something as we see how this evolves? Maybe this will raise the bar for oom handlers as the price for bugs is going to be really high.

Quoted 5 lines
Quoted 2 lines

Also is there any way to handle the oom by increasing the memcg limit? I do not see a callback for that.

There is no kfunc yet, but it's a good idea (which we accidentally discussed few days ago). I'll implement it.

Cool!

Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#26 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 28 lines · Michal Hocko
Quoted 27 lines · Roman Gushchin
Quoted 24 lines · Michal Hocko
Quoted 9 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

It's possible to load one bpf_oom_ops for the system and one bpf_oom_ops for every memory cgroup. In case of a memcg OOM, the cgroup tree is traversed from the OOM'ing memcg up to the root and corresponding BPF OOM handlers are executed until some memory is freed. If no memory is freed, the kernel OOM killer is invoked.

Quoted 8 lines

The struct ops provides the bpf_handle_out_of_memory() callback, which expected to return 1 if it was able to free some memory and 0 otherwise. If 1 is returned, the kernel also checks the bpf_memory_freed field of the oom_control structure, which is expected to be set by kfuncs suitable for releasing memory (which will be introduced later in the patch series). If both are set, OOM is considered handled, otherwise the next OOM handler in the chain is executed: e.g. BPF OOM attached to the parent cgroup or the kernel OOM killer.

I still find this dual reporting a bit confusing. I can see your intention in having a pre-defined "releasers" of the memory to trust BPF handlers more but they do have access to oc->bpf_memory_freed so they can manipulate it. Therefore an additional level of protection is rather weak.

Quoted 4 lines
Quoted 1 line

No, they can't. They have only a read-only access.

Could you explain this a bit more. This must be some BPF magic because they are getting a standard pointer to oom_control.

Yes, but bpf programs (unlike kernel modules) are going through the verifier when being loaded to the kernel. The verifier ensures that programs are safe: e.g. they can't access memory outside of safe areas, they can't can infinite loops, dereference a NULL pointer etc.

So even it looks like a normal argument, it's read only. And the program can't even read the memory outside of the structure itself, e.g. a program doing something like (oc + 1)->bpf_memory_freed won't be allowed to load.

Quoted 16 lines
Quoted 9 lines
Quoted 4 lines

It is also not really clear to me how this works while there is OOM victim on the way out. (i.e. tsk_is_oom_victim() -> abort case). This will result in no killing therefore no bpf_memory_freed, right? Handler itself should consider its work done. How exactly is this handled.

It's a good question, I see your point...
Basically we want to give a handler an option to exit with "I promise, some memory will be freed soon" without doing anything destructive. But keeping it save at the same time.

Yes, something like OOM_BACKOFF, OOM_PROCESSED, OOM_FAILED.

Quoted 3 lines

I don't have a perfect answer out of my head, maybe some sort of a rate-limiter/counter might work? E.g. a handler can promise this N times before the kernel kicks in? Any ideas?

Quoted 10 lines

Counters usually do not work very well for async operations. In this case there is oom_repaer and/or task exit to finish the oom operation. The former is bound and guaranteed to make a forward progress but there is no time frame to assume when that happens as it depends on how many tasks might be queued (usually a single one but this is not something to rely on because of concurrent ooms in memcgs and also multiple tasks could be killed at the same time).
Another complication is that there are multiple levels of OOM to track (global, NUMA, memcg) so any watchdog would have to be aware of that as well.

Yeah, it has to be an atomic counter attached to the bpf oom "instance":
a policy attached to a specific cgroup or system-wide.

Quoted 6 lines

I am really wondering whether we really need to be so careful with handlers. It is not like you would allow any random oom handler to be loaded, right? Would it make sense to start without this protection and converge to something as we see how this evolves? Maybe this will raise the bar for oom handlers as the price for bugs is going to be really high.

Right, bpf programs require CAP_SYSADMIN to be loaded.
I still would prefer to keep it 100% safe, but the more I think about it the more I agree with you: likely limitations of the protection mechanism will create more issues than the value of the protection itself.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Also is there any way to handle the oom by increasing the memcg limit? I do not see a callback for that.

There is no kfunc yet, but it's a good idea (which we accidentally discussed few days ago). I'll implement it.

Cool!

Thank you!

#27 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

In reply to Roman Gushchin
Quoted 43 lines · Roman Gushchin
Quoted 19 lines · Michal Hocko
Quoted 9 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

It's possible to load one bpf_oom_ops for the system and one bpf_oom_ops for every memory cgroup. In case of a memcg OOM, the cgroup tree is traversed from the OOM'ing memcg up to the root and corresponding BPF OOM handlers are executed until some memory is freed. If no memory is freed, the kernel OOM killer is invoked.

Quoted 8 lines

The struct ops provides the bpf_handle_out_of_memory() callback, which expected to return 1 if it was able to free some memory and 0 otherwise. If 1 is returned, the kernel also checks the bpf_memory_freed field of the oom_control structure, which is expected to be set by kfuncs suitable for releasing memory (which will be introduced later in the patch series). If both are set, OOM is considered handled, otherwise the next OOM handler in the chain is executed: e.g. BPF OOM attached to the parent cgroup or the kernel OOM killer.

Quoted 5 lines

I still find this dual reporting a bit confusing. I can see your intention in having a pre-defined "releasers" of the memory to trust BPF handlers more but they do have access to oc->bpf_memory_freed so they can manipulate it. Therefore an additional level of protection is rather weak.

No, they can't. They have only a read-only access.

Quoted 4 lines

It is also not really clear to me how this works while there is OOM victim on the way out. (i.e. tsk_is_oom_victim() -> abort case). This will result in no killing therefore no bpf_memory_freed, right? Handler itself should consider its work done. How exactly is this handled.

It's a good question, I see your point...
Basically we want to give a handler an option to exit with "I promise, some memory will be freed soon" without doing anything destructive. But keeping it save at the same time.

I don't have a perfect answer out of my head, maybe some sort of a rate-limiter/counter might work? E.g. a handler can promise this N times before the kernel kicks in? Any ideas?

Quoted 5 lines
Quoted 2 lines

Also is there any way to handle the oom by increasing the memcg limit? I do not see a callback for that.

There is no kfunc yet, but it's a good idea (which we accidentally discussed few days ago). I'll implement it.

Yes, please, this is something that I had mentioned to you the other day too. With this kind of BPF kfunc, we'll basically be able to handle memcg scoped OOM events inline without necessarily being forced to kill off anything.

#28 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

In reply to Roman Gushchin

Thanks Roman!

Quoted 33 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

mm/bpf_oom.c new file +29 -0
+bool bpf_handle_oom(struct oom_control *oc)
+{
+       struct bpf_struct_ops_link *st_link;
+       struct bpf_oom_ops *bpf_oom_ops;
+       struct mem_cgroup *memcg;
+       struct bpf_map *map;
+       int ret = 0;
+
+       /*
+        * System-wide OOMs are handled by the struct ops attached
+        * to the root memory cgroup
+        */
+       memcg = oc->memcg ? oc->memcg : root_mem_cgroup;
+
+       rcu_read_lock_trace();
+
+       /* Find the nearest bpf_oom_ops traversing the cgroup tree upwards */
+       for (; memcg; memcg = parent_mem_cgroup(memcg)) {
+               st_link = rcu_dereference_check(memcg->css.cgroup->bpf.bpf_oom_link,
+                                               rcu_read_lock_trace_held());
+               if (!st_link)
+                       continue;
+
+               map = rcu_dereference_check((st_link->map),
+                                           rcu_read_lock_trace_held());
+               if (!map)
+                       continue;
+
+               /* Call BPF OOM handler */

mm/bpf_oom.c line 70

Quoted 10 lines
mm/bpf_oom.c new file +10 -0
+               bpf_oom_ops = bpf_struct_ops_data(map);
+               ret = bpf_ops_handle_oom(bpf_oom_ops, st_link, oc);
+               if (ret && oc->bpf_memory_freed)
+                       break;
+               ret = 0;
+       }
+
+       rcu_read_unlock_trace();
+
+       return ret && oc->bpf_memory_freed;

If bpf claims to have freed memory but didn't actually do so, that seems like something potentially worth alerting to. Perhaps something to add to the oom header output?

author maintainer
#29 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 42 lines · Josh Don

Thanks Roman!

Quoted 39 lines · Roman Gushchin

Introduce a bpf struct ops for implementing custom OOM handling policies.

mm/bpf_oom.c new file +35 -0
+bool bpf_handle_oom(struct oom_control *oc)
+{
+       struct bpf_struct_ops_link *st_link;
+       struct bpf_oom_ops *bpf_oom_ops;
+       struct mem_cgroup *memcg;
+       struct bpf_map *map;
+       int ret = 0;
+
+       /*
+        * System-wide OOMs are handled by the struct ops attached
+        * to the root memory cgroup
+        */
+       memcg = oc->memcg ? oc->memcg : root_mem_cgroup;
+
+       rcu_read_lock_trace();
+
+       /* Find the nearest bpf_oom_ops traversing the cgroup tree upwards */
+       for (; memcg; memcg = parent_mem_cgroup(memcg)) {
+               st_link = rcu_dereference_check(memcg->css.cgroup->bpf.bpf_oom_link,
+                                               rcu_read_lock_trace_held());
+               if (!st_link)
+                       continue;
+
+               map = rcu_dereference_check((st_link->map),
+                                           rcu_read_lock_trace_held());
+               if (!map)
+                       continue;
+
+               /* Call BPF OOM handler */
+               bpf_oom_ops = bpf_struct_ops_data(map);
+               ret = bpf_ops_handle_oom(bpf_oom_ops, st_link, oc);
+               if (ret && oc->bpf_memory_freed)
+                       break;
+               ret = 0;
+       }
Quoted 8 lines

mm/bpf_oom.c line 70

Quoted 4 lines
mm/bpf_oom.c new file +4 -0
+
+       rcu_read_unlock_trace();
+
+       return ret && oc->bpf_memory_freed;

If bpf claims to have freed memory but didn't actually do so, that seems like something potentially worth alerting to. Perhaps something to add to the oom header output?

Michal pointed at a more fundamental problem: if a bpf handler performed some actions (e.g. killed a program), how to safely allow other bpf handlers to exit without performing redundant destructive operations? Now it works on marking victim processes, so that subsequent kernel oom handlers just bail out if they see a marked process.

I don't know to extend it to generic actions. E.g. we can have an atomic counter attached to the bpf oom instance (link), we can bump it on performing a destructive operation, but it's not clear when to clear it.

So maybe it's not worth it at all and it's better to drop this protection mechanism altogether.

Thanks!

maintainer
#30 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

In reply to Roman Gushchin

Once additional point I forgot to mention previously

Quoted 4 lines · Roman Gushchin
mm/oom_kill.c +0 -0
@@ -1168,6 +1180,13 @@ bool out_of_memory(struct oom_control *oc)
 		return true;
 	}
 

mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1192

Quoted 10 lines
mm/oom_kill.c +7 -0
+	/*
+	 * Let bpf handle the OOM first. If it was able to free up some memory,
+	 * bail out. Otherwise fall back to the kernel OOM killer.
+	 */
+	if (bpf_handle_oom(oc))
+		return true;
+
 	select_bad_process(oc);
 	/* Found nothing?!?! */
 	if (!oc->chosen) {

Should this check for is_sysrq_oom and always use the in kernel OOM handling for Sysrq triggered ooms as a failsafe measure?

Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#31 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 7 lines · Michal Hocko

Once additional point I forgot to mention previously

Quoted 4 lines · Roman Gushchin
mm/oom_kill.c +0 -0
@@ -1168,6 +1180,13 @@ bool out_of_memory(struct oom_control *oc)
 		return true;
 	}
 
Quoted 13 lines

mm/oom_kill.c bool out_of_memory(struct oom_control *oc) · line 1192

Quoted 10 lines
mm/oom_kill.c +7 -0
+	/*
+	 * Let bpf handle the OOM first. If it was able to free up some memory,
+	 * bail out. Otherwise fall back to the kernel OOM killer.
+	 */
+	if (bpf_handle_oom(oc))
+		return true;
+
 	select_bad_process(oc);
 	/* Found nothing?!?! */
 	if (!oc->chosen) {

Should this check for is_sysrq_oom and always use the in kernel OOM handling for Sysrq triggered ooms as a failsafe measure?

Yep, good point. Will implement in v4.

Thanks!

#32 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

In reply to Roman Gushchin
Quoted 41 lines · Roman Gushchin
mm/bpf_oom.c new file +41 -0
+bool bpf_handle_oom(struct oom_control *oc)
+{
+	struct bpf_struct_ops_link *st_link;
+	struct bpf_oom_ops *bpf_oom_ops;
+	struct mem_cgroup *memcg;
+	struct bpf_map *map;
+	int ret = 0;
+
+	/*
+	 * System-wide OOMs are handled by the struct ops attached
+	 * to the root memory cgroup
+	 */
+	memcg = oc->memcg ? oc->memcg : root_mem_cgroup;
+
+	rcu_read_lock_trace();
+
+	/* Find the nearest bpf_oom_ops traversing the cgroup tree upwards */
+	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
+		st_link = rcu_dereference_check(memcg->css.cgroup->bpf.bpf_oom_link,
+						rcu_read_lock_trace_held());
+		if (!st_link)
+			continue;
+
+		map = rcu_dereference_check((st_link->map),
+					    rcu_read_lock_trace_held());
+		if (!map)
+			continue;
+
+		/* Call BPF OOM handler */
+		bpf_oom_ops = bpf_struct_ops_data(map);
+		ret = bpf_ops_handle_oom(bpf_oom_ops, st_link, oc);
+		if (ret && oc->bpf_memory_freed)
+			break;
+		ret = 0;
+	}
+
+	rcu_read_unlock_trace();
+
+	return ret && oc->bpf_memory_freed;
+}
+

[ ... ]

Quoted 5 lines
mm/bpf_oom.c new file +5 -0
+static int bpf_oom_ops_reg(void *kdata, struct bpf_link *link)
+{
+	struct bpf_struct_ops_link *st_link = (struct bpf_struct_ops_link *)link;
+	struct cgroup *cgrp;
+

mm/bpf_oom.c line 116

Quoted 10 lines
mm/bpf_oom.c new file +10 -0
+	/* The link is not yet fully initialized, but cgroup should be set */
+	if (!link)
+		return -EOPNOTSUPP;
+
+	cgrp = st_link->cgroup;
+	if (!cgrp)
+		return -EINVAL;
+
+	if (cmpxchg(&cgrp->bpf.bpf_oom_link, NULL, st_link))
+		return -EEXIST;

iiuc, this will allow only one oom_ops to be attached to a cgroup. Considering oom_ops is the only user of the cgrp->bpf.struct_ops_links (added in patch 2), the list should have only one element for now.

Copy some context from the patch 2 commit log.

Quoted 6 lines

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

There are NONE, BPF_F_ALLOW_OVERRIDE, and BPF_F_ALLOW_MULTI, which one may be closer to the bpf_handle_oom() semantic. If it needs to change the ordering (or allow multi) in the future, does it need a new flag or the existing BPF_F_xxx flags can be used.

author maintainer
#33 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 65 lines · Martin KaFai Lau
Quoted 41 lines · Roman Gushchin
mm/bpf_oom.c new file +41 -0
+bool bpf_handle_oom(struct oom_control *oc)
+{
+	struct bpf_struct_ops_link *st_link;
+	struct bpf_oom_ops *bpf_oom_ops;
+	struct mem_cgroup *memcg;
+	struct bpf_map *map;
+	int ret = 0;
+
+	/*
+	 * System-wide OOMs are handled by the struct ops attached
+	 * to the root memory cgroup
+	 */
+	memcg = oc->memcg ? oc->memcg : root_mem_cgroup;
+
+	rcu_read_lock_trace();
+
+	/* Find the nearest bpf_oom_ops traversing the cgroup tree upwards */
+	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
+		st_link = rcu_dereference_check(memcg->css.cgroup->bpf.bpf_oom_link,
+						rcu_read_lock_trace_held());
+		if (!st_link)
+			continue;
+
+		map = rcu_dereference_check((st_link->map),
+					    rcu_read_lock_trace_held());
+		if (!map)
+			continue;
+
+		/* Call BPF OOM handler */
+		bpf_oom_ops = bpf_struct_ops_data(map);
+		ret = bpf_ops_handle_oom(bpf_oom_ops, st_link, oc);
+		if (ret && oc->bpf_memory_freed)
+			break;
+		ret = 0;
+	}
+
+	rcu_read_unlock_trace();
+
+	return ret && oc->bpf_memory_freed;
+}
+

[ ... ]

Quoted 5 lines
mm/bpf_oom.c new file +5 -0
+static int bpf_oom_ops_reg(void *kdata, struct bpf_link *link)
+{
+	struct bpf_struct_ops_link *st_link = (struct bpf_struct_ops_link *)link;
+	struct cgroup *cgrp;
+

mm/bpf_oom.c line 116

Quoted 10 lines
mm/bpf_oom.c new file +10 -0
+	/* The link is not yet fully initialized, but cgroup should be set */
+	if (!link)
+		return -EOPNOTSUPP;
+
+	cgrp = st_link->cgroup;
+	if (!cgrp)
+		return -EINVAL;
+
+	if (cmpxchg(&cgrp->bpf.bpf_oom_link, NULL, st_link))
+		return -EEXIST;

iiuc, this will allow only one oom_ops to be attached to a cgroup. Considering oom_ops is the only user of the cgrp->bpf.struct_ops_links (added in patch 2), the list should have only one element for now.

Quoted 1 line

Copy some context from the patch 2 commit log.

Hi Martin!

Sorry, I'm not quite sure what do you mean, can you please elaborate more?

We decided (in conversations at LPC) that 1 bpf oom policy for memcg is good for now (with a potential to extend in the future, if there will be use cases). But it seems like there is a lot of interest to attach struct ops'es to cgroups (there are already a couple of patchsets posted based on my earlier v2 patches), so I tried to make the bpf link mechanics suitable for multiple use cases from scratch.

Did I answer your question?

Quoted 5 lines
Quoted 5 lines

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Quoted 7 lines
Quoted 2 lines

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

There are NONE, BPF_F_ALLOW_OVERRIDE, and BPF_F_ALLOW_MULTI, which one may be closer to the bpf_handle_oom() semantic. If it needs to change the ordering (or allow multi) in the future, does it need a new flag or the existing BPF_F_xxx flags can be used.

I hope that existing flags can be used, but also I'm not sure we ever would need multiple oom handlers per cgroup. Do you have any specific concerns here?

Thanks!

#34 lore raw

Re: [PATCH bpf-next v3 07/17] mm: introduce BPF OOM struct ops

Quoted 74 lines · Roman Gushchin
Quoted 65 lines · Martin KaFai Lau
Quoted 41 lines · Roman Gushchin
mm/bpf_oom.c new file +41 -0
+bool bpf_handle_oom(struct oom_control *oc)
+{
+	struct bpf_struct_ops_link *st_link;
+	struct bpf_oom_ops *bpf_oom_ops;
+	struct mem_cgroup *memcg;
+	struct bpf_map *map;
+	int ret = 0;
+
+	/*
+	 * System-wide OOMs are handled by the struct ops attached
+	 * to the root memory cgroup
+	 */
+	memcg = oc->memcg ? oc->memcg : root_mem_cgroup;
+
+	rcu_read_lock_trace();
+
+	/* Find the nearest bpf_oom_ops traversing the cgroup tree upwards */
+	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
+		st_link = rcu_dereference_check(memcg->css.cgroup->bpf.bpf_oom_link,
+						rcu_read_lock_trace_held());
+		if (!st_link)
+			continue;
+
+		map = rcu_dereference_check((st_link->map),
+					    rcu_read_lock_trace_held());
+		if (!map)
+			continue;
+
+		/* Call BPF OOM handler */
+		bpf_oom_ops = bpf_struct_ops_data(map);
+		ret = bpf_ops_handle_oom(bpf_oom_ops, st_link, oc);
+		if (ret && oc->bpf_memory_freed)
+			break;
+		ret = 0;
+	}
+
+	rcu_read_unlock_trace();
+
+	return ret && oc->bpf_memory_freed;
+}
+

[ ... ]

Quoted 5 lines
mm/bpf_oom.c new file +5 -0
+static int bpf_oom_ops_reg(void *kdata, struct bpf_link *link)
+{
+	struct bpf_struct_ops_link *st_link = (struct bpf_struct_ops_link *)link;
+	struct cgroup *cgrp;
+

mm/bpf_oom.c line 116

Quoted 10 lines
mm/bpf_oom.c new file +10 -0
+	/* The link is not yet fully initialized, but cgroup should be set */
+	if (!link)
+		return -EOPNOTSUPP;
+
+	cgrp = st_link->cgroup;
+	if (!cgrp)
+		return -EINVAL;
+
+	if (cmpxchg(&cgrp->bpf.bpf_oom_link, NULL, st_link))
+		return -EEXIST;

iiuc, this will allow only one oom_ops to be attached to a cgroup. Considering oom_ops is the only user of the cgrp->bpf.struct_ops_links (added in patch 2), the list should have only one element for now.

Quoted 1 line

Copy some context from the patch 2 commit log.

Hi Martin!

Sorry, I'm not quite sure what do you mean, can you please elaborate more?

Quoted 8 lines

We decided (in conversations at LPC) that 1 bpf oom policy for memcg is good for now (with a potential to extend in the future, if there will be use cases). But it seems like there is a lot of interest to attach struct ops'es to cgroups (there are already a couple of patchsets posted based on my earlier v2 patches), so I tried to make the bpf link mechanics suitable for multiple use cases from scratch.

Did I answer your question?

Got it. The link list is for the future struct_ops implementations to attach to a cgroup.

I should have mentioned the context. My bad.

BPF_PROG_TYPE_SOCK_OPS is currently a cgroup BPF prog. I am thinking of adding a bpf_struct_ops support to have similar hooks as in the BPF_PROG_TYPE_SOCK_OPS. There are some issues that need to be worked out. A major one is that the current cgroup progs have expectations on the ordering and override behavior based on the BPF_F_* and the runtime cgroup hierarchy. I was trying to see if there are pieces in this set that can be built upon. The linked list is a start but will need more work to make it performant for networking use.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

This change doesn't answer the question how bpf programs belonging to these struct ops'es will be executed. It will be done individually for every bpf struct ops which supports this.

Please, note that unlike "normal" bpf programs, struct ops'es are not propagated to cgroup sub-trees.

Quoted 8 lines
Quoted 4 lines

There are NONE, BPF_F_ALLOW_OVERRIDE, and BPF_F_ALLOW_MULTI, which one may be closer to the bpf_handle_oom() semantic. If it needs to change the ordering (or allow multi) in the future, does it need a new flag or the existing BPF_F_xxx flags can be used.

I hope that existing flags can be used, but also I'm not sure we ever would need multiple oom handlers per cgroup. Do you have any specific concerns here?

Another question that I have is the default behavior when none of the BPF_F_* is specified when attaching a struct_ops to a cgroup.

From uapi/bpf.h:

* NONE (default): No further BPF programs allowed in the subtree

iiuc, the bpf_handle_oom() is not the same as NONE. Should each struct_ops implementation have its own default policy? For the BPF_PROG_TYPE_SOCK_OPS work, I am thinking the default policy should be BPF_F_ALLOW_MULTI which is always on/set now in the cgroup_bpf_link_attach().

author maintainer
#35 lore raw

PATCH v3 8/17 mm: introduce bpf_oom_kill_process() bpf kfunc

In reply to Roman Gushchin

Introduce bpf_oom_kill_process() bpf kfunc, which is supposed to be used by BPF OOM programs. It allows to kill a process in exactly the same way the OOM killer does: using the OOM reaper, bumping corresponding memcg and global statistics, respecting memory.oom.group etc.

On success, it sets the oom_control's bpf_memory_freed field to true, enabling the bpf program to bypass the kernel OOM killer.

---
 mm/oom_kill.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
mm/oom_kill.c +80 -0
index 44bbcf033804..09897597907f 100644
@@ -46,6 +46,7 @@
 #include <linux/cred.h>
 #include <linux/nmi.h>
 #include <linux/bpf_oom.h>
+#include <linux/btf.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -1290,3 +1291,82 @@ SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags)
 	return -ENOSYS;
 #endif /* CONFIG_MMU */
 }
+
+#ifdef CONFIG_BPF_SYSCALL
+
+__bpf_kfunc_start_defs();
+/**
+ * bpf_oom_kill_process - Kill a process as OOM killer
+ * @oc: pointer to oom_control structure, describes OOM context
+ * @task: task to be killed
+ * @message__str: message to print in dmesg
+ *
+ * Kill a process in a way similar to the kernel OOM killer.
+ * This means dump the necessary information to dmesg, adjust memcg
+ * statistics, leverage the oom reaper, respect memory.oom.group etc.
+ *
+ * bpf_oom_kill_process() marks the forward progress by setting
+ * oc->bpf_memory_freed. If the progress was made, the bpf program
+ * is free to decide if the kernel oom killer should be invoked.
+ * Otherwise it's enforced, so that a bad bpf program can't
+ * deadlock the machine on memory.
+ */
+__bpf_kfunc int bpf_oom_kill_process(struct oom_control *oc,
+				     struct task_struct *task,
+				     const char *message__str)
+{
+	if (oom_unkillable_task(task))
+		return -EPERM;
+
+	if (task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN)
+		return -EINVAL;
+
+	/* paired with put_task_struct() in oom_kill_process() */
+	get_task_struct(task);
+
+	oc->chosen = task;
+
+	oom_kill_process(oc, message__str);
+
+	oc->chosen = NULL;
+	oc->bpf_memory_freed = true;
+
+	return 0;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_oom_kfuncs)
+BTF_ID_FLAGS(func, bpf_oom_kill_process, KF_SLEEPABLE)
+BTF_KFUNCS_END(bpf_oom_kfuncs)
+
+BTF_ID_LIST_SINGLE(bpf_oom_ops_ids, struct, bpf_oom_ops)
+
+static int bpf_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
+{
+	if (prog->type != BPF_PROG_TYPE_STRUCT_OPS ||
+	    prog->aux->attach_btf_id != bpf_oom_ops_ids[0])
+		return -EACCES;
+	return 0;
+}
+
+static const struct btf_kfunc_id_set bpf_oom_kfunc_set = {
+	.owner          = THIS_MODULE,
+	.set            = &bpf_oom_kfuncs,
+	.filter         = bpf_oom_kfunc_filter,
+};
+
+static int __init bpf_oom_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+					&bpf_oom_kfunc_set);
+	if (err)
+		pr_warn("error while registering bpf oom kfuncs: %d", err);
+
+	return err;
+}
+late_initcall(bpf_oom_init);
+
+#endif
Signature · 1 line
2.52.0
#36 lore raw

Re: [PATCH bpf-next v3 08/17] mm: introduce bpf_oom_kill_process() bpf kfunc

Quoted 1 line · Roman Gushchin

+static int bpf_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)

The filter callback is registered for BPF_PROG_TYPE_STRUCT_OPS. It is checking if a kfunc_id is allowed for other struct_ops progs also, e.g. the bpf-tcp-cc struct_ops progs.

mm/oom_kill.c SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags) · line 1349

Quoted 4 lines
mm/oom_kill.c +4 -0
+{
+	if (prog->type != BPF_PROG_TYPE_STRUCT_OPS ||
+	    prog->aux->attach_btf_id != bpf_oom_ops_ids[0])
+		return -EACCES;

The 'return -EACCES' should be the cause of the "calling kernel function XXX is not allowed" error reported by the CI. Take a look at btf_kfunc_is_allowed().

Take a look at bpf_qdisc_kfunc_filter(). I suspect it should be something like this, untested:

        if (btf_id_set8_contains(&bpf_oom_kfuncs, kfunc_id) &&
	    prog->aux->st_ops != &bpf_oom_bpf_ops)
                return -EACCES;

        return 0;
Quoted 13 lines
mm/oom_kill.c +13 -0
+
+static const struct btf_kfunc_id_set bpf_oom_kfunc_set = {
+	.owner          = THIS_MODULE,
+	.set            = &bpf_oom_kfuncs,
+	.filter         = bpf_oom_kfunc_filter,
+};
+
+static int __init bpf_oom_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+					&bpf_oom_kfunc_set);
author maintainer
#37 lore raw

Re: [PATCH bpf-next v3 08/17] mm: introduce bpf_oom_kill_process() bpf kfunc

Quoted 17 lines · Martin KaFai Lau
Quoted 1 line · Roman Gushchin

+static int bpf_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)

The filter callback is registered for BPF_PROG_TYPE_STRUCT_OPS. It is checking if a kfunc_id is allowed for other struct_ops progs also, e.g. the bpf-tcp-cc struct_ops progs.

mm/oom_kill.c SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags) · line 1349

Quoted 4 lines
mm/oom_kill.c +4 -0
+{
+	if (prog->type != BPF_PROG_TYPE_STRUCT_OPS ||
+	    prog->aux->attach_btf_id != bpf_oom_ops_ids[0])
+		return -EACCES;

The 'return -EACCES' should be the cause of the "calling kernel function XXX is not allowed" error reported by the CI. Take a look at btf_kfunc_is_allowed().

Quoted 8 lines

Take a look at bpf_qdisc_kfunc_filter(). I suspect it should be something like this, untested:

        if (btf_id_set8_contains(&bpf_oom_kfuncs, kfunc_id) &&
	    prog->aux->st_ops != &bpf_oom_bpf_ops)
                return -EACCES;

        return 0;

Oh, I see.. It's a bit surprising that these .filter() functions have non-local effects...

Will fix in v4.

Thank you, Martin!

#38 lore raw

Re: [PATCH bpf-next v3 08/17] mm: introduce bpf_oom_kill_process() bpf kfunc

In reply to Roman Gushchin
Quoted 49 lines · Roman Gushchin

Introduce bpf_oom_kill_process() bpf kfunc, which is supposed to be used by BPF OOM programs. It allows to kill a process in exactly the same way the OOM killer does: using the OOM reaper, bumping corresponding memcg and global statistics, respecting memory.oom.group etc.

On success, it sets the oom_control's bpf_memory_freed field to true, enabling the bpf program to bypass the kernel OOM killer.

---
 mm/oom_kill.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
mm/oom_kill.c +20 -0
index 44bbcf033804..09897597907f 100644
@@ -46,6 +46,7 @@
 #include <linux/cred.h>
 #include <linux/nmi.h>
 #include <linux/bpf_oom.h>
+#include <linux/btf.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -1290,3 +1291,82 @@ SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags)
 	return -ENOSYS;
 #endif /* CONFIG_MMU */
 }
+
+#ifdef CONFIG_BPF_SYSCALL
+
+__bpf_kfunc_start_defs();
+/**
+ * bpf_oom_kill_process - Kill a process as OOM killer
+ * @oc: pointer to oom_control structure, describes OOM context
+ * @task: task to be killed
+ * @message__str: message to print in dmesg
+ *
+ * Kill a process in a way similar to the kernel OOM killer.
+ * This means dump the necessary information to dmesg, adjust memcg
+ * statistics, leverage the oom reaper, respect memory.oom.group etc.
+ *
+ * bpf_oom_kill_process() marks the forward progress by setting
+ * oc->bpf_memory_freed. If the progress was made, the bpf program
+ * is free to decide if the kernel oom killer should be invoked.
+ * Otherwise it's enforced, so that a bad bpf program can't
+ * deadlock the machine on memory.

mm/oom_kill.c SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags) · line 1322

Quoted 10 lines
mm/oom_kill.c +10 -0
+ */
+__bpf_kfunc int bpf_oom_kill_process(struct oom_control *oc,
+				     struct task_struct *task,
+				     const char *message__str)
+{
+	if (oom_unkillable_task(task))
+		return -EPERM;
+
+	if (task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN)
+		return -EINVAL;

task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN is also representative of an unkillable task, so why not fold this up into the above conditional? Also, why not bother checking states like mm_flags_test(MMF_OOM_SKIP, task->mm) and in_vfork() here too?

In all fairness I'm a little surprised about constraints like task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN being enforced here. You could argue that the whole purpose of BPF OOM is such that you can implement your own victim selection algorithms entirely in BPF using your own set of heuristics and what not without needing to strictly respect properties like oom_score_adj.

In any case, I think we should at least clearly document such constraints.

Quoted 52 lines
mm/oom_kill.c +49 -0
+	/* paired with put_task_struct() in oom_kill_process() */
+	get_task_struct(task);
+
+	oc->chosen = task;
+
+	oom_kill_process(oc, message__str);
+
+	oc->chosen = NULL;
+	oc->bpf_memory_freed = true;
+
+	return 0;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_oom_kfuncs)
+BTF_ID_FLAGS(func, bpf_oom_kill_process, KF_SLEEPABLE)
+BTF_KFUNCS_END(bpf_oom_kfuncs)
+
+BTF_ID_LIST_SINGLE(bpf_oom_ops_ids, struct, bpf_oom_ops)
+
+static int bpf_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
+{
+	if (prog->type != BPF_PROG_TYPE_STRUCT_OPS ||
+	    prog->aux->attach_btf_id != bpf_oom_ops_ids[0])
+		return -EACCES;
+	return 0;
+}
+
+static const struct btf_kfunc_id_set bpf_oom_kfunc_set = {
+	.owner          = THIS_MODULE,
+	.set            = &bpf_oom_kfuncs,
+	.filter         = bpf_oom_kfunc_filter,
+};
+
+static int __init bpf_oom_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+					&bpf_oom_kfunc_set);
+	if (err)
+		pr_warn("error while registering bpf oom kfuncs: %d", err);
+
+	return err;
+}
+late_initcall(bpf_oom_init);
+
+#endif
Signature · 1 line
2.52.0
#39 lore raw

Re: [PATCH bpf-next v3 08/17] mm: introduce bpf_oom_kill_process() bpf kfunc

Quoted 52 lines · Matt Bobrowski
Quoted 51 lines · Roman Gushchin

Introduce bpf_oom_kill_process() bpf kfunc, which is supposed to be used by BPF OOM programs. It allows to kill a process in exactly the same way the OOM killer does: using the OOM reaper, bumping corresponding memcg and global statistics, respecting memory.oom.group etc.

On success, it sets the oom_control's bpf_memory_freed field to true, enabling the bpf program to bypass the kernel OOM killer.

---
  mm/oom_kill.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 80 insertions(+)
mm/oom_kill.c +22 -0
index 44bbcf033804..09897597907f 100644
@@ -46,6 +46,7 @@
  #include <linux/cred.h>
  #include <linux/nmi.h>
  #include <linux/bpf_oom.h>
+#include <linux/btf.h>
 
  #include <asm/tlb.h>
  #include "internal.h"
@@ -1290,3 +1291,82 @@ SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags)
  	return -ENOSYS;
  #endif /* CONFIG_MMU */
  }
+
+#ifdef CONFIG_BPF_SYSCALL
+
+__bpf_kfunc_start_defs();
+/**
+ * bpf_oom_kill_process - Kill a process as OOM killer
+ * @oc: pointer to oom_control structure, describes OOM context
+ * @task: task to be killed
+ * @message__str: message to print in dmesg
+ *
+ * Kill a process in a way similar to the kernel OOM killer.
+ * This means dump the necessary information to dmesg, adjust memcg
+ * statistics, leverage the oom reaper, respect memory.oom.group etc.
+ *
+ * bpf_oom_kill_process() marks the forward progress by setting
+ * oc->bpf_memory_freed. If the progress was made, the bpf program
+ * is free to decide if the kernel oom killer should be invoked.
+ * Otherwise it's enforced, so that a bad bpf program can't
+ * deadlock the machine on memory.
+ */
+__bpf_kfunc int bpf_oom_kill_process(struct oom_control *oc,
Quoted 8 lines

mm/oom_kill.c SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags) · line 1322

Quoted 8 lines
mm/oom_kill.c +8 -0
+				     struct task_struct *task,
+				     const char *message__str)
+{
+	if (oom_unkillable_task(task))
+		return -EPERM;
+
+	if (task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN)
+		return -EINVAL;

Hi Matt and Roman

Quoted 11 lines

task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN is also representative of an unkillable task, so why not fold this up into the above conditional? Also, why not bother checking states like mm_flags_test(MMF_OOM_SKIP, task->mm) and in_vfork() here too?

In all fairness I'm a little surprised about constraints like task->signal->oom_score_adj == OOM_SCORE_ADJ_MIN being enforced here. You could argue that the whole purpose of BPF OOM is such that you can implement your own victim selection algorithms entirely in BPF using your own set of heuristics and what not without needing to strictly respect properties like oom_score_adj.

Just my 2 cents — historically many users rely on setting oom_score_adj to OOM_SCORE_ADJ_MIN to protect their critical processes.

If we don't check this here and a custom BPF program happens to forget it, users would be quite surprised to find their most important

processes killed despite having explicitly marked them as protected.

So this really comes down to a priority question: when a custom BPF OOM policy decides to kill a process that the admin has explicitly

marked as protected via OOM_SCORE_ADJ_MIN, which one should take precedence?

Quoted 2 lines

In any case, I think we should at least clearly document such constraints.

Fully agree. We need thorough documentation on what bpf_oom_kill_process() enforces on the kernel side,

so that custom BPF programs know exactly what's already handled and what additional checks they may

need to perform depending on their use case.

Thanks

Best regards.
Hao

Quoted 52 lines
Quoted 52 lines
mm/oom_kill.c +49 -0
+	/* paired with put_task_struct() in oom_kill_process() */
+	get_task_struct(task);
+
+	oc->chosen = task;
+
+	oom_kill_process(oc, message__str);
+
+	oc->chosen = NULL;
+	oc->bpf_memory_freed = true;
+
+	return 0;
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(bpf_oom_kfuncs)
+BTF_ID_FLAGS(func, bpf_oom_kill_process, KF_SLEEPABLE)
+BTF_KFUNCS_END(bpf_oom_kfuncs)
+
+BTF_ID_LIST_SINGLE(bpf_oom_ops_ids, struct, bpf_oom_ops)
+
+static int bpf_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
+{
+	if (prog->type != BPF_PROG_TYPE_STRUCT_OPS ||
+	    prog->aux->attach_btf_id != bpf_oom_ops_ids[0])
+		return -EACCES;
+	return 0;
+}
+
+static const struct btf_kfunc_id_set bpf_oom_kfunc_set = {
+	.owner          = THIS_MODULE,
+	.set            = &bpf_oom_kfuncs,
+	.filter         = bpf_oom_kfunc_filter,
+};
+
+static int __init bpf_oom_init(void)
+{
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+					&bpf_oom_kfunc_set);
+	if (err)
+		pr_warn("error while registering bpf oom kfuncs: %d", err);
+
+	return err;
+}
+late_initcall(bpf_oom_init);
+
+#endif
Signature · 1 line
2.52.0
author maintainer
#40 lore raw

PATCH v3 9/17 mm: introduce bpf_out_of_memory() BPF kfunc

In reply to Roman Gushchin

Introduce bpf_out_of_memory() bpf kfunc, which allows to declare an out of memory events and trigger the corresponding kernel OOM handling mechanism.

It takes a trusted memcg pointer (or NULL for system-wide OOMs) as an argument, as well as the page order.

If the BPF_OOM_FLAGS_WAIT_ON_OOM_LOCK flag is not set, only one OOM can be declared and handled in the system at once, so if the function is called in parallel to another OOM handling, it bails out with -EBUSY. This mode is suited for global OOM's: any concurrent OOMs will likely do the job and release some memory. In a blocking mode (which is suited for memcg OOMs) the execution will wait on the oom_lock mutex.

The function is declared as sleepable. It guarantees that it won't be called from an atomic context. It's required by the OOM handling code, which shouldn't be called from a non-blocking context.

Handling of a memcg OOM almost always requires taking of the css_set_lock spinlock. The fact that bpf_out_of_memory() is sleepable also guarantees that it can't be called with acquired css_set_lock, so the kernel can't deadlock on it.

To avoid deadlocks on the oom lock, the function is filtered out for bpf oom struct ops programs and all tracing programs.

---
 include/linux/oom.h |  5 +++
 mm/oom_kill.c       | 85 +++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 88 insertions(+), 2 deletions(-)
include/linux/oom.h +5 -0
index c2dce336bcb4..851dba9287b5 100644
@@ -21,6 +21,11 @@ enum oom_constraint {
 	CONSTRAINT_MEMCG,
 };
 
+enum bpf_oom_flags {
+	BPF_OOM_FLAGS_WAIT_ON_OOM_LOCK = 1 << 0,
+	BPF_OOM_FLAGS_LAST = 1 << 1,
+};
+
 /*
  * Details of the page allocation that triggered the oom killer that are used to
  * determine what should be killed.
mm/oom_kill.c +83 -2

103 lines not on this page · raw message

Signature · 1 line
2.52.0
#41 lore raw

Re: [PATCH bpf-next v3 09/17] mm: introduce bpf_out_of_memory() BPF kfunc

Quoted 82 lines · Roman Gushchin

Introduce bpf_out_of_memory() bpf kfunc, which allows to declare an out of memory events and trigger the corresponding kernel OOM handling mechanism.

It takes a trusted memcg pointer (or NULL for system-wide OOMs) as an argument, as well as the page order.

If the BPF_OOM_FLAGS_WAIT_ON_OOM_LOCK flag is not set, only one OOM can be declared and handled in the system at once, so if the function is called in parallel to another OOM handling, it bails out with -EBUSY. This mode is suited for global OOM's: any concurrent OOMs will likely do the job and release some memory. In a blocking mode (which is suited for memcg OOMs) the execution will wait on the oom_lock mutex.

The function is declared as sleepable. It guarantees that it won't be called from an atomic context. It's required by the OOM handling code, which shouldn't be called from a non-blocking context.

Handling of a memcg OOM almost always requires taking of the css_set_lock spinlock. The fact that bpf_out_of_memory() is sleepable also guarantees that it can't be called with acquired css_set_lock, so the kernel can't deadlock on it.

To avoid deadlocks on the oom lock, the function is filtered out for bpf oom struct ops programs and all tracing programs.

---
 include/linux/oom.h |  5 +++
 mm/oom_kill.c       | 85 +++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 88 insertions(+), 2 deletions(-)
include/linux/oom.h +5 -0
index c2dce336bcb4..851dba9287b5 100644
@@ -21,6 +21,11 @@ enum oom_constraint {
 	CONSTRAINT_MEMCG,
 };
 
+enum bpf_oom_flags {
+	BPF_OOM_FLAGS_WAIT_ON_OOM_LOCK = 1 << 0,
+	BPF_OOM_FLAGS_LAST = 1 << 1,
+};
+
 /*
  * Details of the page allocation that triggered the oom killer that are used to
  * determine what should be killed.
mm/oom_kill.c +26 -0
index 09897597907f..8f63a370b8f5 100644
@@ -1334,6 +1334,53 @@ __bpf_kfunc int bpf_oom_kill_process(struct oom_control *oc,
 	return 0;
 }
 
+/**
+ * bpf_out_of_memory - declare Out Of Memory state and invoke OOM killer
+ * @memcg__nullable: memcg or NULL for system-wide OOMs
+ * @order: order of page which wasn't allocated
+ * @flags: flags
+ *
+ * Declares the Out Of Memory state and invokes the OOM killer.
+ *
+ * OOM handlers are synchronized using the oom_lock mutex. If wait_on_oom_lock
+ * is true, the function will wait on it. Otherwise it bails out with -EBUSY
+ * if oom_lock is contended.
+ *
+ * Generally it's advised to pass wait_on_oom_lock=false for global OOMs
+ * and wait_on_oom_lock=true for memcg-scoped OOMs.
+ *
+ * Returns 1 if the forward progress was achieved and some memory was freed.
+ * Returns a negative value if an error occurred.
+ */
+__bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
+				  int order, u64 flags)
+{
+	struct oom_control oc = {
+		.memcg = memcg__nullable,
+		.gfp_mask = GFP_KERNEL,
+		.order = order,
+	};

mm/oom_kill.c __bpf_kfunc int bpf_oom_kill_process(struct oom_control *oc, · line 1372

Quoted 10 lines
mm/oom_kill.c +10 -0
+	int ret;
+
+	if (flags & ~(BPF_OOM_FLAGS_LAST - 1))
+		return -EINVAL;
+
+	if (oc.order < 0 || oc.order > MAX_PAGE_ORDER)
+		return -EINVAL;
+
+	if (flags & BPF_OOM_FLAGS_WAIT_ON_OOM_LOCK) {
+		ret = mutex_lock_killable(&oom_lock);

If contended and we end up waiting here, some forward progress could have been made in the interim. Enough such that this pending OOM event initiated by the call into bpf_out_of_memory() may no longer even be warranted. What do you think about adding an escape hatch here, which could simply be in the form of a user-defined function callback?

Quoted 68 lines
mm/oom_kill.c +47 -2
+		if (ret)
+			return ret;
+	} else if (!mutex_trylock(&oom_lock))
+		return -EBUSY;
+
+	ret = out_of_memory(&oc);
+
+	mutex_unlock(&oom_lock);
+	return ret;
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(bpf_oom_kfuncs)
@@ -1356,14 +1403,48 @@ static const struct btf_kfunc_id_set bpf_oom_kfunc_set = {
 	.filter         = bpf_oom_kfunc_filter,
 };
 
+BTF_KFUNCS_START(bpf_declare_oom_kfuncs)
+BTF_ID_FLAGS(func, bpf_out_of_memory, KF_SLEEPABLE)
+BTF_KFUNCS_END(bpf_declare_oom_kfuncs)
+
+static int bpf_declare_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
+{
+	if (!btf_id_set8_contains(&bpf_declare_oom_kfuncs, kfunc_id))
+		return 0;
+
+	if (prog->type == BPF_PROG_TYPE_STRUCT_OPS &&
+	    prog->aux->attach_btf_id == bpf_oom_ops_ids[0])
+		return -EACCES;
+
+	if (prog->type == BPF_PROG_TYPE_TRACING)
+		return -EACCES;
+
+	return 0;
+}
+
+static const struct btf_kfunc_id_set bpf_declare_oom_kfunc_set = {
+	.owner          = THIS_MODULE,
+	.set            = &bpf_declare_oom_kfuncs,
+	.filter         = bpf_declare_oom_kfunc_filter,
+};
+
 static int __init bpf_oom_init(void)
 {
 	int err;
 
 	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
 					&bpf_oom_kfunc_set);
-	if (err)
-		pr_warn("error while registering bpf oom kfuncs: %d", err);
+	if (err) {
+		pr_warn("error while registering struct_ops bpf oom kfuncs: %d", err);
+		return err;
+	}
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC,
+					&bpf_declare_oom_kfunc_set);
+	if (err) {
+		pr_warn("error while registering unspec bpf oom kfuncs: %d", err);
+		return err;
+	}
 
 	return err;
 }
Signature · 1 line
2.52.0
author maintainer
#42 lore raw

PATCH v3 10/17 mm: introduce bpf_task_is_oom_victim() kfunc

In reply to Roman Gushchin

Export tsk_is_oom_victim() helper as a BPF kfunc.
It's very useful to avoid redundant oom kills.

---
 mm/oom_kill.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
mm/oom_kill.c +14 -0
index 8f63a370b8f5..53f9f9674658 100644
@@ -1381,10 +1381,24 @@ __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
 	return ret;
 }
 
+/**
+ * bpf_task_is_oom_victim - Check if the task has been marked as an OOM victim
+ * @task: task to check
+ *
+ * Returns true if the task has been previously selected by the OOM killer
+ * to be killed. It's expected that the task will be destroyed soon and some
+ * memory will be freed, so maybe no additional actions required.
+ */
+__bpf_kfunc bool bpf_task_is_oom_victim(struct task_struct *task)
+{
+	return tsk_is_oom_victim(task);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(bpf_oom_kfuncs)
 BTF_ID_FLAGS(func, bpf_oom_kill_process, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_task_is_oom_victim)
 BTF_KFUNCS_END(bpf_oom_kfuncs)
 
 BTF_ID_LIST_SINGLE(bpf_oom_ops_ids, struct, bpf_oom_ops)
Signature · 1 line
2.52.0
#43 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 19 lines · Roman Gushchin

Export tsk_is_oom_victim() helper as a BPF kfunc.
It's very useful to avoid redundant oom kills.

---
 mm/oom_kill.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
mm/oom_kill.c +2 -0
index 8f63a370b8f5..53f9f9674658 100644
@@ -1381,10 +1381,24 @@ __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
 	return ret;
 }
 
+/**
+ * bpf_task_is_oom_victim - Check if the task has been marked as an OOM victim

mm/oom_kill.c __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable, · line 1395

Quoted 10 lines
mm/oom_kill.c +10 -0
+ * @task: task to check
+ *
+ * Returns true if the task has been previously selected by the OOM killer
+ * to be killed. It's expected that the task will be destroyed soon and some
+ * memory will be freed, so maybe no additional actions required.
+ */
+__bpf_kfunc bool bpf_task_is_oom_victim(struct task_struct *task)
+{
+	return tsk_is_oom_victim(task);
+}

Why not just do a direct memory read (i.e., task->signal->oom_mm) within the BPF program? I'm not quite convinced that a BPF kfunc wrapper for something like tsk_is_oom_victim() is warranted as you can literally achieve the same semantics without one.

#44 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 28 lines · Matt Bobrowski
Quoted 26 lines · Roman Gushchin

Export tsk_is_oom_victim() helper as a BPF kfunc.
It's very useful to avoid redundant oom kills.

---
 mm/oom_kill.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
mm/oom_kill.c +9 -0
index 8f63a370b8f5..53f9f9674658 100644
@@ -1381,10 +1381,24 @@ __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
      return ret;
 }
 
+/**
+ * bpf_task_is_oom_victim - Check if the task has been marked as an OOM victim
+ * @task: task to check
+ *
+ * Returns true if the task has been previously selected by the OOM killer
+ * to be killed. It's expected that the task will be destroyed soon and some
+ * memory will be freed, so maybe no additional actions required.
+ */
+__bpf_kfunc bool bpf_task_is_oom_victim(struct task_struct *task)
Quoted 8 lines

mm/oom_kill.c __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable, · line 1395

Quoted 3 lines
mm/oom_kill.c +3 -0
+{
+     return tsk_is_oom_victim(task);
+}

Why not just do a direct memory read (i.e., task->signal->oom_mm) within the BPF program? I'm not quite convinced that a BPF kfunc wrapper for something like tsk_is_oom_victim() is warranted as you can literally achieve the same semantics without one.

+1
there is no need for this kfunc.

author maintainer
#45 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 38 lines · Alexei Starovoitov
Quoted 28 lines · Matt Bobrowski
Quoted 26 lines · Roman Gushchin

Export tsk_is_oom_victim() helper as a BPF kfunc.
It's very useful to avoid redundant oom kills.

---
 mm/oom_kill.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
mm/oom_kill.c +9 -0
index 8f63a370b8f5..53f9f9674658 100644
@@ -1381,10 +1381,24 @@ __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
      return ret;
 }
 
+/**
+ * bpf_task_is_oom_victim - Check if the task has been marked as an OOM victim
+ * @task: task to check
+ *
+ * Returns true if the task has been previously selected by the OOM killer
+ * to be killed. It's expected that the task will be destroyed soon and some
+ * memory will be freed, so maybe no additional actions required.
+ */
+__bpf_kfunc bool bpf_task_is_oom_victim(struct task_struct *task)
Quoted 8 lines

mm/oom_kill.c __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable, · line 1395

Quoted 3 lines
mm/oom_kill.c +3 -0
+{
+     return tsk_is_oom_victim(task);
+}

Why not just do a direct memory read (i.e., task->signal->oom_mm) within the BPF program? I'm not quite convinced that a BPF kfunc wrapper for something like tsk_is_oom_victim() is warranted as you can literally achieve the same semantics without one.

Quoted 2 lines

+1
there is no need for this kfunc.

It was explicitly asked by Michal Hocko, who is (co)maintaining the oom code. I don't have a strong opinion here. I agree that it can be easily open-coded without a kfunc, but at the same time the cost of having an extra kfunc is not high and it makes the API more consistent.

Michal, do you feel strongly about having a dedicated kfunc vs the direct memory read?

maintainer
#46 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 43 lines · Roman Gushchin
Quoted 38 lines · Alexei Starovoitov
Quoted 28 lines · Matt Bobrowski
Quoted 26 lines · Roman Gushchin

Export tsk_is_oom_victim() helper as a BPF kfunc.
It's very useful to avoid redundant oom kills.

---
 mm/oom_kill.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
mm/oom_kill.c +9 -0
index 8f63a370b8f5..53f9f9674658 100644
@@ -1381,10 +1381,24 @@ __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
      return ret;
 }
 
+/**
+ * bpf_task_is_oom_victim - Check if the task has been marked as an OOM victim
+ * @task: task to check
+ *
+ * Returns true if the task has been previously selected by the OOM killer
+ * to be killed. It's expected that the task will be destroyed soon and some
+ * memory will be freed, so maybe no additional actions required.
+ */
+__bpf_kfunc bool bpf_task_is_oom_victim(struct task_struct *task)
Quoted 8 lines

mm/oom_kill.c __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable, · line 1395

Quoted 3 lines
mm/oom_kill.c +3 -0
+{
+     return tsk_is_oom_victim(task);
+}

Why not just do a direct memory read (i.e., task->signal->oom_mm) within the BPF program? I'm not quite convinced that a BPF kfunc wrapper for something like tsk_is_oom_victim() is warranted as you can literally achieve the same semantics without one.

Quoted 2 lines

+1
there is no need for this kfunc.

Quoted 7 lines

It was explicitly asked by Michal Hocko, who is (co)maintaining the oom code. I don't have a strong opinion here. I agree that it can be easily open-coded without a kfunc, but at the same time the cost of having an extra kfunc is not high and it makes the API more consistent.

Michal, do you feel strongly about having a dedicated kfunc vs the direct memory read?

The reason I wanted this an explicit API is that oom states are quite internal part of the oom synchronization. And I would really like to have that completely transparent for oom policies. In other words I do not want to touch all potential oom policies or break them in the worst case just because we need to change this. So while a trivial interface now (and hopefully for a long time) it is really an internal thing.

Do I insist? No, I do not but I would like to hear why this is a bad idea.

Signature · 2 lines
Michal Hocko
SUSE Labs
#47 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 60 lines · Michal Hocko
Quoted 43 lines · Roman Gushchin
Quoted 38 lines · Alexei Starovoitov
Quoted 28 lines · Matt Bobrowski
Quoted 26 lines · Roman Gushchin

Export tsk_is_oom_victim() helper as a BPF kfunc.
It's very useful to avoid redundant oom kills.

---
 mm/oom_kill.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
mm/oom_kill.c +9 -0
index 8f63a370b8f5..53f9f9674658 100644
@@ -1381,10 +1381,24 @@ __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable,
      return ret;
 }
 
+/**
+ * bpf_task_is_oom_victim - Check if the task has been marked as an OOM victim
+ * @task: task to check
+ *
+ * Returns true if the task has been previously selected by the OOM killer
+ * to be killed. It's expected that the task will be destroyed soon and some
+ * memory will be freed, so maybe no additional actions required.
+ */
+__bpf_kfunc bool bpf_task_is_oom_victim(struct task_struct *task)
Quoted 8 lines

mm/oom_kill.c __bpf_kfunc int bpf_out_of_memory(struct mem_cgroup *memcg__nullable, · line 1395

Quoted 3 lines
mm/oom_kill.c +3 -0
+{
+     return tsk_is_oom_victim(task);
+}

Why not just do a direct memory read (i.e., task->signal->oom_mm) within the BPF program? I'm not quite convinced that a BPF kfunc wrapper for something like tsk_is_oom_victim() is warranted as you can literally achieve the same semantics without one.

Quoted 2 lines

+1
there is no need for this kfunc.

Quoted 7 lines

It was explicitly asked by Michal Hocko, who is (co)maintaining the oom code. I don't have a strong opinion here. I agree that it can be easily open-coded without a kfunc, but at the same time the cost of having an extra kfunc is not high and it makes the API more consistent.

Michal, do you feel strongly about having a dedicated kfunc vs the direct memory read?

The reason I wanted this an explicit API is that oom states are quite internal part of the oom synchronization. And I would really like to have that completely transparent for oom policies. In other words I do not want to touch all potential oom policies or break them in the worst case just because we need to change this. So while a trivial interface now (and hopefully for a long time) it is really an internal thing.

Quoted 2 lines

Do I insist? No, I do not but I would like to hear why this is a bad idea.

It's a bad idea, since it doesn't address your goal. bpf prog can access task->signal->oom_mm without kfunc just fine and it will be doing so because performance matters and static inline bool foo(task)
{
return task->signal->oom_mm;
}

will be inlined as 2 loads while kfunc is a function call with 6 registers being scratched.

If anything changes and, say, oom_mm will get renamed whether it was kfunc or not doesn't change much. progs will adopt to a new way easily with CORE. kfuncs can also be renamed/deleted, etc. You're thinking about kfuncs as a stable api. It's definitely not. It's not a layer of isolation either. kfuncs are necessary only for the cases where bpf prog cannot do it on its own.

"internal thing" is also a wrong way of thinking of bpf-oom. bpf-oom _will_ look into oom, cgroup and kernel internals in general. All bpf progs do because they have to do that to achieve their goals. Everything in mm/internal.h have been available to access by bpf progs for a decade now. Did it cause any issue to mm development? No. So let's not build some non-existent wall or "internal oom thing".

maintainer
#48 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 3 lines · Alexei Starovoitov
Quoted 2 lines · Michal Hocko

On Mon 02-02-26 16:14:37, Roman Gushchin wrote:

[...]

Quoted 13 lines
Quoted 10 lines
Quoted 2 lines

Michal, do you feel strongly about having a dedicated kfunc vs the direct memory read?

The reason I wanted this an explicit API is that oom states are quite internal part of the oom synchronization. And I would really like to have that completely transparent for oom policies. In other words I do not want to touch all potential oom policies or break them in the worst case just because we need to change this. So while a trivial interface now (and hopefully for a long time) it is really an internal thing.

Quoted 2 lines

Do I insist? No, I do not but I would like to hear why this is a bad idea.

Quoted 7 lines

It's a bad idea, since it doesn't address your goal. bpf prog can access task->signal->oom_mm without kfunc just fine and it will be doing so because performance matters and static inline bool foo(task)
{
return task->signal->oom_mm;
}

OK, so my understanding was that BPF can only use exported functionality. If those progs can access whatever they get a pointer for and than traverse down the road then this is moot from a large part.

Quoted 2 lines

will be inlined as 2 loads while kfunc is a function call with 6 registers being scratched.

performance is not really crucial in this context. We are OOM, couple of loads vs. registers will not make much difference. It is really more about code writers what they can/should be using. OOM is a piece of complex code with many loose ends that might not be obvious.

Quoted 6 lines

If anything changes and, say, oom_mm will get renamed whether it was kfunc or not doesn't change much. progs will adopt to a new way easily with CORE. kfuncs can also be renamed/deleted, etc. You're thinking about kfuncs as a stable api. It's definitely not. It's not a layer of isolation either. kfuncs are necessary only for the cases where bpf prog cannot do it on its own.

It is obviously not clear to me where that line is for BPF progs. Where is this documented?

Signature · 2 lines
Michal Hocko
SUSE Labs
#49 lore raw

Re: [PATCH bpf-next v3 10/17] mm: introduce bpf_task_is_oom_victim() kfunc

Quoted 27 lines · Michal Hocko
Quoted 3 lines · Alexei Starovoitov
Quoted 2 lines · Michal Hocko

On Mon 02-02-26 16:14:37, Roman Gushchin wrote:

[...]

Quoted 13 lines
Quoted 10 lines
Quoted 2 lines

Michal, do you feel strongly about having a dedicated kfunc vs the direct memory read?

The reason I wanted this an explicit API is that oom states are quite internal part of the oom synchronization. And I would really like to have that completely transparent for oom policies. In other words I do not want to touch all potential oom policies or break them in the worst case just because we need to change this. So while a trivial interface now (and hopefully for a long time) it is really an internal thing.

Quoted 2 lines

Do I insist? No, I do not but I would like to hear why this is a bad idea.

Quoted 7 lines

It's a bad idea, since it doesn't address your goal. bpf prog can access task->signal->oom_mm without kfunc just fine and it will be doing so because performance matters and static inline bool foo(task)
{
return task->signal->oom_mm;
}

Quoted 3 lines

OK, so my understanding was that BPF can only use exported functionality. If those progs can access whatever they get a pointer for and than traverse down the road then this is moot from a large part.

bpf could access all kernel internals from day one 10 years ago. We made it more ergonomic over the years.

Quoted 9 lines
Quoted 6 lines

If anything changes and, say, oom_mm will get renamed whether it was kfunc or not doesn't change much. progs will adopt to a new way easily with CORE. kfuncs can also be renamed/deleted, etc. You're thinking about kfuncs as a stable api. It's definitely not. It's not a layer of isolation either. kfuncs are necessary only for the cases where bpf prog cannot do it on its own.

It is obviously not clear to me where that line is for BPF progs. Where is this documented?

See Documentation/bpf/kfuncs.rst Especially "kfunc lifecycle expectations" section.

author maintainer
#50 lore raw

PATCH v3 11/17 bpf: selftests: introduce read_cgroup_file() helper

In reply to Roman Gushchin

Implement read_cgroup_file() helper to read from cgroup control files, e.g. statistics.

---
 tools/testing/selftests/bpf/cgroup_helpers.c | 45 ++++++++++++++++++++
 tools/testing/selftests/bpf/cgroup_helpers.h |  3 ++
 2 files changed, 48 insertions(+)
tools/testing/selftests/bpf/cgroup_helpers.c +45 -0
index 20cede4db3ce..fc5f22409ce5 100644
@@ -126,6 +126,51 @@ int enable_controllers(const char *relative_path, const char *controllers)
 	return __enable_controllers(cgroup_path, controllers);
 }
 
+static ssize_t __read_cgroup_file(const char *cgroup_path, const char *file,
+				 char *buf, size_t size)
+{
+	char file_path[PATH_MAX + 1];
+	ssize_t ret;
+	int fd;
+
+	snprintf(file_path, sizeof(file_path), "%s/%s", cgroup_path, file);
+	fd = open(file_path, O_RDONLY);
+	if (fd < 0) {
+		log_err("Opening %s", file_path);
+		return -1;
+	}
+
+	ret = read(fd, buf, size);
+	if (ret < 0) {
+		close(fd);
+		log_err("Reading %s", file_path);
+		return -1;
+	}
+
+	close(fd);
+	return ret;
+}
+
+/**
+ * read_cgroup_file() - Read from a cgroup file
+ * @relative_path: The cgroup path, relative to the workdir
+ * @file: The name of the file in cgroupfs to read from
+ * @buf: Buffer to read from the file
+ * @size: Size of the buffer
+ *
+ * Read from a file in the given cgroup's directory.
+ *
+ * If successful, the number of read bytes is returned.
+ */
+ssize_t read_cgroup_file(const char *relative_path, const char *file,
+			 char *buf, size_t size)
+{
+	char cgroup_path[PATH_MAX - 24];
+
+	format_cgroup_path(cgroup_path, relative_path);
+	return __read_cgroup_file(cgroup_path, file, buf, size);
+}
+
 static int __write_cgroup_file(const char *cgroup_path, const char *file,
 			       const char *buf)
 {
tools/testing/selftests/bpf/cgroup_helpers.h +3 -0
index 3857304be874..66a08b64838b 100644
@@ -4,6 +4,7 @@
 
 #include <errno.h>
 #include <string.h>
+#include <sys/types.h>
 
 #define clean_errno() (errno == 0 ? "None" : strerror(errno))
 #define log_err(MSG, ...) fprintf(stderr, "(%s:%d: errno: %s) " MSG "\n", \
@@ -11,6 +12,8 @@
 
 /* cgroupv2 related */
 int enable_controllers(const char *relative_path, const char *controllers);
+ssize_t read_cgroup_file(const char *relative_path, const char *file,
+			char *buf, size_t size);
 int write_cgroup_file(const char *relative_path, const char *file,
 		      const char *buf);
 int write_cgroup_file_parent(const char *relative_path, const char *file,
Signature · 1 line
2.52.0
#51 lore raw

Re: [PATCH bpf-next v3 11/17] bpf: selftests: introduce read_cgroup_file() helper

This isn't a bug, but should this say "Buffer to store data read from the file" or "Buffer to read into"? The current wording suggests data is read from the buffer, but the code reads from the file…
Quoted 4 lines
tools/testing/selftests/bpf/cgroup_helpers.c +0 -0
index 20cede4db3ce..fc5f22409ce5 100644

[ ... ]

Quoted 5 lines

+/**
+ * read_cgroup_file() - Read from a cgroup file
+ * @relative_path: The cgroup path, relative to the workdir
+ * @file: The name of the file in cgroupfs to read from
+ * @buf: Buffer to read from the file

          ^^^^^^^^^^^^^^^^^^^^^^^^

This isn't a bug, but should this say "Buffer to store data read from the file" or "Buffer to read into"? The current wording suggests data is read from the buffer, but the code reads from the file into the buffer. Compare with write_cgroup_file() which says "Buffer to write to the file" - that phrasing works for writes but the inverse doesn't quite work for reads.

Quoted 6 lines

+ * @size: Size of the buffer
+ *
+ * Read from a file in the given cgroup's directory.
+ *
+ * If successful, the number of read bytes is returned.
+ */

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/21382620878

AI-authorship-score: low
AI-authorship-explanation: The code closely follows existing patterns in the file with a minor documentation error typical of copy-paste adaptation, suggesting human authorship.
issues-found: 1
issue-severity-score: low
issue-severity-explanation: Minor documentation error in test code where @buf description says the opposite of what the code does.

author maintainer
#52 lore raw

PATCH v3 12/17 bpf: selftests: BPF OOM struct ops test

In reply to Roman Gushchin

Implement a kselftest for the OOM handling functionality.

The OOM handling policy which is implemented in BPF is to kill all tasks belonging to the biggest leaf cgroup, which doesn't contain unkillable tasks (tasks with oom_score_adj set to -1000). Pagecache size is excluded from the accounting.

The test creates a hierarchy of memory cgroups, causes an OOM at the top level, checks that the expected process is killed and verifies the memcg's oom statistics.

The same BPF OOM policy is attached to a memory cgroup and system-wide. In the first case the program does nothing and returns false, so it's executed the second time, when it properly handles the OOM.

---
 .../selftests/bpf/prog_tests/test_oom.c       | 256 ++++++++++++++++++
 tools/testing/selftests/bpf/progs/test_oom.c  | 111 ++++++++
 2 files changed, 367 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_oom.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_oom.c
tools/testing/selftests/bpf/prog_tests/test_oom.c new file +256 -0
new file mode 100644
index 000000000000..a1eadbe1ae83
@@ -0,0 +1,256 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <test_progs.h>
+#include <bpf/btf.h>
+#include <bpf/bpf.h>
+
+#include "cgroup_helpers.h"
+#include "test_oom.skel.h"
+
+struct cgroup_desc {
+	const char *path;
+	int fd;
+	unsigned long long id;
+	int pid;
+	size_t target;
+	size_t max;
+	int oom_score_adj;
+	bool victim;
+};
+
+#define MB (1024 * 1024)
+#define OOM_SCORE_ADJ_MIN	(-1000)
+#define OOM_SCORE_ADJ_MAX	1000
+
+static struct cgroup_desc cgroups[] = {
+	{ .path = "/oom_test", .max = 80 * MB},
+	{ .path = "/oom_test/cg1", .target = 10 * MB,
+	  .oom_score_adj = OOM_SCORE_ADJ_MAX },
+	{ .path = "/oom_test/cg2", .target = 40 * MB,
+	  .oom_score_adj = OOM_SCORE_ADJ_MIN },
+	{ .path = "/oom_test/cg3" },
+	{ .path = "/oom_test/cg3/cg4", .target = 30 * MB,
+	  .victim = true },
+	{ .path = "/oom_test/cg3/cg5", .target = 20 * MB },
+};
+
+static int spawn_task(struct cgroup_desc *desc)
+{
+	char *ptr;
+	int pid;
+
+	pid = fork();
+	if (pid < 0)
+		return pid;
+
+	if (pid > 0) {
+		/* parent */
+		desc->pid = pid;
+		return 0;
+	}
+
+	/* child */
+	if (desc->oom_score_adj) {
+		char buf[64];
+		int fd = open("/proc/self/oom_score_adj", O_WRONLY);
+
+		if (fd < 0)
+			return -1;
+
+		snprintf(buf, sizeof(buf), "%d", desc->oom_score_adj);
+		write(fd, buf, sizeof(buf));
+		close(fd);
+	}
+
+	ptr = (char *)malloc(desc->target);
+	if (!ptr)
+		return -ENOMEM;
+
+	memset(ptr, 'a', desc->target);
+
+	while (1)
+		sleep(1000);
+
+	return 0;
+}
+
+static void setup_environment(void)
+{
+	int i, err;
+
+	err = setup_cgroup_environment();
+	if (!ASSERT_OK(err, "setup_cgroup_environment"))
+		goto cleanup;
+
+	for (i = 0; i < ARRAY_SIZE(cgroups); i++) {
+		cgroups[i].fd = create_and_get_cgroup(cgroups[i].path);
+		if (!ASSERT_GE(cgroups[i].fd, 0, "create_and_get_cgroup"))
+			goto cleanup;
+
+		cgroups[i].id = get_cgroup_id(cgroups[i].path);
+		if (!ASSERT_GT(cgroups[i].id, 0, "get_cgroup_id"))
+			goto cleanup;
+
+		/* Freeze the top-level cgroup */
+		if (i == 0) {
+			/* Freeze the top-level cgroup */
+			err = write_cgroup_file(cgroups[i].path, "cgroup.freeze", "1");
+			if (!ASSERT_OK(err, "freeze cgroup"))
+				goto cleanup;
+		}
+
+		/* Recursively enable the memory controller */
+		if (!cgroups[i].target) {
+
+			err = write_cgroup_file(cgroups[i].path, "cgroup.subtree_control",
+						"+memory");
+			if (!ASSERT_OK(err, "enable memory controller"))
+				goto cleanup;
+		}
+
+		/* Set memory.max */
+		if (cgroups[i].max) {
+			char buf[256];
+
+			snprintf(buf, sizeof(buf), "%lu", cgroups[i].max);
+			err = write_cgroup_file(cgroups[i].path, "memory.max", buf);
+			if (!ASSERT_OK(err, "set memory.max"))
+				goto cleanup;
+
+			snprintf(buf, sizeof(buf), "0");
+			write_cgroup_file(cgroups[i].path, "memory.swap.max", buf);
+
+		}
+
+		/* Spawn tasks creating memory pressure */
+		if (cgroups[i].target) {
+			char buf[256];
+
+			err = spawn_task(&cgroups[i]);
+			if (!ASSERT_OK(err, "spawn task"))
+				goto cleanup;
+
+			snprintf(buf, sizeof(buf), "%d", cgroups[i].pid);
+			err = write_cgroup_file(cgroups[i].path, "cgroup.procs", buf);
+			if (!ASSERT_OK(err, "put child into a cgroup"))
+				goto cleanup;
+		}
+	}
+
+	return;
+
+cleanup:
+	cleanup_cgroup_environment();
+
+	// TODO return an error?
+}
+
+static int run_and_wait_for_oom(void)
+{
+	int ret = -1;
+	bool first = true;
+	char buf[4096] = {};
+	size_t size;
+
+	/* Unfreeze the top-level cgroup */
+	ret = write_cgroup_file(cgroups[0].path, "cgroup.freeze", "0");
+	if (!ASSERT_OK(ret, "freeze cgroup"))
+		return -1;
+
+	for (;;) {
+		int i, status;
+		pid_t pid = wait(&status);
+
+		if (pid == -1) {
+			if (errno == EINTR)
+				continue;
+			/* ECHILD */
+			break;
+		}
+
+		if (!first)
+			continue;
+
+		first = false;
+
+		/* Check which process was terminated first */
+		for (i = 0; i < ARRAY_SIZE(cgroups); i++) {
+			if (!ASSERT_OK(cgroups[i].victim !=
+				       (pid == cgroups[i].pid),
+				       "correct process was killed")) {
+				ret = -1;
+				break;
+			}
+
+			if (!cgroups[i].victim)
+				continue;
+
+			/* Check the memcg oom counter */
+			size = read_cgroup_file(cgroups[i].path,
+						"memory.events",
+						buf, sizeof(buf));
+			if (!ASSERT_OK(size <= 0, "read memory.events")) {
+				ret = -1;
+				break;
+			}
+
+			if (!ASSERT_OK(strstr(buf, "oom_kill 1") == NULL,
+				       "oom_kill count check")) {
+				ret = -1;
+				break;
+			}
+		}
+
+		/* Kill all remaining tasks */
+		for (i = 0; i < ARRAY_SIZE(cgroups); i++)
+			if (cgroups[i].pid && cgroups[i].pid != pid)
+				kill(cgroups[i].pid, SIGKILL);
+	}
+
+	return ret;
+}
+
+void test_oom(void)
+{
+	DECLARE_LIBBPF_OPTS(bpf_struct_ops_opts, opts);
+	struct bpf_link *link1 = NULL, *link2 = NULL;
+	struct test_oom *skel;
+	int err = 0;
+
+	setup_environment();
+
+	skel = test_oom__open_and_load();
+	if (!skel) {
+		err = -errno;
+		CHECK_FAIL(err);
+		goto cleanup;
+	}
+
+	opts.flags = BPF_F_CGROUP_FD;
+	opts.target_fd = cgroups[0].fd;
+	link1 = bpf_map__attach_struct_ops_opts(skel->maps.test_bpf_oom, &opts);
+	if (!link1) {
+		err = -errno;
+		CHECK_FAIL(err);
+		goto cleanup;
+	}
+
+	opts.target_fd = get_root_cgroup();
+	link2 = bpf_map__attach_struct_ops_opts(skel->maps.test_bpf_oom, &opts);
+	if (!link2) {
+		err = -errno;
+		CHECK_FAIL(err);
+		goto cleanup;
+	}
+
+	/* Unfreeze all child tasks and create the memory pressure */
+	err = run_and_wait_for_oom();
+	CHECK_FAIL(err);
+
+cleanup:
+	bpf_link__destroy(link1);
+	bpf_link__destroy(link2);
+	write_cgroup_file(cgroups[0].path, "cgroup.kill", "1");
+	write_cgroup_file(cgroups[0].path, "cgroup.freeze", "0");
+	cleanup_cgroup_environment();
+	test_oom__destroy(skel);
+}
tools/testing/selftests/bpf/progs/test_oom.c new file +111 -0

111 lines not on this page · raw message

Signature · 1 line
2.52.0
author maintainer
#53 lore raw

PATCH v3 13/17 sched: psi: add a trace point to psi_avgs_work()

In reply to Roman Gushchin

Add a trace point to psi_avgs_work().

It can be used to attach a bpf handler which can monitor PSI values system-wide or for specific cgroup(s) and potentially perform some actions, e.g. declare an OOM.

---
 include/trace/events/psi.h | 27 +++++++++++++++++++++++++++
 kernel/sched/psi.c         |  6 ++++++
 2 files changed, 33 insertions(+)
 create mode 100644 include/trace/events/psi.h
include/trace/events/psi.h new file +27 -0
new file mode 100644
index 000000000000..57c46de18616
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM psi
+
+#if !defined(_TRACE_PSI_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_PSI_H
+
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(psi_avgs_work,
+	TP_PROTO(struct psi_group *group),
+	TP_ARGS(group),
+	TP_STRUCT__entry(
+		__field(struct psi_group *, group)
+	),
+
+	TP_fast_assign(
+		__entry->group = group;
+	),
+
+	TP_printk("group=%p", __entry->group)
+);
+
+#endif /* _TRACE_PSI_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
kernel/sched/psi.c +6 -0
index 59fdb7ebbf22..72757ba2ed96 100644
@@ -141,6 +141,10 @@
 #include <linux/psi.h>
 #include "sched.h"
 
+#define CREATE_TRACE_POINTS
+#include <trace/events/psi.h>
+#undef CREATE_TRACE_POINTS
+
 static int psi_bug __read_mostly;
 
 DEFINE_STATIC_KEY_FALSE(psi_disabled);
@@ -607,6 +611,8 @@ static void psi_avgs_work(struct work_struct *work)
 				group->avg_next_update - now) + 1);
 	}
 
+	trace_psi_avgs_work(group);
+
 	mutex_unlock(&group->avgs_lock);
 }
 
Signature · 1 line
2.52.0
author maintainer
#54 lore raw

PATCH v3 14/17 sched: psi: add cgroup_id field to psi_group structure

In reply to Roman Gushchin

To allow a more efficient filtering of cgroups in the psi work tracepoint handler, let's add a u64 cgroup_id field to the psi_group structure. For system PSI, 0 will be used.

---
 include/linux/psi_types.h | 4 ++++
 kernel/sched/psi.c        | 1 +
 2 files changed, 5 insertions(+)
include/linux/psi_types.h +4 -0
index dd10c22299ab..749a08d48abd 100644
@@ -159,6 +159,10 @@ struct psi_trigger {
 
 struct psi_group {
 	struct psi_group *parent;
+
+	/* Cgroup id for cgroup PSI, 0 for system PSI */
+	u64 cgroup_id;
+
 	bool enabled;
 
 	/* Protects data used by the aggregator */
kernel/sched/psi.c +1 -0
index 72757ba2ed96..cf1ec4dc242b 100644
@@ -1124,6 +1124,7 @@ int psi_cgroup_alloc(struct cgroup *cgroup)
 	if (!cgroup->psi)
 		return -ENOMEM;
 
+	cgroup->psi->cgroup_id = cgroup_id(cgroup);
 	cgroup->psi->pcpu = alloc_percpu(struct psi_group_cpu);
 	if (!cgroup->psi->pcpu) {
 		kfree(cgroup->psi);
Signature · 1 line
2.52.0
author maintainer
#55 lore raw

PATCH v3 15/17 bpf: allow calling bpf_out_of_memory() from a PSI tracepoint

In reply to Roman Gushchin

Allow calling bpf_out_of_memory() from a PSI tracepoint to enable PSI-based OOM killer policies.

---
 mm/oom_kill.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
mm/oom_kill.c +9 -1
index 53f9f9674658..276cf8a34449 100644
@@ -1421,6 +1421,13 @@ BTF_KFUNCS_START(bpf_declare_oom_kfuncs)
 BTF_ID_FLAGS(func, bpf_out_of_memory, KF_SLEEPABLE)
 BTF_KFUNCS_END(bpf_declare_oom_kfuncs)
 
+BTF_ID_LIST(bpf_oom_trace_ids)
+#ifdef CONFIG_PSI
+BTF_ID(typedef, btf_trace_psi_avgs_work)
+#else
+BTF_ID_UNUSED
+#endif
+
 static int bpf_declare_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_id)
 {
 	if (!btf_id_set8_contains(&bpf_declare_oom_kfuncs, kfunc_id))
@@ -1430,7 +1437,8 @@ static int bpf_declare_oom_kfunc_filter(const struct bpf_prog *prog, u32 kfunc_i
 	    prog->aux->attach_btf_id == bpf_oom_ops_ids[0])
 		return -EACCES;
 
-	if (prog->type == BPF_PROG_TYPE_TRACING)
+	if (prog->type == BPF_PROG_TYPE_TRACING &&
+	    prog->aux->attach_btf_id != bpf_oom_trace_ids[0])
 		return -EACCES;
 
 	return 0;
Signature · 1 line
2.52.0
maintainer
#56 lore raw
In reply to Roman Gushchin
Quoted 12 lines · Roman Gushchin

This patchset adds an ability to customize the out of memory handling using bpf.

It focuses on two parts:
1) OOM handling policy,
2) PSI-based OOM invocation.

The idea to use bpf for customizing the OOM handling is not new, but unlike the previous proposal [1], which augmented the existing task ranking policy, this one tries to be as generic as possible and leverage the full power of the modern bpf.

Quoted 5 lines

It provides a generic interface which is called before the existing OOM killer code and allows implementing any policy, e.g. picking a victim task or memory cgroup or potentially even releasing memory in other ways, e.g. deleting tmpfs files (the last one might require some additional but relatively simple changes).

Are you planning to write any highlevel documentation on how to use the existing infrastructure to implement proper/correct OOM handlers with these generic interfaces?

Signature · 2 lines
Michal Hocko
SUSE Labs
author maintainer
#57 lore raw
Quoted 19 lines · Michal Hocko
Quoted 12 lines · Roman Gushchin

This patchset adds an ability to customize the out of memory handling using bpf.

It focuses on two parts:
1) OOM handling policy,
2) PSI-based OOM invocation.

The idea to use bpf for customizing the OOM handling is not new, but unlike the previous proposal [1], which augmented the existing task ranking policy, this one tries to be as generic as possible and leverage the full power of the modern bpf.

Quoted 5 lines

It provides a generic interface which is called before the existing OOM killer code and allows implementing any policy, e.g. picking a victim task or memory cgroup or potentially even releasing memory in other ways, e.g. deleting tmpfs files (the last one might require some additional but relatively simple changes).

Quoted 3 lines

Are you planning to write any highlevel documentation on how to use the existing infrastructure to implement proper/correct OOM handlers with these generic interfaces?

What do you expect from such a document, can you, please, elaborate? I'm asking because the main promise of bpf is to provide some sort of a safe playground, so anyone can experiment with writing their bpf implementations (like sched_ext schedulers or bpf oom policies) with minimum risk. Yes, it might work sub-optimally and kill too many tasks, but it won't crash or deadlock the system.
So in way I don't want to prescribe the "right way" of writing oom handler, but it totally makes sense to provide an example.

As of now the best way to get an example of a bpf handler is to look into the commit "[PATCH bpf-next v3 12/17] bpf: selftests: BPF OOM struct ops test".

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Thanks!

maintainer
#58 lore raw
Quoted 21 lines · Roman Gushchin
Quoted 19 lines · Michal Hocko
Quoted 17 lines · Roman Gushchin

This patchset adds an ability to customize the out of memory handling using bpf.

It focuses on two parts:
1) OOM handling policy,
2) PSI-based OOM invocation.

The idea to use bpf for customizing the OOM handling is not new, but unlike the previous proposal [1], which augmented the existing task ranking policy, this one tries to be as generic as possible and leverage the full power of the modern bpf.

It provides a generic interface which is called before the existing OOM killer code and allows implementing any policy, e.g. picking a victim task or memory cgroup or potentially even releasing memory in other ways, e.g. deleting tmpfs files (the last one might require some additional but relatively simple changes).

Quoted 5 lines
Quoted 3 lines

Are you planning to write any highlevel documentation on how to use the existing infrastructure to implement proper/correct OOM handlers with these generic interfaces?

What do you expect from such a document, can you, please, elaborate?

Sure. Essentially an expected structure of the handler. What is the API it can use, what is has to do and what it must not do. Essentially a single place you can read and get enough information to start developing your oom handler.

Quoted 11 lines

I'm asking because the main promise of bpf is to provide some sort of a safe playground, so anyone can experiment with writing their bpf implementations (like sched_ext schedulers or bpf oom policies) with minimum risk. Yes, it might work sub-optimally and kill too many tasks, but it won't crash or deadlock the system.
So in way I don't want to prescribe the "right way" of writing oom handler, but it totally makes sense to provide an example.

As of now the best way to get an example of a bpf handler is to look into the commit "[PATCH bpf-next v3 12/17] bpf: selftests: BPF OOM struct ops test".

Examples are really great but having a central place to document available API is much more helpful IMHO. The generally scattered nature of BPF hooks makes it really hard to even know what is available to oom handlers to use.

Quoted 6 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Signature · 2 lines
Michal Hocko
SUSE Labs
#59 lore raw
Quoted 9 lines · Michal Hocko
Quoted 8 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Quoted 5 lines

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Production ready bpf-oom program(s) must be part of this set. We've seen enough attempts to add bpf st_ops in various parts of the kernel without providing realistic bpf progs that will drive those hooks. It's great to have flexibility and people need to have a freedom to develop their own bpf-oom policy, but the author of the patch set who's advocating for the new bpf hooks must provide their real production progs and share their real use case with the community.
It's not cool to hide it.
In that sense enabling auto-loading without requiring an end user to install the toolchain and build bpf programs/rust/whatnot is necessary too.
bpf-oom can be a self contained part of vmlinux binary. We already have a mechanism to do that.
This way the end user doesn't need to be a bpf expert, doesn't need to install clang, build the tools, etc.
They can just enable fancy new bpf-oom policy and see whether it's helping their apps or not while knowing nothing about bpf.

author maintainer
#60 lore raw
Quoted 16 lines · Alexei Starovoitov
Quoted 9 lines · Michal Hocko
Quoted 8 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Quoted 5 lines

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Quoted 9 lines

Production ready bpf-oom program(s) must be part of this set. We've seen enough attempts to add bpf st_ops in various parts of the kernel without providing realistic bpf progs that will drive those hooks. It's great to have flexibility and people need to have a freedom to develop their own bpf-oom policy, but the author of the patch set who's advocating for the new bpf hooks must provide their real production progs and share their real use case with the community.
It's not cool to hide it.

In my case it's not about hiding, it's a chicken and egg problem:
the upstream first model contradicts with the idea to include the production results into the patchset. In other words, I want to settle down the interface before shipping something to prod.

I guess the compromise here is to initially include a bpf oom policy inspired by what systemd-oomd does and what is proven to work for a broad range of users. Policies suited for large datacenters can be added later, but also their generic usefulness might be limited by the need of proprietary userspace orchestration engines.

Quoted 9 lines

In that sense enabling auto-loading without requiring an end user to install the toolchain and build bpf programs/rust/whatnot is necessary too.
bpf-oom can be a self contained part of vmlinux binary. We already have a mechanism to do that.
This way the end user doesn't need to be a bpf expert, doesn't need to install clang, build the tools, etc.
They can just enable fancy new bpf-oom policy and see whether it's helping their apps or not while knowing nothing about bpf.

Fully agree here. Will implement in v4.

Thanks!

#61 lore raw
Quoted 29 lines · Roman Gushchin
Quoted 16 lines · Alexei Starovoitov
Quoted 9 lines · Michal Hocko
Quoted 8 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Quoted 5 lines

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Quoted 9 lines

Production ready bpf-oom program(s) must be part of this set. We've seen enough attempts to add bpf st_ops in various parts of the kernel without providing realistic bpf progs that will drive those hooks. It's great to have flexibility and people need to have a freedom to develop their own bpf-oom policy, but the author of the patch set who's advocating for the new bpf hooks must provide their real production progs and share their real use case with the community.
It's not cool to hide it.

Quoted 8 lines

In my case it's not about hiding, it's a chicken and egg problem:
the upstream first model contradicts with the idea to include the production results into the patchset. In other words, I want to settle down the interface before shipping something to prod.

I guess the compromise here is to initially include a bpf oom policy inspired by what systemd-oomd does and what is proven to work for a broad range of users.

Works for me.

Quoted 3 lines

Policies suited for large datacenters can be
added later, but also their generic usefulness might be limited by the need of proprietary userspace orchestration engines.

Agree. That's the flexibility part that makes the whole thing worth while and the reason to do such oom policy as bpf progs. But something tangible and useful needs to be there from day one. systmed-oomd-like sounds very reasonable.

#62 lore raw
In reply to Alexei Starovoitov
Quoted 26 lines · Alexei Starovoitov
Quoted 9 lines · Michal Hocko
Quoted 8 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Quoted 5 lines

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Production ready bpf-oom program(s) must be part of this set. We've seen enough attempts to add bpf st_ops in various parts of the kernel without providing realistic bpf progs that will drive those hooks. It's great to have flexibility and people need to have a freedom to develop their own bpf-oom policy, but the author of the patch set who's advocating for the new bpf hooks must provide their real production progs and share their real use case with the community.
It's not cool to hide it.
In that sense enabling auto-loading without requiring an end user

Quoted 8 lines

to install the toolchain and build bpf programs/rust/whatnot is necessary too.
bpf-oom can be a self contained part of vmlinux binary. We already have a mechanism to do that.
This way the end user doesn't need to be a bpf expert, doesn't need to install clang, build the tools, etc.
They can just enable fancy new bpf-oom policy and see whether it's helping their apps or not while knowing nothing about bpf.

For the auto-loading capability you speak of here, I'm currently interpreting it as being some form of conceptually similar extension to the BPF preload functionality. Have I understood this correctly? If so, I feel as though something like this would be a completely independent stream of work, orthogonal to this BPF OOM feature, right? Or, is that you'd like this new auto-loading capability completed as a hard prerequisite before pulling in the BPF OOM feature?

#63 lore raw
Quoted 37 lines · Matt Bobrowski
Quoted 26 lines · Alexei Starovoitov
Quoted 9 lines · Michal Hocko
Quoted 8 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Quoted 5 lines

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Production ready bpf-oom program(s) must be part of this set. We've seen enough attempts to add bpf st_ops in various parts of the kernel without providing realistic bpf progs that will drive those hooks. It's great to have flexibility and people need to have a freedom to develop their own bpf-oom policy, but the author of the patch set who's advocating for the new bpf hooks must provide their real production progs and share their real use case with the community.
It's not cool to hide it.
In that sense enabling auto-loading without requiring an end user

Quoted 8 lines

to install the toolchain and build bpf programs/rust/whatnot is necessary too.
bpf-oom can be a self contained part of vmlinux binary. We already have a mechanism to do that.
This way the end user doesn't need to be a bpf expert, doesn't need to install clang, build the tools, etc.
They can just enable fancy new bpf-oom policy and see whether it's helping their apps or not while knowing nothing about bpf.

Quoted 7 lines

For the auto-loading capability you speak of here, I'm currently interpreting it as being some form of conceptually similar extension to the BPF preload functionality. Have I understood this correctly? If so, I feel as though something like this would be a completely independent stream of work, orthogonal to this BPF OOM feature, right? Or, is that you'd like this new auto-loading capability completed as a hard prerequisite before pulling in the BPF OOM feature?

It's not a hard prerequisite, but it has to be thought through. bpf side is ready today. bpf preload is an example of it. The oom side needs to design an interface to do it. sysctl to enable builtin bpf-oom policy is probably too rigid. Maybe a file in cgroupfs? Writing a name of bpf-oom policy would trigger load and attach to that cgroup.
Or you can plug it exactly like bpf preload:
when bpffs is mounted all builtin bpf progs get loaded and create ".debug" files in bpffs.

I recall we discussed an ability to create files in bpffs from tracepoints. This way bpffs can replicate cgroupfs directory structure without user space involvement. New cgroup -> new directory in cgroupfs -> tracepoint -> bpf prog -> new directory in bpffs -> create "enable_bpf_oom.debug" file in there.
Writing to that file we trigger bpf prog that will attach bpf-oom prog to that cgroup.
Could be any combination of the above or something else, but needs to be designed and agreed upon.
Otherwise, I'm afraid, we will have bpf-oom progs in selftests and users who want to experiment with it would need kernel source code, clang, etc to try it. We need to lower the barrier to use it.

#64 lore raw
Quoted 56 lines · Alexei Starovoitov
Quoted 37 lines · Matt Bobrowski
Quoted 26 lines · Alexei Starovoitov
Quoted 9 lines · Michal Hocko
Quoted 8 lines

Another viable idea (also suggested by Andrew Morton) is to develop a production ready memcg-aware OOM killer in BPF, put the source code into the kernel tree and make it loadable by default (obviously under a config option). Myself or one of my colleagues will try to explore it a bit later: the tricky part is this by-default loading because there are no existing precedents.

Quoted 5 lines

It certainly makes sense to have trusted implementation of a commonly requested oom policy that we couldn't implement due to specific nature that doesn't really apply to many users. And have that in the tree. I am not thrilled about auto-loading because this could be easily done by a simple tooling.

Production ready bpf-oom program(s) must be part of this set. We've seen enough attempts to add bpf st_ops in various parts of the kernel without providing realistic bpf progs that will drive those hooks. It's great to have flexibility and people need to have a freedom to develop their own bpf-oom policy, but the author of the patch set who's advocating for the new bpf hooks must provide their real production progs and share their real use case with the community.
It's not cool to hide it.
In that sense enabling auto-loading without requiring an end user

Quoted 8 lines

to install the toolchain and build bpf programs/rust/whatnot is necessary too.
bpf-oom can be a self contained part of vmlinux binary. We already have a mechanism to do that.
This way the end user doesn't need to be a bpf expert, doesn't need to install clang, build the tools, etc.
They can just enable fancy new bpf-oom policy and see whether it's helping their apps or not while knowing nothing about bpf.

Quoted 7 lines

For the auto-loading capability you speak of here, I'm currently interpreting it as being some form of conceptually similar extension to the BPF preload functionality. Have I understood this correctly? If so, I feel as though something like this would be a completely independent stream of work, orthogonal to this BPF OOM feature, right? Or, is that you'd like this new auto-loading capability completed as a hard prerequisite before pulling in the BPF OOM feature?

It's not a hard prerequisite, but it has to be thought through. bpf side is ready today. bpf preload is an example of it. The oom side needs to design an interface to do it. sysctl to enable builtin bpf-oom policy is probably too rigid. Maybe a file in cgroupfs? Writing a name of bpf-oom policy would trigger load and attach to that cgroup.
Or you can plug it exactly like bpf preload:
when bpffs is mounted all builtin bpf progs get loaded and create ".debug" files in bpffs.

Quoted 12 lines

I recall we discussed an ability to create files in bpffs from tracepoints. This way bpffs can replicate cgroupfs directory structure without user space involvement. New cgroup -> new directory in cgroupfs -> tracepoint -> bpf prog -> new directory in bpffs -> create "enable_bpf_oom.debug" file in there.
Writing to that file we trigger bpf prog that will attach bpf-oom prog to that cgroup.
Could be any combination of the above or something else, but needs to be designed and agreed upon.
Otherwise, I'm afraid, we will have bpf-oom progs in selftests and users who want to experiment with it would need kernel source code, clang, etc to try it. We need to lower the barrier to use it.

OK, I see what you're saying here. I'll have a chat to Roman about this and see what his thoughts are on it.