bpf: Handle MEM_RCU type properly
Subsystems (5)
v2 cover bpf: Handle MEM_RCU type properly
Patch set [1] added rcu support for bpf programs. In [1], a rcu pointer is considered to be trusted and not null. This is actually not true in some cases. The rcu pointer could be null, and for non-null rcu pointer, it may have reference count of 0. This small patch set fixed this problem. Patch 1 is the kernel fix. Patch 2 adjusted selftests properly. Patch 3 added documentation for newly-introduced KF_RCU flag.
[1] https://lore.kernel.org/all/[email protected]/
Changelogs:
v1 -> v2:
- rcu ptr could be NULL.
- non_null_rcu_ptr->rcu_field can be marked as MEM_RCU as well.
- Adjust the code to avoid existing error message change.
Yonghong Song (3):
bpf: Handle MEM_RCU type properly
selftests/bpf: Fix rcu_read_lock test with new MEM_RCU semantics docs/bpf: Add KF_RCU documentation
Documentation/bpf/kfuncs.rst | 9 +++ include/linux/bpf_verifier.h | 2 +- include/linux/btf.h | 1 + kernel/bpf/helpers.c | 14 +++++ kernel/bpf/verifier.c | 45 ++++++++++----- .../selftests/bpf/progs/rcu_read_lock.c | 55 +++++++++++++++---- 6 files changed, 102 insertions(+), 24 deletions(-)
Signature · 1 line
2.30.2
PATCH v2 1/3 bpf: Handle MEM_RCU type properly
Commit 9bb00b2895cb ("bpf: Add kfunc bpf_rcu_read_lock/unlock()") introduced MEM_RCU and bpf_rcu_read_lock/unlock() support. In that commit, a rcu pointer is tagged with both MEM_RCU and PTR_TRUSTED so that it can be passed into kfuncs or helpers as an argument.
Martin raised a good question in [1] such that the rcu pointer, although being able to accessing the object, might have reference count of 0. This might cause a problem if the rcu pointer is passed to a kfunc which expects trusted arguments where ref count should be greater than 0.
This patch makes the following changes related to MEM_RCU pointer:
- MEM_RCU pointer might be NULL (PTR_MAYBE_NULL).
- Introduce KF_RCU so MEM_RCU ptr can be acquired with a KF_RCU tagged kfunc which assumes ref count of rcu ptr could be zero.
- For mem access 'b = ptr->a', say 'ptr' is a MEM_RCU ptr, and 'a' is tagged with __rcu as well. Let us mark 'b' as MEM_RCU | PTR_MAYBE_NULL.
[1] https://lore.kernel.org/bpf/[email protected]/
--- include/linux/bpf_verifier.h | 2 +- include/linux/btf.h | 1 + kernel/bpf/helpers.c | 14 +++++++++++ kernel/bpf/verifier.c | 45 +++++++++++++++++++++++++----------- 4 files changed, 48 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +1 -1
@@ -682,7 +682,7 @@ static inline bool bpf_prog_check_recur(const struct bpf_prog *prog) } } -#define BPF_REG_TRUSTED_MODIFIERS (MEM_ALLOC | MEM_RCU | PTR_TRUSTED) +#define BPF_REG_TRUSTED_MODIFIERS (MEM_ALLOC | PTR_TRUSTED)static inline bool bpf_type_has_unsafe_modifiers (u32 type) {
include/linux/btf.h +1 -0
@@ -70,6 +70,7 @@#define KF_TRUSTED_ARGS (1 << 4)/* kfunc only takes trusted pointer arguments */ #define KF_SLEEPABLE (1 << 5)/* kfunc may sleep */ #define KF_DESTRUCTIVE (1 << 6)/* kfunc performs destructive actions */ +#define KF_RCU (1 << 7)/* kfunc only takes rcu pointer arguments */ /* * Return the name of the passed struct, if exists, or halt the build if for
kernel/bpf/helpers.c +14 -0
@@ -1837,6 +1837,19 @@ struct task_struct *bpf_task_acquire(struct task_struct *p)return p; } +/** +* bpf_task_acquire_not_zero - Acquire a reference to a rcu task object. A task +* acquired by this kfunc which is not stored in a map as a kptr, must be +* released by calling bpf_task_release(). +* @p: The task on which a reference is being acquired. +*/ +struct task_struct *bpf_task_acquire_not_zero (struct task_struct *p) +{ +if (!refcount_inc_not_zero (&p->rcu_users)) +return NULL ; +return p; +} +/** * bpf_task_kptr_get - Acquire a reference on a struct task_struct kptr. A task * kptr acquired by this kfunc which is not subsequently stored in a map, must
@@ -2013,6 +2026,7 @@ BTF_ID_FLAGS(func, bpf_list_push_back) BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_task_acquire_not_zero, KF_ACQUIRE | KF_RCU | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_kptr_get, KF_ACQUIRE | KF_KPTR_GET | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE)#ifdef CONFIG_CGROUPS
kernel/bpf/verifier.c +32 -13
113 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH v2 2/3 selftests/bpf: Fix rcu_read_lock test with new MEM_RCU semantics
In reply to Yonghong SongAdd MEM_RCU pointer null checking for related tests. Also modified task_acquire test so it takes a rcu ptr 'ptr' where 'ptr = rcu_ptr->rcu_field'.
--- .../selftests/bpf/progs/rcu_read_lock.c | 55 +++++++++++++++---- 1 file changed, 45 insertions(+), 10 deletions(-)
tools/testing/selftests/bpf/progs/rcu_read_lock.c +45 -10
@@ -23,13 +23,14 @@ struct bpf_key *bpf_lookup_user_key(__u32 serial, __u64 flags) __ksym; void bpf_key_put(struct bpf_key *key) __ksym; void bpf_rcu_read_lock(void) __ksym; void bpf_rcu_read_unlock(void) __ksym; -struct task_struct *bpf_task_acquire (struct task_struct *p) __ksym; +struct task_struct *bpf_task_acquire_not_zero (struct task_struct *p) __ksym; void bpf_task_release(struct task_struct *p) __ksym; SEC("?fentry.s/" SYS_PREFIX "sys_getpgid")int get_cgroup_id (void *ctx) {struct task_struct *task; +struct css_set *cgroups; task =bpf_get_current_task_btf ();if (task->pid != target_pid)
@@ -37,7 +38,11 @@ int get_cgroup_id(void *ctx)/* simulate bpf_get_current_cgroup_id() helper */ bpf_rcu_read_lock (); - cgroup_id = task->cgroups->dfl_cgrp->kn->id; + cgroups = task->cgroups; +if (!cgroups) + goto unlock; + cgroup_id = cgroups->dfl_cgrp->kn->id; +unlock:bpf_rcu_read_unlock ();return 0 ; }
@@ -56,6 +61,8 @@ int task_succ(void *ctx)bpf_rcu_read_lock ();/* region including helper using rcu ptr real_parent */ real_parent = task->real_parent; +if (!real_parent) + goto out; ptr =bpf_task_storage_get (&map_a, real_parent, &init_val,BPF_LOCAL_STORAGE_GET_F_CREATE );if (!ptr)
@@ -92,7 +99,10 @@ int two_regions(void *ctx)bpf_rcu_read_unlock ();bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +out:bpf_rcu_read_unlock ();return 0 ; }
@@ -105,7 +115,10 @@ int non_sleepable_1(void *ctx) task =bpf_get_current_task_btf ();bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +out:bpf_rcu_read_unlock ();return 0 ; }
@@ -121,7 +134,10 @@ int non_sleepable_2(void *ctx)bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +out:bpf_rcu_read_unlock ();return 0 ; }
@@ -129,16 +145,28 @@ int non_sleepable_2(void *ctx) SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep")int task_acquire (void *ctx) { -struct task_struct *task, *real_parent; +struct task_struct *task, *real_parent, *gparent; task =bpf_get_current_task_btf ();bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; + +/* rcu_ptr->rcu_field */ + gparent = real_parent->real_parent; +if (!gparent) + goto out; +/* acquire a reference which can be used outside rcu read lock region */ - real_parent =bpf_task_acquire (real_parent); + gparent =bpf_task_acquire_not_zero (gparent); +if (!gparent) + goto out; + + (void )bpf_task_storage_get (&map_a, gparent,0 ,0 ); +bpf_task_release (gparent); +out:bpf_rcu_read_unlock (); - (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); -bpf_task_release (real_parent);return 0 ; }
@@ -181,9 +209,12 @@ int non_sleepable_rcu_mismatch(void *ctx)/* non-sleepable: missing bpf_rcu_read_unlock() in one path */ bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 );if (real_parent)bpf_rcu_read_unlock (); +out:return 0 ; }
@@ -199,16 +230,17 @@ int inproper_sleepable_helper(void *ctx)/* sleepable helper in rcu read lock region */ bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; regs = (struct pt_regs *)bpf_task_pt_regs (real_parent); -if (!regs) { -bpf_rcu_read_unlock (); -return 0 ; - } +if (!regs) + goto out; ptr = (void *)PT_REGS_IP (regs); (void )bpf_copy_from_user_task (&value,sizeof (uint32_t ), ptr, task,0 ); user_data = value; (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +out:bpf_rcu_read_unlock ();return 0 ; }
@@ -239,7 +271,10 @@ int nested_rcu_region(void *ctx)bpf_rcu_read_lock ();bpf_rcu_read_lock (); real_parent = task->real_parent; +if (!real_parent) + goto out; (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +out:bpf_rcu_read_unlock ();bpf_rcu_read_unlock ();return 0 ;
Signature · 1 line
2.30.2
PATCH v2 3/3 docs/bpf: Add KF_RCU documentation
In reply to Yonghong SongAdd proper KF_RCU documentation in kfuncs.rst.
--- Documentation/bpf/kfuncs.rst | 9 +++++++++ 1 file changed, 9 insertions(+)
Documentation/bpf/kfuncs.rst +9 -0
@@ -191,6 +191,15 @@ rebooting or panicking. Due to this additional restrictions apply to these calls. At the moment they only require CAP_SYS_BOOT capability, but more can be added later. +2.4.8 KF_RCU flag +----------------- + +The KF_RCU flag is used for kfuncs which have a rcu ptr as its argument. +When used together with KF_ACQUIRE, it indicates the kfunc should have a +single argument which must be a trusted argument or a MEM_RCU pointer. +The argument may have reference count of 0 and the kfunc must take this +into consideration. + 2.5 Registering the kfuncs --------------------------
Signature · 1 line
2.30.2
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 9 lines · On Sat, 3 Dec 2022 10:45:57 -0800 you wrote:
Patch set [1] added rcu support for bpf programs. In [1], a rcu pointer is considered to be trusted and not null. This is actually not true in some cases. The rcu pointer could be null, and for non-null rcu pointer, it may have reference count of 0. This small patch set fixed this problem. Patch 1 is the kernel fix. Patch 2 adjusted selftests properly. Patch 3 added documentation for newly-introduced
KF_RCUflag.[...]
Here is the summary with links:
- [bpf-next,v2,1/3] bpf: Handle MEM_RCU type properly
https://git.kernel.org/bpf/bpf-next/c/fca1aa75518c - [bpf-next,v2,2/3] selftests/bpf: Fix rcu_read_lock test with new MEM_RCU semantics https://git.kernel.org/bpf/bpf-next/c/8723ec22a31d - [bpf-next,v2,3/3] docs/bpf: Add KF_RCU documentation https://git.kernel.org/bpf/bpf-next/c/f53625649888
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html