bpf: Add bpf_rcu_read_lock() support
Subsystems (7)
v7 cover bpf: Add bpf_rcu_read_lock() support
Currently, without rcu attribute info in BTF, the verifier treats rcu tagged pointer as a normal pointer. This might be a problem for sleepable program where rcu_read_lock()/unlock() is not available. For example, for a sleepable fentry program, if rcu protected memory access is interleaved with a sleepable helper/kfunc, it is possible the memory access after the sleepable helper/kfunc might be invalid since the object might have been freed then. Even without
a sleepable helper/kfunc, without rcu_read_lock() protection, it is possible that the rcu protected object might be release in the middle of bpf program execution which may cause incorrect result.
To prevent above cases, enable btf_type_tag("rcu") attributes, introduce new bpf_rcu_read_lock/unlock() kfuncs and add verifier support.
In the rest of patch set, Patch 1 enabled btf_type_tag for __rcu attribute. Patche 2 is a refactoring patch. Patch 3 added new bpf_rcu_read_lock/unlock() kfuncs and verifier support. Patch 4 added some tests for these two new kfuncs.
Changelogs:
v6 -> v7:
. rebase on top of bpf-next.
. remove the patch which enables sleepable program using
cgrp_local_storage map. This is orthogonal to this patch set and will be addressed separately.
. mark the rcu pointer dereference result as UNTRUSTED if inside
a bpf_rcu_read_lock() region.
v5 -> v6:
. fix selftest prog miss_unlock which tested nested locking. . add comments in selftest prog cgrp_succ to explain how to handle
nested memory access after rcu memory load.
v4 -> v5:
. add new test to aarch64 deny list.
v3 -> v4:
. fix selftest failures when built with gcc. gcc doesn't support
btf_type_tag yet and some tests relies on that. skip these tests if vmlinux BTF does not have btf_type_tag("rcu").
v2 -> v3:
. went back to MEM_RCU approach with invalidate rcu ptr registers
at bpf_rcu_read_unlock() place.
. remove KF_RCU_LOCK/UNLOCK flag and compare btf_id at verification
time instead.
v1 -> v2:
. use kfunc instead of helper for bpf_rcu_read_lock/unlock. . not use MEM_RCU bpf_type_flag, instead use active_rcu_lock
in reg state to identify rcu ptr's.
. Add more self tests.
. add new test to s390x deny list.
Yonghong Song (4):
compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))
bpf: Abstract out functions to check sleepable helpers
bpf: Add kfunc bpf_rcu_read_lock/unlock()
selftests/bpf: Add tests for bpf_rcu_read_lock()
include/linux/bpf.h | 3 + include/linux/bpf_lsm.h | 6 + include/linux/bpf_verifier.h | 2 + include/linux/compiler_types.h | 3 +- include/linux/trace_events.h | 8 + kernel/bpf/bpf_lsm.c | 20 +- kernel/bpf/btf.c | 3 + kernel/bpf/helpers.c | 12 + kernel/bpf/verifier.c | 156 ++++++++- kernel/trace/bpf_trace.c | 22 +- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/rcu_read_lock.c | 153 +++++++++ .../selftests/bpf/progs/rcu_read_lock.c | 325 ++++++++++++++++++ 14 files changed, 695 insertions(+), 20 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c create mode 100644 tools/testing/selftests/bpf/progs/rcu_read_lock.c
Signature · 1 line
2.30.2
PATCH v7 1/4 compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))
Currently, without rcu attribute info in BTF, the verifier treats rcu tagged pointer as a normal pointer. This might be a problem for sleepable program where rcu_read_lock()/unlock() is not available. For example, for a sleepable fentry program, if rcu protected memory access is interleaved with a sleepable helper/kfunc, it is possible the memory access after the sleepable helper/kfunc might be invalid since the object might have been freed then. To prevent such cases, introducing rcu tagging for memory accesses in verifier can help to reject such programs.
To enable rcu tagging in BTF, during kernel compilation,
define __rcu as attribute btf_type_tag("rcu") so __rcu information can be preserved in dwarf and btf, and later can be used for bpf prog verification.
--- include/linux/compiler_types.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/compiler_types.h +2 -1
@@ -49,7 +49,8 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }# endif # define __iomem# define __percpu BTF_TYPE_TAG(percpu) -# define __rcu +# define __rcu BTF_TYPE_TAG(rcu) +# define __chk_user_ptr (x) (void)0# define __chk_io_ptr (x) (void)0/* context/locking */
Signature · 1 line
2.30.2
PATCH v7 2/4 bpf: Abstract out functions to check sleepable helpers
In reply to Yonghong SongAbstract out two functions to check whether a particular helper is sleepable or not for bpf_lsm and bpf_trace. These two functions will be used later to check whether a helper is sleepable or not in verifier. There is no functionality change.
--- include/linux/bpf_lsm.h | 6 ++++++ include/linux/trace_events.h | 8 ++++++++ kernel/bpf/bpf_lsm.c | 20 ++++++++++++++++---- kernel/trace/bpf_trace.c | 22 ++++++++++++++++++---- 4 files changed, 48 insertions(+), 8 deletions(-)
include/linux/bpf_lsm.h +6 -0
@@ -28,6 +28,7 @@ int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, const struct bpf_prog *prog);bool bpf_lsm_is_sleepable_hook (u32 btf_id); +const struct bpf_func_proto *bpf_lsm_sleepable_func_proto (enum bpf_func_id func_id);static inline struct bpf_storage_blob *bpf_inode (const struct inode *inode)
@@ -50,6 +51,11 @@ static inline bool bpf_lsm_is_sleepable_hook(u32 btf_id) {return false; } +static inline const struct bpf_func_proto * +bpf_lsm_sleepable_func_proto (enum bpf_func_id func_id) +{ +return NULL ; +}static inline int bpf_lsm_verify_prog (struct bpf_verifier_log *vlog,const struct bpf_prog *prog)
include/linux/trace_events.h +8 -0
@@ -9,6 +9,7 @@#include <linux/hardirq.h> #include <linux/perf_event.h> #include <linux/tracepoint.h> +#include <uapi/linux/bpf.h> struct trace_array ;struct array_buffer ;
@@ -16,6 +17,7 @@ struct tracer;struct dentry ;struct bpf_prog ;union bpf_attr ; +struct bpf_func_proto ; const char *trace_print_flags_seq(struct trace_seq *p, const char *delim, unsigned long flags,
@@ -748,6 +750,7 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id, u32 *fd_type, const char **buf, u64 *probe_offset, u64 *probe_addr);int bpf_kprobe_multi_link_attach (const union bpf_attr *attr,struct bpf_prog *prog); +const struct bpf_func_proto *bpf_tracing_sleepable_func_proto (enum bpf_func_id func_id);#else static inline unsigned int trace_call_bpf (struct trace_event_call *call,void *ctx) {
@@ -794,6 +797,11 @@ bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) {return -EOPNOTSUPP ; } +static inline const struct bpf_func_proto * +bpf_tracing_sleepable_func_proto (enum bpf_func_id func_id) +{ +return NULL ; +}#endif enum {
kernel/bpf/bpf_lsm.c +16 -4
@@ -192,6 +192,18 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto = { .arg1_type = ARG_PTR_TO_CTX, }; +const struct bpf_func_proto *bpf_lsm_sleepable_func_proto (enum bpf_func_id func_id) +{ +switch (func_id) { +case BPF_FUNC_ima_inode_hash: +return &bpf_ima_inode_hash_proto; +case BPF_FUNC_ima_file_hash: +return &bpf_ima_file_hash_proto; +default : +return NULL ; + } +} +static const struct bpf_func_proto *bpf_lsm_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) {
@@ -203,6 +215,10 @@ bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return func_proto; } + func_proto =bpf_lsm_sleepable_func_proto (func_id); +if (func_proto) +return prog->aux->sleepable ? func_proto :NULL ; +switch (func_id) {case BPF_FUNC_inode_storage_get:return &bpf_inode_storage_get_proto;
@@ -220,10 +236,6 @@ bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_spin_unlock_proto;case BPF_FUNC_bprm_opts_set:return &bpf_bprm_opts_set_proto; -case BPF_FUNC_ima_inode_hash: -return prog->aux->sleepable ? &bpf_ima_inode_hash_proto :NULL ; -case BPF_FUNC_ima_file_hash: -return prog->aux->sleepable ? &bpf_ima_file_hash_proto :NULL ;case BPF_FUNC_get_attach_cookie:return bpf_prog_has_trampoline (prog) ? &bpf_get_attach_cookie_proto :NULL ;#ifdef CONFIG_NET
kernel/trace/bpf_trace.c +18 -4
@@ -1383,9 +1383,27 @@ static int __init bpf_key_sig_kfuncs_init(void)late_initcall (bpf_key_sig_kfuncs_init);#endif /* CONFIG_KEYS */ +const struct bpf_func_proto *bpf_tracing_sleepable_func_proto (enum bpf_func_id func_id) +{ +switch (func_id) { +case BPF_FUNC_copy_from_user: +return &bpf_copy_from_user_proto; +case BPF_FUNC_copy_from_user_task: +return &bpf_copy_from_user_task_proto; +default : +return NULL ; + } +} +static const struct bpf_func_proto *bpf_tracing_func_proto (enum bpf_func_id func_id,const struct bpf_prog *prog) { +const struct bpf_func_proto *func_proto; + + func_proto =bpf_tracing_sleepable_func_proto (func_id); +if (func_proto) +return prog->aux->sleepable ? func_proto :NULL ; +switch (func_id) {case BPF_FUNC_map_lookup_elem:return &bpf_map_lookup_elem_proto;
@@ -1484,10 +1502,6 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)return &bpf_jiffies64_proto;case BPF_FUNC_get_task_stack:return &bpf_get_task_stack_proto; -case BPF_FUNC_copy_from_user: -return prog->aux->sleepable ? &bpf_copy_from_user_proto :NULL ; -case BPF_FUNC_copy_from_user_task: -return prog->aux->sleepable ? &bpf_copy_from_user_task_proto :NULL ;case BPF_FUNC_snprintf_btf:return &bpf_snprintf_btf_proto;case BPF_FUNC_per_cpu_ptr:
Signature · 1 line
2.30.2
Re: [PATCH bpf-next v7 2/4] bpf: Abstract out functions to check sleepable helpers
Quoted 8 lines · Yonghong Song
Abstract out two functions to check whether a particular helper is sleepable or not for
bpf_lsmandbpf_trace. These two functions will be used later to check whether a helper is sleepable or not in verifier. There is no functionality change.Signed-off-by Yonghong Song <[email protected]>
PATCH v7 3/4 bpf: Add kfunc bpf_rcu_read_lock/unlock()
In reply to Yonghong SongAdd two kfunc's bpf_rcu_read_lock() and bpf_rcu_read_unlock(). These two kfunc's can be used for all program types. The following is an example about how rcu pointer are used w.r.t. bpf_rcu_read_lock()/bpf_rcu_read_unlock().
struct task_struct {
...
struct task_struct *last_wakee;
struct task_struct __rcu *real_parent;
...
};Let us say prog does 'task = bpf_get_current_task_btf()' to get a 'task' pointer. The basic rules are:
- 'real_parent = task->real_parent' should be inside bpf_rcu_read_lock region. this is to simulate rcu_dereference() operation.
- 'last_wakee = real_parent->last_wakee' should be iinside bpf_rcu_read_lock region since it tries to access rcu protected memory.
- the ptr 'last_wakee' will be marked as PTR_UNTRUSTED since in general it is not clear whether the object pointed by 'last_wakee' is valid or not even inside bpf_rcu_read_lock region.
Note that for non-sleepable progs, it is permitted that rcu pointer load and accessing rcu protected memory can be outside rcu read lock region. But verification will be reported for sleepable progs.
To prevent rcu pointer leaks outside the rcu read lock region. The verifier will clear all rcu pointer register state to unknown, i.e., scalar_value, so later dereference becomes impossible.
--- include/linux/bpf.h | 3 + include/linux/bpf_verifier.h | 2 + kernel/bpf/btf.c | 3 + kernel/bpf/helpers.c | 12 +++ kernel/bpf/verifier.c | 156 ++++++++++++++++++++++++++++++++--- 5 files changed, 165 insertions(+), 11 deletions(-)
include/linux/bpf.h +3 -0
@@ -572,6 +572,9 @@ enum bpf_type_flag {*/ PTR_TRUSTED =BIT (12 +BPF_BASE_TYPE_BITS ), +/* MEM is tagged with rcu and memory access needs rcu_read_lock protection. */ +MEM_RCU =BIT (13 +BPF_BASE_TYPE_BITS ), + __BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, };
include/linux/bpf_verifier.h +2 -0
@@ -344,6 +344,7 @@ struct bpf_verifier_state {u32 id; } active_lock;bool speculative; +bool active_rcu_lock;/* first and last insn idx of this verifier state */ u32 first_insn_idx;
@@ -445,6 +446,7 @@ struct bpf_insn_aux_data {u32 seen;/* this insn was processed by the verifier at env->pass_cnt */ bool sanitize_stack_spill;/* subject to Spectre v4 sanitation */ bool zext_dst;/* this insn zero extends dst reg */ +bool storage_get_func_atomic;/* bpf_*_storage_get() with atomic memory alloc */ u8 alu_state;/* used in combination with alu_limit */ /* below fields are initialized once */
kernel/bpf/btf.c +3 -0
@@ -6237,6 +6237,9 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,/* check __percpu tag */ if (strcmp (tag_value,"percpu" ) ==0 ) tmp_flag =MEM_PERCPU ; +/* check __rcu tag */ +if (strcmp (tag_value,"rcu" ) ==0 ) + tmp_flag =MEM_RCU ; } stype =btf_type_skip_modifiers (btf, mtype->type, &id);
kernel/bpf/helpers.c +12 -0
@@ -1889,6 +1889,16 @@ void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k)return obj__ign; } +void bpf_rcu_read_lock (void ) +{ +rcu_read_lock (); +} + +void bpf_rcu_read_unlock (void ) +{ +rcu_read_unlock (); +} +__diag_pop ();BTF_SET8_START (generic_btf_ids)
@@ -1919,6 +1929,8 @@ BTF_ID(func, bpf_task_release)BTF_SET8_START (common_btf_ids)BTF_ID_FLAGS (func ,bpf_cast_to_kern_ctx )BTF_ID_FLAGS (func, bpf_rdonly_cast) +BTF_ID_FLAGS (func, bpf_rcu_read_lock) +BTF_ID_FLAGS (func, bpf_rcu_read_unlock)BTF_SET8_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/verifier.c +145 -11
309 lines not on this page · raw message
Signature · 1 line
2.30.2
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 19 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0@@ -4704,6 +4715,15 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,return -EACCES ; } +/* Access rcu protected memory */ +if ((reg->type &MEM_RCU ) && env->prog->aux->sleepable && + !env->cur_state->active_rcu_lock) { +verbose (env, +"R%d is ptr_%s access rcu-protected memory with off=%d, not rcu protected\n" , + regno, tname, off); +return -EACCES ; + } +if (env->ops->btf_struct_access && !type_is_alloc (reg->type)) {if (!btf_is_kernel (reg->btf)) {verbose (env,"verifier internal error: reg->btf must be kernel btf\n" );@@ -4731,12 +4751,27 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,if (ret <0 )return ret;
kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · line 4762
Quoted 10 lines
kernel/bpf/verifier.c +9
-0+/* The value is a rcu pointer. The load needs to be in a rcu lock region, +* similar to rcu_dereference(). +*/ +if ((flag &MEM_RCU ) && env->prog->aux->sleepable && !env->cur_state->active_rcu_lock) { +verbose (env, +"R%d is rcu dereference ptr_%s with off=%d, not in rcu_read_lock region\n" , + regno, tname, off); +return -EACCES ; + }
Would this make the existing rdonly use case fail?
SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
int task_real_parent(void *ctx)
{
struct task_struct *task, *real_parent;
task = bpf_get_current_task_btf();
real_parent = task->real_parent;
bpf_printk("pid %u\n", real_parent->pid);
return 0;
}Re: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 42 lines · Martin KaFai Lau
Quoted 23 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0@@ -4704,6 +4715,15 @@ static int check_ptr_to_btf_access(structbpf_verifier_env *env, return -EACCES; } + /* Access rcu protected memory */ + if ((reg->type & MEM_RCU) && env->prog->aux->sleepable && + !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is ptr_%s access rcu-protected memory with off=%d, not rcu protected\n", + regno, tname, off); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type)) { if (!btf_is_kernel(reg->btf)) {
verbose(env, "verifier internal error: reg->btf must be kernel btf\n");kernel/bpf/verifier.c +1
-0@@ -4731,12 +4751,27 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; + /* The value is a rcu pointer. The load needs to be in a rcu lockregion, kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct · line 4765
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+* similar to rcu_dereference(). +*/ + if ((flag & MEM_RCU) && env->prog->aux->sleepable && !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is rcu dereference ptr_%s with off=%d, not in rcu_read_lock region\n", + regno, tname, off); + return -EACCES; + }Would this make the existing rdonly use case fail?
SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
inttask_real_parent(void *ctx)
{
structtask_struct*task,*real_parent;
Quoted 5 lines
task =
bpf_get_current_task_btf();real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid); return 0;
}
Right, it will fail. To fix the issue, user can do
bpf_rcu_read_lock();
real_parent = task->real_parent; bpf_printk("pid %u\n", real_parent->pid); bpf_rcu_read_unlock();
But this raised a good question. How do we deal with legacy sleepable programs with newly-added rcu tagging capabilities.
My current option is to error out if rcu usage is not right. But this might break existing sleepable programs.
Another option intends to not break existing, like above, codes. In this case, MEM_RCU will not tagged if it is
not inside bpf_rcu_read_lock() region. In this case,
the above non-rcu-protected code should work. And the following should work as well although it is a little
bit awkward.
real_parent = task->real_parent; // real_parent not tagged with rcu bpf_rcu_read_lock();
bpf_printk("pid %u\n", real_parent->pid); bpf_rcu_read_unlock();
Maybe we should take the second choice in the above instead?
Quoted 1 line
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 60 lines · Yonghong Song
Quoted 41 lines · Martin KaFai Lau
Quoted 22 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0@@ -4704,6 +4715,15 @@ static int check_ptr_to_btf_access(structbpf_verifier_env *env, return -EACCES; } + /* Access rcu protected memory */ + if ((reg->type & MEM_RCU) && env->prog->aux->sleepable && + !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is ptr_%s access rcu-protected memory with off=%d, not rcu protected\n", + regno, tname, off); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type)) { if (!btf_is_kernel(reg->btf)) {
verbose(env, "verifier internal error: reg->btf must be kernel btf\n");kernel/bpf/verifier.c +1
-0@@ -4731,12 +4751,27 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; + /* The value is a rcu pointer. The load needs to be in a rcu lock region,kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct · line 4764
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+* similar to rcu_dereference(). +*/ + if ((flag & MEM_RCU) && env->prog->aux->sleepable && !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is rcu dereference ptr_%s with off=%d, not in rcu_read_lock region\n", + regno, tname, off); + return -EACCES; + }Would this make the existing rdonly use case fail?
SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
inttask_real_parent(void *ctx)
{
structtask_struct*task,*real_parent;Quoted 5 lines
task =
bpf_get_current_task_btf();real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid); return 0;
}Right, it will fail. To fix the issue, user can do
bpf_rcu_read_lock();
real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();But this raised a good question. How do we deal with legacy sleepable programs with newly-added rcu tagging capabilities.
Quoted 6 lines
My current option is to error out if rcu usage is not right. But this might break existing sleepable programs.
Another option intends to not break existing, like above, codes. In this case,
MEM_RCUwill not tagged if it is not insidebpf_rcu_read_lock()region.
hmm.... it is to make MEM_RCU to mean a reg is protected by the current active_rcu_lock or not?
Quoted 7 lines
In this case, the above non-rcu-protected code should work. And the following should work as well although it is a little
bit awkward.
real_parent=task->real_parent; //real_parentnot tagged with rcubpf_rcu_read_lock();
bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();
I think it should be fine. bpf_rcu_read_lock() just not useful in this example but nothing break or crash. Also, after bpf_rcu_read_unlock(), real_parent will continue to be readable because the MEM_RCU is not set?
On top of the active_rcu_lock, should MEM_RCU be set only when it is dereferenced from a PTR_TRUSTED ptr (or with ref_obj_id != 0)? I am thinking about the following more common case:
/* bpf_get_current_task_btf() may need to be changed
* to set PTR_TRUSTED at the retval?
*/
/* task: PTR_TO_BTF_ID | PTR_TRUSTED */
task = bpf_get_current_task_btf();
bpf_rcu_read_lock();
/* real_parent: PTR_TO_BTF_ID | PTR_TRUSTED | MEM_RCU */
real_parent = task->real_parent;
/* bpf_task_acquire() needs to change to use refcount_inc_not_zero */
real_parent = bpf_task_acquire(real_parent);
bpf_rcu_read_unlock();
/* real_parent is accessible here (after checking NULL) and
* can be passed to kfunc
*/Re: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 62 lines · Martin KaFai Lau
Quoted 61 lines · Yonghong Song
Quoted 42 lines · Martin KaFai Lau
Quoted 23 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0@@ -4704,6 +4715,15 @@ static int check_ptr_to_btf_access(structbpf_verifier_env *env, return -EACCES; } + /* Access rcu protected memory */ + if ((reg->type & MEM_RCU) && env->prog->aux->sleepable && + !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is ptr_%s access rcu-protected memory with off=%d, not rcu protected\n", + regno, tname, off); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type)) { if (!btf_is_kernel(reg->btf)) {
verbose(env, "verifier internal error: reg->btf must be kernel btf\n");kernel/bpf/verifier.c +1
-0@@ -4731,12 +4751,27 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; + /* The value is a rcu pointer. The load needs to be in a rculock region, kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct · line 4765
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+* similar to rcu_dereference(). +*/ + if ((flag & MEM_RCU) && env->prog->aux->sleepable && !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is rcu dereference ptr_%s with off=%d, not in rcu_read_lock region\n", + regno, tname, off); + return -EACCES; + }Would this make the existing rdonly use case fail?
SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
inttask_real_parent(void *ctx)
{
structtask_struct*task,*real_parent;Quoted 5 lines
task =
bpf_get_current_task_btf();real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid); return 0;
}Right, it will fail. To fix the issue, user can do
bpf_rcu_read_lock();
real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();But this raised a good question. How do we deal with legacy sleepable programs with newly-added rcu tagging capabilities.
Quoted 8 lines
Quoted 6 lines
My current option is to error out if rcu usage is not right. But this might break existing sleepable programs.
Another option intends to not break existing, like above, codes. In this case,
MEM_RCUwill not tagged if it is not insidebpf_rcu_read_lock()region.hmm.... it is to make
MEM_RCUto mean a reg is protected by the currentactive_rcu_lockor not?
Yes, for example, in 'real_parent = task->real_parent' where 'real_parent' in task_struct is tagged with __rcu in the struct definition. So the 'real_parent' variable in the above assignment will be tagged with MEM_RCU.
Quoted 10 lines
Quoted 8 lines
In this case, the above non-rcu-protected code should work. And the following should work as well although it is a little
bit awkward.
real_parent=task->real_parent; //real_parentnot tagged with rcubpf_rcu_read_lock();
bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();I think it should be fine.
bpf_rcu_read_lock()just not useful in this example but nothing break or crash. Also, afterbpf_rcu_read_unlock(),real_parentwill continue to be readable because theMEM_RCUis not set?
That is correct. the variable real_parent is not tagged with MEM_RCU and it will stay that way for the rest of its life cycle.
With new PTR_TRUSTED mechanism, real_parent will be marked as normal PTR_TO_BTF_ID and it is not marked as PTR_UNTRUSTED for backward compatibility. So in the above code, real_parent->pid is just a normal load (not related to rcu/trusted/untrusted). People may think it is okay, but actually it does not okay. Verifier could add more state to issue proper warnings, but I am not sure whether it is worthwhile or not. As you mentioned, nothing breaks. It is just the current existing way. So we should be able to live with this.
Quoted 2 lines
On top of the
active_rcu_lock, shouldMEM_RCUbe set only when it is dereferenced from aPTR_TRUSTEDptr (or withref_obj_id!= 0)?
I didn't consider PTR_TRUSTED because it is just introduced yesterday...
My current implementation inherits the old ptr_to_btf_id way where by default any ptr_to_btf_id is trusted. But since we have PTR_TRUSTED we should be able to use it for a stronger guarantee.
Quoted 17 lines
I am thinking about the following more common case:
/*
bpf_get_current_task_btf()may need to be changed
* to setPTR_TRUSTEDat the retval?
*/
/* task:PTR_TO_BTF_ID|PTR_TRUSTED*/ task =bpf_get_current_task_btf();bpf_rcu_read_lock();
/*
real_parent:PTR_TO_BTF_ID|PTR_TRUSTED|MEM_RCU*/real_parent=task->real_parent;/*
bpf_task_acquire()needs to change to userefcount_inc_not_zero*/
real_parent =bpf_task_acquire(real_parent);
Quoted 6 lines
bpf_rcu_read_unlock();
/*
real_parentis accessible here (after checking NULL) and
* can be passed to kfunc
*/
Yes, the above is a typical use case. Or alternatively after
real_parent = task->real_parent;
/* use real_parent inside the bpf_rcu_read_lock() region */I will try to utilize PTR_TRUSTED concept in the next revision.
John Fastabend
reviewerRe: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 140 lines · Yonghong Song
Quoted 66 lines · Martin KaFai Lau
Quoted 65 lines · Yonghong Song
Quoted 43 lines · Martin KaFai Lau
Quoted 24 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0@@ -4704,6 +4715,15 @@ static int check_ptr_to_btf_access(structbpf_verifier_env *env, return -EACCES; } + /* Access rcu protected memory */ + if ((reg->type & MEM_RCU) && env->prog->aux->sleepable && + !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is ptr_%s access rcu-protected memory with off=%d, not rcu protected\n", + regno, tname, off); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type)) { if (!btf_is_kernel(reg->btf)) {
verbose(env, "verifier internal error: reg->btf must be kernel btf\n");kernel/bpf/verifier.c +1
-0@@ -4731,12 +4751,27 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; + /* The value is a rcu pointer. The load needs to be in a rculock region, kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct · line 4765
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+* similar to rcu_dereference(). +*/ + if ((flag & MEM_RCU) && env->prog->aux->sleepable && !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is rcu dereference ptr_%s with off=%d, not in rcu_read_lock region\n", + regno, tname, off); + return -EACCES; + }Would this make the existing rdonly use case fail?
SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
inttask_real_parent(void *ctx)
{
structtask_struct*task,*real_parent;Quoted 5 lines
task =
bpf_get_current_task_btf();real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid); return 0;
}Right, it will fail. To fix the issue, user can do
bpf_rcu_read_lock();
real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();But this raised a good question. How do we deal with legacy sleepable programs with newly-added rcu tagging capabilities.
My current option is to error out if rcu usage is not right. But this might break existing sleepable programs.
Quoted 6 lines
Quoted 3 lines
Another option intends to not break existing, like above, codes. In this case,
MEM_RCUwill not tagged if it is not insidebpf_rcu_read_lock()region.hmm.... it is to make
MEM_RCUto mean a reg is protected by the currentactive_rcu_lockor not?Yes, for example, in '
real_parent=task->real_parent' where 'real_parent' intask_structis tagged with__rcuin the struct definition. So the 'real_parent' variable in the above assignment will be tagged withMEM_RCU.Quoted 9 lines
Quoted 8 lines
In this case, the above non-rcu-protected code should work. And the following should work as well although it is a little
bit awkward.
real_parent=task->real_parent; //real_parentnot tagged with rcubpf_rcu_read_lock();
bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();Quoted 3 lines
I think it should be fine.
bpf_rcu_read_lock()just not useful in this example but nothing break or crash. Also, afterbpf_rcu_read_unlock(),real_parentwill continue to be readable because theMEM_RCUis not set?That is correct. the variable
real_parentis not tagged withMEM_RCUand it will stay that way for the rest of its life cycle.With new
PTR_TRUSTEDmechanism,real_parentwill be marked as normalPTR_TO_BTF_IDand it is not marked asPTR_UNTRUSTEDfor backward compatibility. So in the above code,real_parent->pidis just a normal load (not related to rcu/trusted/untrusted). People may think it is okay, but actually it does not okay. Verifier could add more state to issue proper warnings, but I am not sure whether it is worthwhile or not. As you mentioned, nothing breaks. It is just the current existing way. So we should be able to live with this.Quoted 3 lines
On top of the
active_rcu_lock, shouldMEM_RCUbe set only when it is dereferenced from aPTR_TRUSTEDptr (or withref_obj_id!= 0)?I didn't consider
PTR_TRUSTEDbecause it is just introduced yesterday...My current implementation inherits the old
ptr_to_btf_idway where by default anyptr_to_btf_idis trusted. But since we havePTR_TRUSTEDwe should be able to use it for a stronger guarantee.Quoted 17 lines
I am thinking about the following more common case:
/*
bpf_get_current_task_btf()may need to be changed
* to setPTR_TRUSTEDat the retval?
*/
/* task:PTR_TO_BTF_ID|PTR_TRUSTED*/ task =bpf_get_current_task_btf();bpf_rcu_read_lock();
/*
real_parent:PTR_TO_BTF_ID|PTR_TRUSTED|MEM_RCU*/real_parent=task->real_parent;/*
bpf_task_acquire()needs to change to userefcount_inc_not_zero*/
real_parent =bpf_task_acquire(real_parent);Quoted 6 lines
bpf_rcu_read_unlock();
/*
real_parentis accessible here (after checking NULL) and
* can be passed to kfunc
*/
Quoted 5 lines
Yes, the above is a typical use case. Or alternatively after
real_parent = task->real_parent; /* use real_parent inside the bpf_rcu_read_lock() region */I will try to utilize
PTR_TRUSTEDconcept in the next revision.
Also perhaps interesting is when task is read out of a map with reference already pinned. I think you should clear the MEM_RCU tag on all referenced objects?
Re: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 145 lines · John Fastabend
Quoted 144 lines · Yonghong Song
Quoted 66 lines · Martin KaFai Lau
Quoted 65 lines · Yonghong Song
Quoted 43 lines · Martin KaFai Lau
Quoted 24 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0@@ -4704,6 +4715,15 @@ static int check_ptr_to_btf_access(structbpf_verifier_env *env, return -EACCES; } + /* Access rcu protected memory */ + if ((reg->type & MEM_RCU) && env->prog->aux->sleepable && + !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is ptr_%s access rcu-protected memory with off=%d, not rcu protected\n", + regno, tname, off); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type)) { if (!btf_is_kernel(reg->btf)) {verbose(env, "verifier internal error:
reg->btfmust
be kernel btf\n");kernel/bpf/verifier.c +1
-0@@ -4731,12 +4751,27 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; + /* The value is a rcu pointer. The load needs to be in a rculock region, kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct · line 4765
Quoted 10 lines
kernel/bpf/verifier.c +8
-0+* similar to rcu_dereference(). +*/ + if ((flag & MEM_RCU) && env->prog->aux->sleepable && !env->cur_state->active_rcu_lock) { + verbose(env, + "R%d is rcu dereference ptr_%s with off=%d, not in rcu_read_lock region\n", + regno, tname, off); + return -EACCES; + }Would this make the existing rdonly use case fail?
SEC("fentry.s/" SYS_PREFIX "sys_getpgid")
inttask_real_parent(void *ctx)
{
structtask_struct*task,*real_parent;Quoted 5 lines
task =
bpf_get_current_task_btf();real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid); return 0;
}Right, it will fail. To fix the issue, user can do
bpf_rcu_read_lock();
real_parent=task->real_parent;bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();But this raised a good question. How do we deal with legacy sleepable programs with newly-added rcu tagging capabilities.
My current option is to error out if rcu usage is not right. But this might break existing sleepable programs.
Quoted 6 lines
Quoted 3 lines
Another option intends to not break existing, like above, codes. In this case,
MEM_RCUwill not tagged if it is not insidebpf_rcu_read_lock()region.hmm.... it is to make
MEM_RCUto mean a reg is protected by the currentactive_rcu_lockor not?Yes, for example, in '
real_parent=task->real_parent' where 'real_parent' intask_structis tagged with__rcuin the struct definition. So the 'real_parent' variable in the above assignment will be tagged withMEM_RCU.Quoted 9 lines
Quoted 8 lines
In this case, the above non-rcu-protected code should work. And the following should work as well although it is a little
bit awkward.
real_parent=task->real_parent; //real_parentnot tagged with rcubpf_rcu_read_lock();
bpf_printk("pid %u\n", real_parent->pid);bpf_rcu_read_unlock();Quoted 3 lines
I think it should be fine.
bpf_rcu_read_lock()just not useful in this example but nothing break or crash. Also, afterbpf_rcu_read_unlock(),real_parentwill continue to be readable because theMEM_RCUis not set?That is correct. the variable
real_parentis not tagged withMEM_RCUand it will stay that way for the rest of its life cycle.With new
PTR_TRUSTEDmechanism,real_parentwill be marked as normalPTR_TO_BTF_IDand it is not marked asPTR_UNTRUSTEDfor backward compatibility. So in the above code,real_parent->pidis just a normal load (not related to rcu/trusted/untrusted). People may think it is okay, but actually it does not okay. Verifier could add more state to issue proper warnings, but I am not sure whether it is worthwhile or not. As you mentioned, nothing breaks. It is just the current existing way. So we should be able to live with this.Quoted 3 lines
On top of the
active_rcu_lock, shouldMEM_RCUbe set only when it is dereferenced from aPTR_TRUSTEDptr (or withref_obj_id!= 0)?I didn't consider
PTR_TRUSTEDbecause it is just introduced yesterday...My current implementation inherits the old
ptr_to_btf_idway where by default anyptr_to_btf_idis trusted. But since we havePTR_TRUSTEDwe should be able to use it for a stronger guarantee.Quoted 17 lines
I am thinking about the following more common case:
/*
bpf_get_current_task_btf()may need to be changed
* to setPTR_TRUSTEDat the retval?
*/
/* task:PTR_TO_BTF_ID|PTR_TRUSTED*/ task =bpf_get_current_task_btf();bpf_rcu_read_lock(); /* real_parent: PTR_TO_BTF_ID | PTR_TRUSTED | MEM_RCU */ real_parent = task->real_parent;/*
bpf_task_acquire()needs to change to userefcount_inc_not_zero*/
real_parent =bpf_task_acquire(real_parent);Quoted 6 lines
bpf_rcu_read_unlock();/*
real_parentis accessible here (after checking NULL) and
* can be passed to kfunc
*/Yes, the above is a typical use case. Or alternatively after
real_parent = task->real_parent; /* use real_parent inside the bpf_rcu_read_lock() region */
Quoted 5 lines
Quoted 1 line
I will try to utilize
PTR_TRUSTEDconcept in the next revision.Also perhaps interesting is when task is read out of a map with reference already pinned. I think you should clear the
MEM_RCUtag on all referenced objects?
The register tagged with MEM_RCU will not be a referenced obj. MEM_RCU tag only appears to a register inside the rcu read lock region as the rcu_reference() result. So the obj tagged with MEM_RCU is protected with rcu read lock and it is valid and trusted and there is no need to acquire additional reference. If user calls another kfunc to acquire a reference, then the resulted ptr will not have MEM_RCU tag but with non-zero ref_obj_id.
The MEM_RCU reg will be invalidated when seeing bpf_rcu_read_unlock() to prevent rcu-protected ptr to leak out of the rcu read lock region.
Re: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
In reply to Yonghong Songkernel/bpf/verifier.c static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · line 7596
Quoted 8 lines · Yonghong Song
kernel/bpf/verifier.c +8
-0+if (env->cur_state->active_rcu_lock) { +if (bpf_lsm_sleepable_func_proto (func_id) || +bpf_tracing_sleepable_func_proto (func_id)) { +verbose (env,"sleepable helper %s#%din rcu_read_lock region\n" , +func_id_name (func_id), func_id); +return -EINVAL ; + } +
Even after patch 2 refactoring the above bit is still quite fragile.
Ex: bpf_d_path is not included, but it should be.
How about we add 'bool might_sleep' to bpf_func_proto and mark existing 5 functions with it and refactor patch 2 differently.
We won't be doing prog->aux->sleepable ? in bpf_tracing_func_proto() anymore. Those cbs will be returning func_proto-s,
but the verifier later will check might_sleep flag.
Re: [PATCH bpf-next v7 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 11 lines · Alexei Starovoitov
Quoted 9 lines · Yonghong Song
+ if (
env->cur_state->active_rcu_lock) {
+ if (bpf_lsm_sleepable_func_proto(func_id)||
+bpf_tracing_sleepable_func_proto(func_id)) { + verbose(env, "sleepable helper %s#%dinrcu_read_lockregion\n",
+func_id_name(func_id),func_id);
+ return-EINVAL;
+ }
+
Quoted 8 lines
Even after patch 2 refactoring the above bit is still quite fragile.
Ex:bpf_d_pathis not included, but it should be.How about we add 'bool
might_sleep' tobpf_func_protoand mark existing 5 functions with it and refactor patch 2 differently.
We won't be doingprog->aux->sleepable? inbpf_tracing_func_proto()anymore. Those cbs will be returning func_proto-s,
but the verifier later will checkmight_sleepflag.
Ya, bpf_func_proto->might_sleep indeed better. I could do that. The only problem is bpf_d_path.
static bool bpf_d_path_allowed(const struct bpf_prog *prog)
{
if (prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_ITER)
return true;
if (prog->type == BPF_PROG_TYPE_LSM)
return bpf_lsm_is_sleepable_hook(prog->aux->attach_btf_id);
return btf_id_set_contains(&btf_allowlist_d_path,
prog->aux->attach_btf_id);
}If I understand correctly, bpf_d_path helper doesn't mean
the helper itself will be sleepable. For example, bpf_d_path can only appear in sleepable programs if program type is BPF_PROG_TYPE_LSM, from 6f100640ca5b ("bpf: Expose bpf_d_path helper to sleepable LSM hooks") it looks like the reason is those sleepable lsm programs provide better context so bpf_d_path won't have potential lock
or other issues. So essentially, bpf_d_path helper itself
won't be a helper causing the prog to sleep, right? If this is
the case, we only assign might_sleepable to the other 4 helpers.
PATCH v7 4/4 selftests/bpf: Add tests for bpf_rcu_read_lock()
In reply to Yonghong SongAdd a few positive/negative tests to test bpf_rcu_read_lock() and its corresponding verifier support. The new test will fail on s390x and aarch64, so an entry is added to each of their respective deny lists.
--- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/rcu_read_lock.c | 153 +++++++++ .../selftests/bpf/progs/rcu_read_lock.c | 325 ++++++++++++++++++ 4 files changed, 480 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c create mode 100644 tools/testing/selftests/bpf/progs/rcu_read_lock.c
tools/testing/selftests/bpf/DENYLIST.aarch64 +1 -0
@@ -45,6 +45,7 @@ modify_return # modify_return__attach failed module_attach # skel_attach skeleton attach failed: -524 mptcp/base # run_test mptcp unexpected error: -524 (errno 524) netcnt # packets unexpected packets: actual 10001 != expected 10000 +rcu_read_lock # failed to attach: ERROR: strerror_r(-524)=22 recursion # skel_attach unexpected error: -524 (errno 524) ringbuf # skel_attach skeleton attachment failed: -1 setget_sockopt # attach_cgroup unexpected error: -524
tools/testing/selftests/bpf/DENYLIST.s390x +1 -0
@@ -42,6 +42,7 @@ module_attach # skel_attach skeleton attach failed: - mptcp netcnt # failed to load BPF skeleton 'netcnt_prog': -7 (?) probe_user # check_kprobe_res wrong kprobe res from probe read (?) +rcu_read_lock # failed to find kernel BTF type ID of '__x64_sys_getpgid': -3 (?) recursion # skel_attach unexpected error: -524 (trampoline) ringbuf # skel_load skeleton load failed (?) select_reuseport # intermittently fails on new s390x setup
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c new file +153 -0
153 lines not on this page · raw message
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +325 -0
325 lines not on this page · raw message
Signature · 1 line
2.30.2
John Fastabend
reviewerRE: [PATCH bpf-next v7 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
Quoted 6 lines · Yonghong Song
Add a few positive/negative tests to test
bpf_rcu_read_lock()and its corresponding verifier support. The new test will fail on s390x and aarch64, so an entry is added to each of their respective deny lists.Signed-off-by Yonghong Song <[email protected]>
[...]
Quoted 5 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +5
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int nested_rcu_region (void *ctx) +{ +struct task_struct *task, *real_parent; +
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 262
Quoted 10 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+/* nested rcu read lock regions */ + task =bpf_get_current_task_btf (); +bpf_rcu_read_lock (); +bpf_rcu_read_lock (); + real_parent = task->real_parent; + (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +bpf_rcu_read_unlock (); +bpf_rcu_read_unlock (); +return 0 ; +}
I think you also need the nested imbalance case is this handled? It looks like the active_rcu is just a bool?
+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep")
+int nested_rcu_region(void *ctx)
+{
+ struct task_struct *task, *real_parent;
+
+ /* nested rcu read lock regions */
+ task = bpf_get_current_task_btf();
+ bpf_rcu_read_lock();
+ bpf_rcu_read_lock();
+ real_parent = task->real_parent;
+ (void)bpf_task_storage_get(&map_a, real_parent, 0, 0);
+ // imbalance unlock()
+ bpf_rcu_read_unlock();
+ return 0;
+}
Re: [PATCH bpf-next v7 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
Quoted 20 lines · John Fastabend
Quoted 6 lines · Yonghong Song
Add a few positive/negative tests to test
bpf_rcu_read_lock()and its corresponding verifier support. The new test will fail on s390x and aarch64, so an entry is added to each of their respective deny lists.Signed-off-by Yonghong Song <[email protected]>[...]
Quoted 10 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int nested_rcu_region (void *ctx) +{ +struct task_struct *task, *real_parent; + +/* nested rcu read lock regions */ + task =bpf_get_current_task_btf (); +bpf_rcu_read_lock (); +bpf_rcu_read_lock (); + real_parent = task->real_parent;
Quoted 8 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 98
Quoted 5 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +5
-0+ (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +bpf_rcu_read_unlock (); +bpf_rcu_read_unlock (); +return 0 ; +}I think you also need the nested imbalance case is this handled? It looks like the
active_rcuis just a bool?
Currently we don't support nested bpf_rcu_read_lock() regions. So the error will appear for the second bpf_rcu_read_lock() in the above code, regardless of the eventual balance or not-balance.
Quoted 16 lines
+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep")
+int nested_rcu_region(void *ctx)
+{
+ struct task_struct *task, *real_parent;
+
+ /* nested rcu read lock regions */
+ task = bpf_get_current_task_btf();
+ bpf_rcu_read_lock();
+ bpf_rcu_read_lock();
+ real_parent = task->real_parent;
+ (void)bpf_task_storage_get(&map_a, real_parent, 0, 0);
+ // imbalance unlock()
+ bpf_rcu_read_unlock();
+ return 0;
+}