bpf: Add bpf_rcu_read_lock() support
Subsystems (7)
v8 cover bpf: Add bpf_rcu_read_lock() support
Currently, without rcu attribute info in BTF, the verifier treats rcu tagged pointer as a normal pointer. This might be a problem for sleepable program where rcu_read_lock()/unlock() is not available. For example, for a sleepable fentry program, if rcu protected memory access is interleaved with a sleepable helper/kfunc, it is possible the memory access after the sleepable helper/kfunc might be invalid since the object might have been freed then. Even without
a sleepable helper/kfunc, without rcu_read_lock() protection, it is possible that the rcu protected object might be release in the middle of bpf program execution which may cause incorrect result.
To prevent above cases, enable btf_type_tag("rcu") attributes, introduce new bpf_rcu_read_lock/unlock() kfuncs and add verifier support.
In the rest of patch set, Patch 1 enabled btf_type_tag for __rcu attribute. Patche 2 added might_sleep in bpf_func_proto. Patch 3 added new bpf_rcu_read_lock/unlock() kfuncs and verifier support.
Patch 4 added some tests for these two new kfuncs.
Changelogs:
v7 -> v8:
. add might_sleep in bpf_func_proto so we can easily identify whether
a helper is sleepable or not.
. do not enforce rcu rules for sleepable, e.g., rcu dereference must
be in a bpf_rcu_read_lock region. This is to keep old code working fine.
. Mark 'b' in 'b = a->b' (b is tagged with __rcu) as MEM_RCU only if
'b = a->b' in rcu read region and 'a' is trusted. This adds safety guarantee for 'b' inside the rcu read region.
v6 -> v7:
. rebase on top of bpf-next.
. remove the patch which enables sleepable program using
cgrp_local_storage map. This is orthogonal to this patch set and will be addressed separately.
. mark the rcu pointer dereference result as UNTRUSTED if inside
a bpf_rcu_read_lock() region.
v5 -> v6:
. fix selftest prog miss_unlock which tested nested locking. . add comments in selftest prog cgrp_succ to explain how to handle
nested memory access after rcu memory load.
v4 -> v5:
. add new test to aarch64 deny list.
v3 -> v4:
. fix selftest failures when built with gcc. gcc doesn't support
btf_type_tag yet and some tests relies on that. skip these tests if vmlinux BTF does not have btf_type_tag("rcu").
v2 -> v3:
. went back to MEM_RCU approach with invalidate rcu ptr registers
at bpf_rcu_read_unlock() place.
. remove KF_RCU_LOCK/UNLOCK flag and compare btf_id at verification
time instead.
v1 -> v2:
. use kfunc instead of helper for bpf_rcu_read_lock/unlock. . not use MEM_RCU bpf_type_flag, instead use active_rcu_lock
in reg state to identify rcu ptr's.
. Add more self tests.
. add new test to s390x deny list.
Yonghong Song (4):
compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))
bpf: Introduce might_sleep field in bpf_func_proto
bpf: Add kfunc bpf_rcu_read_lock/unlock()
selftests/bpf: Add tests for bpf_rcu_read_lock()
include/linux/bpf.h | 4 + include/linux/bpf_verifier.h | 4 +- include/linux/compiler_types.h | 3 +- kernel/bpf/bpf_lsm.c | 6 +- kernel/bpf/btf.c | 3 + kernel/bpf/helpers.c | 14 + kernel/bpf/verifier.c | 123 ++++++- kernel/trace/bpf_trace.c | 4 +- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/rcu_read_lock.c | 138 ++++++++ .../selftests/bpf/progs/rcu_read_lock.c | 306 ++++++++++++++++++ 12 files changed, 591 insertions(+), 16 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c create mode 100644 tools/testing/selftests/bpf/progs/rcu_read_lock.c
Signature · 1 line
2.30.2
PATCH v8 1/4 compiler_types: Define __rcu as __attribute__((btf_type_tag("rcu")))
Currently, without rcu attribute info in BTF, the verifier treats rcu tagged pointer as a normal pointer. This might be a problem for sleepable program where rcu_read_lock()/unlock() is not available. For example, for a sleepable fentry program, if rcu protected memory access is interleaved with a sleepable helper/kfunc, it is possible the memory access after the sleepable helper/kfunc might be invalid since the object might have been freed then. To prevent such cases, introducing rcu tagging for memory accesses in verifier can help to reject such programs.
To enable rcu tagging in BTF, during kernel compilation,
define __rcu as attribute btf_type_tag("rcu") so __rcu information can be preserved in dwarf and btf, and later can be used for bpf prog verification.
--- include/linux/compiler_types.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/compiler_types.h +2 -1
@@ -49,7 +49,8 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }# endif # define __iomem# define __percpu BTF_TYPE_TAG(percpu) -# define __rcu +# define __rcu BTF_TYPE_TAG(rcu) +# define __chk_user_ptr (x) (void)0# define __chk_io_ptr (x) (void)0/* context/locking */
Signature · 1 line
2.30.2
PATCH v8 2/4 bpf: Introduce might_sleep field in bpf_func_proto
In reply to Yonghong SongIntroduce bpf_func_proto->might_sleep to indicate a particular helper might sleep. This will make later check whether a helper might be sleepable or not easier.
--- include/linux/bpf.h | 1 + kernel/bpf/bpf_lsm.c | 6 ++++-- kernel/bpf/helpers.c | 2 ++ kernel/bpf/verifier.c | 5 +++++ kernel/trace/bpf_trace.c | 4 ++-- 5 files changed, 14 insertions(+), 4 deletions(-)
include/linux/bpf.h +1 -0
@@ -682,6 +682,7 @@ struct bpf_func_proto {u64 (*func)(u64 r1,u64 r2,u64 r3,u64 r4,u64 r5);bool gpl_only;bool pkt_access; +bool might_sleep;enum bpf_return_type ret_type; union { struct {
kernel/bpf/bpf_lsm.c +4 -2
@@ -151,6 +151,7 @@ BTF_ID_LIST_SINGLE(bpf_ima_inode_hash_btf_ids, struct, inode)static const struct bpf_func_proto bpf_ima_inode_hash_proto = { .func = bpf_ima_inode_hash, .gpl_only = false, + .might_sleep = true, .ret_type =RET_INTEGER , .arg1_type =ARG_PTR_TO_BTF_ID , .arg1_btf_id = &bpf_ima_inode_hash_btf_ids[0 ],
@@ -169,6 +170,7 @@ BTF_ID_LIST_SINGLE(bpf_ima_file_hash_btf_ids, struct, file)static const struct bpf_func_proto bpf_ima_file_hash_proto = { .func = bpf_ima_file_hash, .gpl_only = false, + .might_sleep = true, .ret_type =RET_INTEGER , .arg1_type =ARG_PTR_TO_BTF_ID , .arg1_btf_id = &bpf_ima_file_hash_btf_ids[0 ],
@@ -221,9 +223,9 @@ bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)case BPF_FUNC_bprm_opts_set:return &bpf_bprm_opts_set_proto;case BPF_FUNC_ima_inode_hash: -return prog->aux->sleepable ? &bpf_ima_inode_hash_proto :NULL ; +return &bpf_ima_inode_hash_proto;case BPF_FUNC_ima_file_hash: -return prog->aux->sleepable ? &bpf_ima_file_hash_proto :NULL ; +return &bpf_ima_file_hash_proto;case BPF_FUNC_get_attach_cookie:return bpf_prog_has_trampoline (prog) ? &bpf_get_attach_cookie_proto :NULL ;#ifdef CONFIG_NET
kernel/bpf/helpers.c +2 -0
@@ -660,6 +660,7 @@ BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,const struct bpf_func_proto bpf_copy_from_user_proto = { .func = bpf_copy_from_user, .gpl_only = false, + .might_sleep = true, .ret_type =RET_INTEGER , .arg1_type =ARG_PTR_TO_UNINIT_MEM , .arg2_type =ARG_CONST_SIZE_OR_ZERO ,
@@ -690,6 +691,7 @@ BPF_CALL_5(bpf_copy_from_user_task, void *, dst, u32, size,const struct bpf_func_proto bpf_copy_from_user_task_proto = { .func = bpf_copy_from_user_task, .gpl_only = true, + .might_sleep = true, .ret_type =RET_INTEGER , .arg1_type =ARG_PTR_TO_UNINIT_MEM , .arg2_type =ARG_CONST_SIZE_OR_ZERO ,
kernel/bpf/verifier.c +5 -0
@@ -7516,6 +7516,11 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnreturn -EINVAL ; } +if (!env->prog->aux->sleepable && fn->might_sleep) { +verbose (env,"helper call might sleep in a non-sleepable prog\n" ); +return -EINVAL ; + } +/* With LD_ABS/IND some JITs save/restore skb from r1. */ changes_data =bpf_helper_changes_pkt_data (fn->func);if (changes_data && fn->arg1_type !=ARG_PTR_TO_CTX ) {
kernel/trace/bpf_trace.c +2 -2
@@ -1485,9 +1485,9 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)case BPF_FUNC_get_task_stack:return &bpf_get_task_stack_proto;case BPF_FUNC_copy_from_user: -return prog->aux->sleepable ? &bpf_copy_from_user_proto :NULL ; +return &bpf_copy_from_user_proto;case BPF_FUNC_copy_from_user_task: -return prog->aux->sleepable ? &bpf_copy_from_user_task_proto :NULL ; +return &bpf_copy_from_user_task_proto;case BPF_FUNC_snprintf_btf:return &bpf_snprintf_btf_proto;case BPF_FUNC_per_cpu_ptr:
Signature · 1 line
2.30.2
PATCH v8 3/4 bpf: Add kfunc bpf_rcu_read_lock/unlock()
In reply to Yonghong SongAdd two kfunc's bpf_rcu_read_lock() and bpf_rcu_read_unlock(). These two kfunc's can be used for all program types. The following is an example about how rcu pointer are used w.r.t. bpf_rcu_read_lock()/bpf_rcu_read_unlock().
struct task_struct {
...
struct task_struct *last_wakee;
struct task_struct __rcu *real_parent;
...
};Let us say prog does 'task = bpf_get_current_task_btf()' to get a 'task' pointer. The basic rules are:
- 'real_parent = task->real_parent' should be inside bpf_rcu_read_lock region. this is to simulate rcu_dereference() operation. The 'real_parent' is marked as MEM_RCU only if (1). task->real_parent is inside bpf_rcu_read_lock region, and (2). task is a trusted ptr, marked with PTR_TRUSTED or ref_obj_id != 0. In other words, MEM_RCU marked ptr can be 'trusted' inside the bpf_rcu_read_lock region.
- 'last_wakee = real_parent->last_wakee' should be inside bpf_rcu_read_lock region since it tries to access rcu protected memory.
- the ptr 'last_wakee' will be marked as PTR_UNTRUSTED since in general it is not clear whether the object pointed by 'last_wakee' is valid or not even inside bpf_rcu_read_lock region.
To prevent rcu pointer leaks outside the rcu read lock region. The verifier will clear all rcu pointer register state to unknown, i.e., scalar_value, at bpf_rcu_read_unlock() kfunc call site,
so later dereference becomes impossible.
The current implementation does not support nested rcu read lock region in the prog.
--- include/linux/bpf.h | 3 + include/linux/bpf_verifier.h | 4 +- kernel/bpf/btf.c | 3 + kernel/bpf/helpers.c | 12 ++++ kernel/bpf/verifier.c | 118 ++++++++++++++++++++++++++++++++--- 5 files changed, 129 insertions(+), 11 deletions(-)
include/linux/bpf.h +3 -0
@@ -572,6 +572,9 @@ enum bpf_type_flag {*/ PTR_TRUSTED =BIT (12 +BPF_BASE_TYPE_BITS ), +/* MEM is tagged with rcu and memory access needs rcu_read_lock protection. */ +MEM_RCU =BIT (13 +BPF_BASE_TYPE_BITS ), + __BPF_TYPE_FLAG_MAX, __BPF_TYPE_LAST_FLAG = __BPF_TYPE_FLAG_MAX - 1, };
include/linux/bpf_verifier.h +3 -1
@@ -344,6 +344,7 @@ struct bpf_verifier_state {u32 id; } active_lock;bool speculative; +bool active_rcu_lock;/* first and last insn idx of this verifier state */ u32 first_insn_idx;
@@ -445,6 +446,7 @@ struct bpf_insn_aux_data {u32 seen;/* this insn was processed by the verifier at env->pass_cnt */ bool sanitize_stack_spill;/* subject to Spectre v4 sanitation */ bool zext_dst;/* this insn zero extends dst reg */ +bool storage_get_func_atomic;/* bpf_*_storage_get() with atomic memory alloc */ u8 alu_state;/* used in combination with alu_limit */ /* below fields are initialized once */
@@ -680,7 +682,7 @@ static inline bool bpf_prog_check_recur(const struct bpf_prog *prog) } } -#define BPF_REG_TRUSTED_MODIFIERS (MEM_ALLOC | PTR_TRUSTED) +#define BPF_REG_TRUSTED_MODIFIERS (MEM_ALLOC | MEM_RCU | PTR_TRUSTED)static inline bool bpf_type_has_unsafe_modifiers (u32 type) {
kernel/bpf/btf.c +3 -0
@@ -6237,6 +6237,9 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf,/* check __percpu tag */ if (strcmp (tag_value,"percpu" ) ==0 ) tmp_flag =MEM_PERCPU ; +/* check __rcu tag */ +if (strcmp (tag_value,"rcu" ) ==0 ) + tmp_flag =MEM_RCU ; } stype =btf_type_skip_modifiers (btf, mtype->type, &id);
kernel/bpf/helpers.c +12 -0
@@ -1891,6 +1891,16 @@ void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k)return obj__ign; } +void bpf_rcu_read_lock (void ) +{ +rcu_read_lock (); +} + +void bpf_rcu_read_unlock (void ) +{ +rcu_read_unlock (); +} +__diag_pop ();BTF_SET8_START (generic_btf_ids)
@@ -1921,6 +1931,8 @@ BTF_ID(func, bpf_task_release)BTF_SET8_START (common_btf_ids)BTF_ID_FLAGS (func ,bpf_cast_to_kern_ctx )BTF_ID_FLAGS (func, bpf_rdonly_cast) +BTF_ID_FLAGS (func, bpf_rcu_read_lock) +BTF_ID_FLAGS (func, bpf_rcu_read_unlock)BTF_SET8_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/verifier.c +108 -10
225 lines not on this page · raw message
Signature · 1 line
2.30.2
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · line 4763
Quoted 9 lines · Yonghong Song
kernel/bpf/verifier.c +9
-0+if (flag &MEM_RCU ) { +/* Mark value register as MEM_RCU only if it is protected by +* bpf_rcu_read_lock() and the ptr reg is trusted (PTR_TRUSTED or +* ref_obj_id != 0). MEM_RCU itself can already indicate +* trustedness inside the rcu read lock region. But Mark it +* as PTR_TRUSTED as well similar to MEM_ALLOC. +*/ +if (!env->cur_state->active_rcu_lock || + (!(reg->type &PTR_TRUSTED ) && !reg->ref_obj_id))
Can is_trusted_reg() be reused or MEM_ALLOC is not applicable here?
kernel/bpf/verifier.c static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · line 4771
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+ flag &= ~MEM_RCU ; + else + flag |= PTR_TRUSTED; + }else if (reg->type &MEM_RCU ) { +/* ptr (reg) is marked as MEM_RCU, but value reg is not marked as MEM_RCU. +* Mark the value reg as PTR_UNTRUSTED conservatively. +*/ + flag |=PTR_UNTRUSTED ;
Should PTR_UNTRUSTED tagging be limited to ret == PTR_TO_BTF_ID instead of tagging SCALAR also?
[ ... ]
kernel/bpf/verifier.c static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 11843
Quoted 5 lines
kernel/bpf/verifier.c +1
-0@@ -11754,6 +11840,11 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn)return -EINVAL ; } +if (env->prog->aux->sleepable && env->cur_state->active_rcu_lock) {
I don't know the details about ld_abs :). Why sleepable check is needed here?
Others lgtm.
Re: [PATCH bpf-next v8 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 12 lines · Martin KaFai Lau
Quoted 10 lines · Yonghong Song
+ if (flag &
MEM_RCU) {
+ /* Mark value register asMEM_RCUonly if it is protected by + *bpf_rcu_read_lock()and the ptr reg is trusted (PTR_TRUSTEDor
+ *ref_obj_id!= 0).MEM_RCUitself can already indicate + * trustedness inside the rcu read lock region. But Mark it + * asPTR_TRUSTEDas well similar toMEM_ALLOC.
+ */
+ if (!env->cur_state->active_rcu_lock ||
+ (!(reg->type &PTR_TRUSTED) && !reg->ref_obj_id))
Quoted 1 line
Can
is_trusted_reg()be reused orMEM_ALLOCis not applicable here?
Currently MEM_ALLOC is returned by bpf_obj_new() which takes prog btf. currently MEM_RCU is only enabled with btf_struct_access() for vmlinux btf. So if MEM_ALLOC is in reg->type, then MEM_RCU should not appear.
But I guess we could still use is_trusted_reg() here since it does cover the use case here.
Quoted 11 lines
Quoted 10 lines
+ flag &= ~MEM_RCU;
+ else
+ flag |=PTR_TRUSTED;
+ } else if (reg->type&MEM_RCU) {
+ /* ptr (reg) is marked asMEM_RCU, but value reg is not marked asMEM_RCU.
+ * Mark the value reg asPTR_UNTRUSTEDconservatively. + */
+ flag |=PTR_UNTRUSTED;
Quoted 1 line
Should
PTR_UNTRUSTEDtagging be limited to ret ==PTR_TO_BTF_IDinstead of tagging SCALAR also?
We should be okay here. flag is a local variable. It is used in below function when reg_type is not SCALAR_VALUE.
static void mark_btf_ld_reg(struct bpf_verifier_env *env,
struct bpf_reg_state *regs, u32 regno, enum bpf_reg_type reg_type, struct btf *btf, u32 btf_id, enum bpf_type_flag flag)
{
if (reg_type == SCALAR_VALUE) {
mark_reg_unknown(env, regs, regno);
return;
}
mark_reg_known_zero(env, regs, regno);
regs[regno].type = PTR_TO_BTF_ID | flag;
regs[regno].btf = btf;
regs[regno].btf_id = btf_id;
}Quoted 10 lines
[ ... ]
kernel/bpf/verifier.c static int check_ld_abs(struct · line 11843
Quoted 5 lines
kernel/bpf/verifier.c +1
-0@@ -11754,6 +11840,11 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return -EINVAL; } + if (env->prog->aux->sleepable && env->cur_state->active_rcu_lock) {I don't know the details about
ld_abs:). Why sleepable check is needed here?
Do we still care about ld_abs??
Actually I added this since spin_lock excludes this. But taking a deep
look at the function convert_bpf_ld_abs, it is converted to a bunch of
bpf insns and bpf_skb_load_helper_{8,16,32}() which eventually callsskb_copy_bits(). Checking skb_copy_bits(), it seems the function is not sleepable. Will remove this in the next revision.
Quoted 2 lines
Others lgtm.
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 11 lines · Yonghong Song
We should be okay here. flag is a local variable. It is used in below function when
reg_typeis notSCALAR_VALUE.static void mark_btf_ld_reg(struct
bpf_verifier_env*env,
structbpf_reg_state*regs, u32 regno, enumbpf_reg_typereg_type, struct btf *btf, u32btf_id, enumbpf_type_flagflag)
{
if (reg_type==SCALAR_VALUE) {mark_reg_unknown(env, regs, regno);
Ah, got it.
Quoted 11 lines
Quoted 7 lines
kernel/bpf/verifier.c static int check_ld_abs(struct bpf_verifier_env · line 11843
Quoted 5 lines
kernel/bpf/verifier.c +1
-0@@ -11754,6 +11840,11 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return -EINVAL; } + if (env->prog->aux->sleepable && env->cur_state->active_rcu_lock) {I don't know the details about
ld_abs:). Why sleepable check is needed here?Do we still care about
ld_abs??Actually I added this since
spin_lockexcludes this. But taking a deep
From looking at check_ld_abs() again, I just noticed this comment:
/* Disallow usage of BPF_LD_[ABS|IND] with reference tracking, as
* gen_ld_abs() may terminate the program at runtime, leading to
* reference leak.
*/
I think active_rcu_lock should be tested. My question was more on why the env->prog->aux->sleepable test is also needed.
Re: [PATCH bpf-next v8 3/4] bpf: Add kfunc bpf_rcu_read_lock/unlock()
Quoted 28 lines · Martin KaFai Lau
Quoted 11 lines · Yonghong Song
We should be okay here. flag is a local variable. It is used in below function when
reg_typeis notSCALAR_VALUE.static void mark_btf_ld_reg(struct
bpf_verifier_env*env, structbpf_reg_state*regs, u32 regno, enumbpf_reg_typereg_type, struct btf *btf, u32btf_id, enumbpf_type_flagflag)
{
if (reg_type==SCALAR_VALUE) {mark_reg_unknown(env, regs, regno);Ah, got it.
Quoted 9 lines
Quoted 8 lines
Review static int check_ld_abs(struct · line 11844
Quoted 6 lines
(quoted diff) +1
-0@@ -11754,6 +11840,11 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return -EINVAL; } + if (env->prog->aux->sleepable && env->cur_state->active_rcu_lock) {I don't know the details about
ld_abs:). Why sleepable check is needed here?Quoted 3 lines
Do we still care about
ld_abs??Actually I added this since
spin_lockexcludes this. But taking a deep
Quoted 8 lines
From looking at
check_ld_abs()again, I just noticed this comment:/* Disallow usage of BPF_LD_[ABS|IND] with reference tracking, as
*gen_ld_abs()may terminate the program at runtime, leading to
* reference leak.
*/I think
active_rcu_lockshould be tested. My question was more on why theenv->prog->aux->sleepabletest is also needed.
Will remove env->prog->aux->sleepable in the next version. It is a leftover missed with v8 not to focus on sleepable aspect of the lock.
PATCH v8 4/4 selftests/bpf: Add tests for bpf_rcu_read_lock()
In reply to Yonghong SongAdd a few positive/negative tests to test bpf_rcu_read_lock() and its corresponding verifier support. The new test will fail on s390x and aarch64, so an entry is added to each of their respective deny lists.
--- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/rcu_read_lock.c | 138 ++++++++ .../selftests/bpf/progs/rcu_read_lock.c | 306 ++++++++++++++++++ 4 files changed, 446 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c create mode 100644 tools/testing/selftests/bpf/progs/rcu_read_lock.c
tools/testing/selftests/bpf/DENYLIST.aarch64 +1 -0
@@ -45,6 +45,7 @@ modify_return # modify_return__attach failed module_attach # skel_attach skeleton attach failed: -524 mptcp/base # run_test mptcp unexpected error: -524 (errno 524) netcnt # packets unexpected packets: actual 10001 != expected 10000 +rcu_read_lock # failed to attach: ERROR: strerror_r(-524)=22 recursion # skel_attach unexpected error: -524 (errno 524) ringbuf # skel_attach skeleton attachment failed: -1 setget_sockopt # attach_cgroup unexpected error: -524
tools/testing/selftests/bpf/DENYLIST.s390x +1 -0
@@ -42,6 +42,7 @@ module_attach # skel_attach skeleton attach failed: - mptcp netcnt # failed to load BPF skeleton 'netcnt_prog': -7 (?) probe_user # check_kprobe_res wrong kprobe res from probe read (?) +rcu_read_lock # failed to find kernel BTF type ID of '__x64_sys_getpgid': -3 (?) recursion # skel_attach unexpected error: -524 (trampoline) ringbuf # skel_load skeleton load failed (?) select_reuseport # intermittently fails on new s390x setup
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c new file +138 -0
138 lines not on this page · raw message
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +306 -0
306 lines not on this page · raw message
Signature · 1 line
2.30.2
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 81
Quoted 10 lines · Yonghong Song
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int task_acquire (void *ctx) +{ +struct task_struct *task, *real_parent; + + task =bpf_get_current_task_btf (); +bpf_rcu_read_lock (); + real_parent = task->real_parent; +/* acquire a reference which can be used outside rcu read lock region */ + real_parent =bpf_task_acquire (real_parent);
Does the bpf_task_acquire() kfunc need a change to do refcount_inc_not_zero() and KF_RET_NULL?
Also, some more 'skip' checks in prog_tests/rcu_read_lock.c is needed for gcc. This is failing in gcc CI:
https://github.com/kernel-patches/bpf/actions/runs/3527747280/jobs/5917628248#step:6:5624
; bpf_rcu_read_lock();
2: (85) call bpf_rcu_read_lock#26650
; real_parent = task->real_parent;
3: (79) r1 = *(u64 *)(r6 +1416) ; R1_w=ptr_task_struct(off=0,imm=0)
R6_w=trusted_ptr_task_struct(off=0,imm=0)
; real_parent = bpf_task_acquire(real_parent);
4: (85) call bpf_task_acquire#26666
R1 must be referenced or trusted
processed 5 insns (limit 1000000) max_states_per_insn 0 total_states 0peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'task_acquire': failed to load: -22
Quoted 5 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +5
-0+bpf_rcu_read_unlock (); + (void )bpf_task_storage_get (&map_a, real_parent,0 ,0 ); +bpf_task_release (real_parent); +return 0 ; +}
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 96
Quoted 10 lines
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+ +SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int no_lock (void *ctx) +{ +struct task_struct *task, *real_parent; + +/* no bpf_rcu_read_lock(), old code still works */ + task =bpf_get_current_task_btf (); + real_parent = task->real_parent; +bpf_printk ("pid %u\n" , real_parent->pid);
nit. Can bpf_printk be avoided here?
Others lgtm.
Re: [PATCH bpf-next v8 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
Quoted 13 lines · Martin KaFai Lau
Quoted 11 lines · Yonghong Song
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +9
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int task_acquire (void *ctx) +{ + struct task_struct *task, *real_parent; + + task = bpf_get_current_task_btf(); + bpf_rcu_read_lock(); + real_parent = task->real_parent; + /* acquire a reference which can be used outside rcu read lockregion */
+real_parent=bpf_task_acquire(real_parent);Does the
bpf_task_acquire()kfunc need a change to dorefcount_inc_not_zero()andKF_RET_NULL?
We have this definition in kernel:BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS)
So the argument is trusted args so, either marked as PTR_TRUSTED/MEM_ALLOC or have a reference acquired already, so I guess we should be fine here.
Quoted 5 lines
Also, some more 'skip' checks in
prog_tests/rcu_read_lock.cis needed for gcc. This is failing in gcc CI:https://github.com/kernel-patches/bpf/actions/runs/3527747280/jobs/5917628248#step:6:5624
Quoted 12 lines
;
bpf_rcu_read_lock();
2: (85) call bpf_rcu_read_lock#26650
;real_parent=task->real_parent;
3: (79) r1 = *(u64 *)(r6 +1416) ;
R1_w=ptr_task_struct(off=0,imm=0) R6_w=trusted_ptr_task_struct(off=0,imm=0)
;real_parent=bpf_task_acquire(real_parent);
4: (85) call bpf_task_acquire#26666
R1 must be referenced or trusted
processed 5 insns (limit 1000000)max_states_per_insn0total_states
0peak_states0mark_read0
-- END PROG LOAD LOG --
libbpf: prog 'task_acquire': failed to load: -22
Yes, we should skip this for gcc compiled kernel since rcu tag is not available.
Quoted 10 lines
Quoted 10 lines
+
bpf_rcu_read_unlock();
+ (void)bpf_task_storage_get(&map_a,real_parent, 0, 0); +bpf_task_release(real_parent);
+ return 0;tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +5
-0+} + +SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int no_lock (void *ctx) +{
Quoted 8 lines
Quoted 6 lines
+ struct
task_struct*task,*real_parent;
+
+ /* nobpf_rcu_read_lock(), old code still works */ + task =bpf_get_current_task_btf();
+real_parent=task->real_parent;
+bpf_printk("pid %u\n", real_parent->pid);nit. Can
bpf_printkbe avoided here?
I could add a target_pid comparison to prevent the issue. But will follow your suggestion to use a different function instead of bpf_printk.
Quoted 3 lines
Others lgtm.
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Martin KaFai Lau
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 81
Quoted 10 lines · Yonghong Song
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int task_acquire (void *ctx) +{ + struct task_struct *task, *real_parent; + + task = bpf_get_current_task_btf(); + bpf_rcu_read_lock(); + real_parent = task->real_parent; + /* acquire a reference which can be used outside rcu read lock region */ + real_parent = bpf_task_acquire(real_parent);Does the
bpf_task_acquire()kfunc need a change to dorefcount_inc_not_zero()andKF_RET_NULL?
Quoted 5 lines
We have this definition in kernel:
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS)So the argument is trusted args so, either marked as PTR_TRUSTED/MEM_ALLOC or have a reference acquired already, so I guess we should be fine here.
The verifier part is fine on {KF_TRUSTED_ARGS, PTR_TRUSTED}.
iiuc, PTR_TRUSTED means the kfunc can safely dereference the pointer because the ptr has not been freed yet but does not mean its refcnt > 0 and not on its way to be freed after the rcu gp.
If real_parent's refcnt is 0 here, bpf_task_acquire() will resurrect a task which is on its way to be freed and the task can be stored in a map, so a UAF.
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
Quoted 24 lines · Martin KaFai Lau
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Martin KaFai Lau
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 81
Quoted 10 lines · Yonghong Song
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int task_acquire (void *ctx) +{ + struct task_struct *task, *real_parent; + + task = bpf_get_current_task_btf(); + bpf_rcu_read_lock(); + real_parent = task->real_parent; + /* acquire a reference which can be used outside rcu read lock region */ + real_parent = bpf_task_acquire(real_parent);Does the
bpf_task_acquire()kfunc need a change to dorefcount_inc_not_zero()andKF_RET_NULL?Quoted 5 lines
We have this definition in kernel:
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS)So the argument is trusted args so, either marked as PTR_TRUSTED/MEM_ALLOC or have a reference acquired already, so I guess we should be fine here.
Quoted 5 lines
The verifier part is fine on {
KF_TRUSTED_ARGS,PTR_TRUSTED}.iiuc,
PTR_TRUSTEDmeans the kfunc can safely dereference the pointer because the ptr has not been freed yet but does not mean its refcnt > 0 and not on its way to be freed after the rcu gp.If real_parent's refcnt is 0 here,
bpf_task_acquire()will resurrect a task which is on its way to be freed and the task can be stored in a map, so a UAF.
This could be addressed as a follow up though since it is not specific to this set.
A message that is not in the archive (sent off-list or not yet received): SN6PR1501MB20649F820B6FFE58166817E5CA0C9@SN6PR1501MB2064.namprd15.prod.outlook.com
Martin KaFai Lau
reviewerRe: [PATCH bpf-next v8 4/4] selftests/bpf: Add tests for bpf_rcu_read_lock()
Quoted 36 lines · Yonghong Song
From: Martin KaFai Lau <[email protected]>
Date: Tuesday, November 22, 2022 at 5:53 PM
To: Yonghong Song <[email protected]>
Cc: Alexei Starovoitov <[email protected]>, Andrii Nakryiko <[email protected]>, Daniel Borkmann <[email protected]>, Kernel Team <[email protected]>, Martin KaFai Lau <[email protected]>, [email protected] <[email protected]>
Subject: Re: [PATCH bpf-next v8 4/4] selftests/bpf: Add tests forbpf_rcu_read_lock()Quoted 28 lines · Martin KaFai Lau
Quoted 17 lines · Yonghong Song
Quoted 13 lines · Martin KaFai Lau
tools/testing/selftests/bpf/progs/rcu_read_lock.c line 81
Quoted 10 lines · Yonghong Song
tools/testing/selftests/bpf/progs/rcu_read_lock.c new file +10
-0+SEC("?fentry.s/" SYS_PREFIX "sys_nanosleep") +int task_acquire (void *ctx) +{ +struct task_struct *task, *real_parent; + + task =bpf_get_current_task_btf (); +bpf_rcu_read_lock (); + real_parent = task->real_parent; +/* acquire a reference which can be used outside rcu read lock region */ + real_parent =bpf_task_acquire (real_parent);Does the
bpf_task_acquire()kfunc need a change to dorefcount_inc_not_zero()andKF_RET_NULL?Quoted 6 lines
We have this definition in kernel:
BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_TRUSTED_ARGS)So the argument is trusted args so, either marked as PTR_TRUSTED/MEM_ALLOC or have a reference acquired already, so
I guess we should be fine here.The verifier part is fine on {
KF_TRUSTED_ARGS,PTR_TRUSTED}.
Quoted 7 lines
Quoted 6 lines
iiuc,
PTR_TRUSTEDmeans the kfunc can safely dereference the pointer because the ptr has not been freed yet but does not mean its refcnt > 0 and not on its way to be freed after the rcu gp.If real_parent's refcnt is 0 here,
bpf_task_acquire()will resurrect a task which is on its way to be freed and the task can be stored in a map, so a UAF.I see. Maybe we need strong trusted vs. weak trusted variants. Strong trusted means refcnt > 0 and weak means no guarantee? Or we consider everything as week and tries to grab a reference anyway? In most if not all cases, ‘current’ should represent a strong trusted
btf_idI guess.
yeah, "current" task here is fine. current->real_parent is questionable.
imo, I think this check may be better done in runtime. The bpf_*_acquire() kfunc should always do refcount_inc_not_zero() + KF_RET_NULL. Otherwise, it may end up requiring to tag which ctx has a zero/non-zero refcnt. eg. the security_sk_alloc() hook, the sk's refcnt is 0 and later the kernel does a refcount_set(&sk->sk_refcnt, 1).
Quoted 4 lines
Quoted 2 lines
This could be addressed as a follow up though since it is not specific to this set.
Right, we have the same potential problem for both task and cgroup acquire functions.