Handle possible NULL trusted raw_tp arguments
Subsystems (5)
Kumar Kartikeya Dwivedi
maintainerv2 cover Handle possible NULL trusted raw_tp arguments
More context is available in [0], but the TLDR; is that the verifier incorrectly assumes that any raw tracepoint argument will always be non-NULL. This means that even when users correctly check possible NULL arguments, the verifier can remove the NULL check due to incorrect knowledge of the NULL-ness of the pointer. Secondly, kernel helpers or kfuncs taking these trusted tracepoint arguments incorrectly assume that all arguments will always be valid non-NULL.
In this set, we mark raw_tp arguments as PTR_MAYBE_NULL on top of PTR_TRUSTED, but special case their behavior when dereferencing them or pointer arithmetic over them is involved. When passing trusted args to helpers or kfuncs, raw_tp programs are permitted to pass possibly NULL pointers in such cases.
Any loads into such maybe NULL trusted PTR_TO_BTF_ID is promoted to a PROBE_MEM load to handle emanating page faults. The verifier will ensure NULL checks on such pointers are preserved and do not lead to dead code elimination.
This new behavior is not applied when ref_obj_id is non-zero, as those pointers do not belong to raw_tp arguments, but instead acquired objects.
Since helpers and kfuncs already require attention for PTR_TO_BTF_ID (non-trusted) pointers, we do not implement any protection for such cases in this patch set, and leave it as future work for an upcoming series.
A selftest is included with this patch set to verify the new behavior, and it crashes the kernel without the first patch.
[0]: https://lore.kernel.org/bpf/CAADnVQLMPPavJQR6JFsi3dtaaLHB816JN4HCV_TFWohJ61D+wQ@mail.gmail.com
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
* Add patch to clean up users of gettid (Andrii)
* Avoid nested blocks in sefltest (Andrii)
* Prevent code motion optimization in selftest using barrier()
Kumar Kartikeya Dwivedi (3):
bpf: Mark raw_tp arguments with PTR_MAYBE_NULL selftests/bpf: Clean up open-coded gettid syscall invocations selftests/bpf: Add tests for raw_tp null handling
include/linux/bpf.h | 6 ++ kernel/bpf/btf.c | 5 +- kernel/bpf/verifier.c | 75 +++++++++++++++++-- .../selftests/bpf/benchs/bench_trigger.c | 3 +- .../bpf/bpf_testmod/bpf_testmod-events.h | 8 ++ .../selftests/bpf/bpf_testmod/bpf_testmod.c | 2 + tools/testing/selftests/bpf/bpf_util.h | 9 +++ .../bpf/map_tests/task_storage_map.c | 3 +- .../selftests/bpf/prog_tests/bpf_cookie.c | 2 +- .../selftests/bpf/prog_tests/bpf_iter.c | 6 +- .../bpf/prog_tests/cgrp_local_storage.c | 10 +-- .../selftests/bpf/prog_tests/core_reloc.c | 2 +- .../selftests/bpf/prog_tests/linked_funcs.c | 2 +- .../bpf/prog_tests/ns_current_pid_tgid.c | 2 +- .../selftests/bpf/prog_tests/raw_tp_null.c | 25 +++++++ .../selftests/bpf/prog_tests/rcu_read_lock.c | 4 +- .../bpf/prog_tests/task_local_storage.c | 10 +-- .../bpf/prog_tests/uprobe_multi_test.c | 2 +- .../testing/selftests/bpf/progs/raw_tp_null.c | 32 ++++++++ .../bpf/progs/test_tp_btf_nullable.c | 6 +- 20 files changed, 183 insertions(+), 31 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/raw_tp_null.c create mode 100644 tools/testing/selftests/bpf/progs/raw_tp_null.c
base-commit: e626a13f6fbb4697f8734333432dca577628d09a
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v2 1/3 bpf: Mark raw_tp arguments with PTR_MAYBE_NULL
Arguments to a raw tracepoint are tagged as trusted, which carries the semantics that the pointer will be non-NULL. However, in certain cases, a raw tracepoint argument may end up being NULL. More context about this issue is available in [0].
Thus, there is a discrepancy between the reality, that raw_tp arguments can actually be NULL, and the verifier's knowledge, that they are never NULL, causing explicit NULL checks to be deleted, and accesses to such pointers potentially crashing the kernel.
To fix this, mark raw_tp arguments as PTR_MAYBE_NULL, and then special case the dereference and pointer arithmetic to permit it, and allow passing them into helpers/kfuncs; these exceptions are made for raw_tp programs only. Ensure that we don't do this when ref_obj_id > 0, as in that case this is an acquired object and doesn't need such adjustment.
The reason we do mask_raw_tp_trusted_reg logic is because other will recheck in places whether the register is a trusted_reg, and then consider our register as untrusted when detecting the presence of the PTR_MAYBE_NULL flag.
To allow safe dereference, we enable PROBE_MEM marking when we see loads into trusted pointers with PTR_MAYBE_NULL.
While trusted raw_tp arguments can also be passed into helpers or kfuncs where such broken assumption may cause issues, a future patch set will tackle their case separately, as PTR_TO_BTF_ID (without PTR_TRUSTED) can already be passed into helpers and causes similar problems. Thus, they are left alone for now.
It is possible that these checks also permit passing non-raw_tp args that are trusted PTR_TO_BTF_ID with null marking. In such a case, allowing dereference when pointer is NULL expands allowed behavior, so won't regress existing programs, and the case of passing these into helpers is the same as above and will be dealt with later.
Also update the failure case in tp_btf_nullable selftest to capture the new behavior, as the verifier will no longer cause an error when directly dereference a raw tracepoint argument marked as __nullable.
[0]: https://lore.kernel.org/bpf/[email protected]
--- include/linux/bpf.h | 6 ++ kernel/bpf/btf.c | 5 +- kernel/bpf/verifier.c | 75 +++++++++++++++++-- .../bpf/progs/test_tp_btf_nullable.c | 6 +- 4 files changed, 83 insertions(+), 9 deletions(-)
include/linux/bpf.h +6 -0
@@ -3495,4 +3495,10 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) return prog->aux->func_idx != 0; } +static inline bool bpf_prog_is_raw_tp (const struct bpf_prog *prog) +{ +return prog->type ==BPF_PROG_TYPE_TRACING && + prog->expected_attach_type ==BPF_TRACE_RAW_TP ; +} +#endif /* _LINUX_BPF_H */
kernel/bpf/btf.c +4 -1
@@ -6588,7 +6588,10 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,if (prog_args_trusted (prog)) info->reg_type |=PTR_TRUSTED ; -if (btf_param_match_suffix (btf, &args[arg],"__nullable" )) +/* Raw tracepoint arguments always get marked as maybe NULL */ +if (bpf_prog_is_raw_tp (prog)) + info->reg_type |=PTR_MAYBE_NULL ; +else if (btf_param_match_suffix (btf, &args[arg],"__nullable" )) info->reg_type |=PTR_MAYBE_NULL ;if (tgt_prog) {
kernel/bpf/verifier.c +68 -7
219 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_tp_btf_nullable.c +5 -1
@@ -7,7 +7,11 @@#include "bpf_misc.h" SEC ("tp_btf/bpf_testmod_test_nullable_bare" ) -__failure __msg("R1 invalid mem access 'trusted_ptr_or_null_'") +/* This used to be a failure test, but raw_tp nullable arguments can now +* directly be dereferenced, whether they have nullable annotation or not, +* and don't need to be explicitly checked. + */ +__success int BPF_PROG(handle_tp_btf_nullable_bare1, struct bpf_testmod_test_read_ctx *nullable_ctx) {return nullable_ctx->len;
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 1/3] bpf: Mark raw_tp arguments with PTR_MAYBE_NULL
Quoted 157 lines · Kumar Kartikeya Dwivedi
Arguments to a raw tracepoint are tagged as trusted, which carries the semantics that the pointer will be non-NULL. However, in certain cases, a raw tracepoint argument may end up being NULL. More context about this issue is available in [0].
Thus, there is a discrepancy between the reality, that
raw_tparguments can actually be NULL, and the verifier's knowledge, that they are never NULL, causing explicit NULL checks to be deleted, and accesses to such pointers potentially crashing the kernel.To fix this, mark
raw_tparguments asPTR_MAYBE_NULL, and then special case the dereference and pointer arithmetic to permit it, and allow passing them into helpers/kfuncs; these exceptions are made forraw_tpprograms only. Ensure that we don't do this whenref_obj_id> 0, as in that case this is an acquired object and doesn't need such adjustment.The reason we do
mask_raw_tp_trusted_reglogic is because other will recheck in places whether the register is atrusted_reg, and then consider our register as untrusted when detecting the presence of thePTR_MAYBE_NULLflag.To allow safe dereference, we enable
PROBE_MEMmarking when we see loads into trusted pointers withPTR_MAYBE_NULL.While trusted
raw_tparguments can also be passed into helpers or kfuncs where such broken assumption may cause issues, a future patch set will tackle their case separately, asPTR_TO_BTF_ID(withoutPTR_TRUSTED) can already be passed into helpers and causes similar problems. Thus, they are left alone for now.It is possible that these checks also permit passing non-raw_tp args that are trusted
PTR_TO_BTF_IDwith null marking. In such a case, allowing dereference when pointer is NULL expands allowed behavior, so won't regress existing programs, and the case of passing these into helpers is the same as above and will be dealt with later.Also update the failure case in
tp_btf_nullableselftest to capture the new behavior, as the verifier will no longer cause an error when directly dereference a raw tracepoint argument marked as__nullable.[0]: https://lore.kernel.org/bpf/[email protected]
Reported-by Juri Lelli <[email protected]>Fixes 3f00c5239344 ("bpf: Allow trusted pointers to be passed to KF_TRUSTED_ARGS kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 6 ++ kernel/bpf/btf.c | 5 +- kernel/bpf/verifier.c | 75 +++++++++++++++++-- .../bpf/progs/test_tp_btf_nullable.c | 6 +- 4 files changed, 83 insertions(+), 9 deletions(-)include/linux/bpf.h +6
-0@@ -3495,4 +3495,10 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) return prog->aux->func_idx != 0; } +static inline bool bpf_prog_is_raw_tp (const struct bpf_prog *prog) +{ +return prog->type ==BPF_PROG_TYPE_TRACING && + prog->expected_attach_type ==BPF_TRACE_RAW_TP ; +} +#endif /* _LINUX_BPF_H */ kernel/bpf/btf.c +4
-1@@ -6588,7 +6588,10 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,if (prog_args_trusted (prog)) info->reg_type |=PTR_TRUSTED ; -if (btf_param_match_suffix (btf, &args[arg],"__nullable" )) +/* Raw tracepoint arguments always get marked as maybe NULL */ +if (bpf_prog_is_raw_tp (prog)) + info->reg_type |=PTR_MAYBE_NULL ; +else if (btf_param_match_suffix (btf, &args[arg],"__nullable" )) info->reg_type |=PTR_MAYBE_NULL ;if (tgt_prog) {kernel/bpf/verifier.c +37
-2@@ -418,6 +418,21 @@ static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg)return rec; } +static bool mask_raw_tp_reg (const struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +if (reg->type != (PTR_TO_BTF_ID |PTR_TRUSTED |PTR_MAYBE_NULL ) || + !bpf_prog_is_raw_tp (env->prog) || reg->ref_obj_id) +return false; + reg->type &= ~PTR_MAYBE_NULL ; +return true; +} + +static void unmask_raw_tp_reg (struct bpf_reg_state *reg,bool result) +{ +if (result) + reg->type |=PTR_MAYBE_NULL ; +} +static bool subprog_is_global (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info_aux *aux = env->prog->aux->func_info_aux;@@ -6622,6 +6637,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,const char *field_name =NULL ;enum bpf_type_flag flag =0 ;u32 btf_id =0 ; +bool mask;int ret;if (!env->allow_ptr_leaks) {@@ -6693,7 +6709,21 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,if (ret <0 )return ret; - +/* For raw_tp progs, we allow dereference of PTR_MAYBE_NULL +* trusted PTR_TO_BTF_ID, these are the ones that are possibly +* arguments to the raw_tp. Since internal checks in for trusted +* reg in check_ptr_to_btf_access would consider PTR_MAYBE_NULL +* modifier as problematic, mask it out temporarily for the +* check. Don't apply this to pointers with ref_obj_id > 0, as +* those won't be raw_tp args. +* +* We may end up applying this relaxation to other trusted +* PTR_TO_BTF_ID with maybe null flag, since we cannot +* distinguish PTR_MAYBE_NULL tagged for arguments vs normal +* tagging, but that should expand allowed behavior, and not +* cause regression for existing behavior. +*/ + mask =mask_raw_tp_reg (env, reg);if (ret !=PTR_TO_BTF_ID ) {/* just mark; */ @@ -6754,8 +6784,13 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,clear_trusted_flags (&flag); } -if (atype ==BPF_READ && value_regno >=0 ) +if (atype ==BPF_READ && value_regno >=0 ) {mark_btf_ld_reg (env, regs, value_regno, ret, reg->btf, btf_id, flag); +/* We've assigned a new type to regno, so don't undo masking. */ +if (regno == value_regno) + mask = false; + } +unmask_raw_tp_reg (reg, mask);return 0 ; }
kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 7178
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -7140,7 +7175,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (!err && t ==BPF_READ && value_regno >=0 )mark_reg_unknown (env, regs, value_regno); }else if (base_type (reg->type) ==PTR_TO_BTF_ID && - !type_may_be_null (reg->type)) { + (bpf_prog_is_raw_tp (env->prog) || !type_may_be_null (reg->type))) {
While looking at this again, I'm wondering if this check is too relaxed. Since we're matching on base_type, this will probably end up allowing any PTR_TO_BTF_ID with PTR_MAYBE_NULL in raw_tp progs. So I think I need to narrow it down, unless I'm missing something.
Will wait for a while for comments before respinning v3.
Quoted 3 lines
kernel/bpf/verifier.c +0
-0err =check_ptr_to_btf_access (env, regs, regno, off, size, t, value_regno);
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v2 1/3] bpf: Mark raw_tp arguments with PTR_MAYBE_NULL
Quoted 170 lines · Kumar Kartikeya Dwivedi
Quoted 157 lines · Kumar Kartikeya Dwivedi
Arguments to a raw tracepoint are tagged as trusted, which carries the semantics that the pointer will be non-NULL. However, in certain cases, a raw tracepoint argument may end up being NULL. More context about this issue is available in [0].
Thus, there is a discrepancy between the reality, that
raw_tparguments can actually be NULL, and the verifier's knowledge, that they are never NULL, causing explicit NULL checks to be deleted, and accesses to such pointers potentially crashing the kernel.To fix this, mark
raw_tparguments asPTR_MAYBE_NULL, and then special case the dereference and pointer arithmetic to permit it, and allow passing them into helpers/kfuncs; these exceptions are made forraw_tpprograms only. Ensure that we don't do this whenref_obj_id> 0, as in that case this is an acquired object and doesn't need such adjustment.The reason we do
mask_raw_tp_trusted_reglogic is because other will recheck in places whether the register is atrusted_reg, and then consider our register as untrusted when detecting the presence of thePTR_MAYBE_NULLflag.To allow safe dereference, we enable
PROBE_MEMmarking when we see loads into trusted pointers withPTR_MAYBE_NULL.While trusted
raw_tparguments can also be passed into helpers or kfuncs where such broken assumption may cause issues, a future patch set will tackle their case separately, asPTR_TO_BTF_ID(withoutPTR_TRUSTED) can already be passed into helpers and causes similar problems. Thus, they are left alone for now.It is possible that these checks also permit passing non-raw_tp args that are trusted
PTR_TO_BTF_IDwith null marking. In such a case, allowing dereference when pointer is NULL expands allowed behavior, so won't regress existing programs, and the case of passing these into helpers is the same as above and will be dealt with later.Also update the failure case in
tp_btf_nullableselftest to capture the new behavior, as the verifier will no longer cause an error when directly dereference a raw tracepoint argument marked as__nullable.[0]: https://lore.kernel.org/bpf/[email protected]
Reported-by Juri Lelli <[email protected]>Fixes 3f00c5239344 ("bpf: Allow trusted pointers to be passed to KF_TRUSTED_ARGS kfuncs")Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- include/linux/bpf.h | 6 ++ kernel/bpf/btf.c | 5 +- kernel/bpf/verifier.c | 75 +++++++++++++++++-- .../bpf/progs/test_tp_btf_nullable.c | 6 +- 4 files changed, 83 insertions(+), 9 deletions(-)include/linux/bpf.h +6
-0@@ -3495,4 +3495,10 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) return prog->aux->func_idx != 0; } +static inline bool bpf_prog_is_raw_tp (const struct bpf_prog *prog) +{ +return prog->type ==BPF_PROG_TYPE_TRACING && + prog->expected_attach_type ==BPF_TRACE_RAW_TP ; +} +#endif /* _LINUX_BPF_H */ kernel/bpf/btf.c +4
-1@@ -6588,7 +6588,10 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,if (prog_args_trusted (prog)) info->reg_type |=PTR_TRUSTED ; -if (btf_param_match_suffix (btf, &args[arg],"__nullable" )) +/* Raw tracepoint arguments always get marked as maybe NULL */ +if (bpf_prog_is_raw_tp (prog)) + info->reg_type |=PTR_MAYBE_NULL ; +else if (btf_param_match_suffix (btf, &args[arg],"__nullable" )) info->reg_type |=PTR_MAYBE_NULL ;if (tgt_prog) {kernel/bpf/verifier.c +37
-2@@ -418,6 +418,21 @@ static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg)return rec; } +static bool mask_raw_tp_reg (const struct bpf_verifier_env *env,struct bpf_reg_state *reg) +{ +if (reg->type != (PTR_TO_BTF_ID |PTR_TRUSTED |PTR_MAYBE_NULL ) || + !bpf_prog_is_raw_tp (env->prog) || reg->ref_obj_id) +return false; + reg->type &= ~PTR_MAYBE_NULL ; +return true; +} + +static void unmask_raw_tp_reg (struct bpf_reg_state *reg,bool result) +{ +if (result) + reg->type |=PTR_MAYBE_NULL ; +} +static bool subprog_is_global (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info_aux *aux = env->prog->aux->func_info_aux;@@ -6622,6 +6637,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,const char *field_name =NULL ;enum bpf_type_flag flag =0 ;u32 btf_id =0 ; +bool mask;int ret;if (!env->allow_ptr_leaks) {@@ -6693,7 +6709,21 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,if (ret <0 )return ret; - +/* For raw_tp progs, we allow dereference of PTR_MAYBE_NULL +* trusted PTR_TO_BTF_ID, these are the ones that are possibly +* arguments to the raw_tp. Since internal checks in for trusted +* reg in check_ptr_to_btf_access would consider PTR_MAYBE_NULL +* modifier as problematic, mask it out temporarily for the +* check. Don't apply this to pointers with ref_obj_id > 0, as +* those won't be raw_tp args. +* +* We may end up applying this relaxation to other trusted +* PTR_TO_BTF_ID with maybe null flag, since we cannot +* distinguish PTR_MAYBE_NULL tagged for arguments vs normal +* tagging, but that should expand allowed behavior, and not +* cause regression for existing behavior. +*/ + mask =mask_raw_tp_reg (env, reg);if (ret !=PTR_TO_BTF_ID ) {/* just mark; */ @@ -6754,8 +6784,13 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env,clear_trusted_flags (&flag); } -if (atype ==BPF_READ && value_regno >=0 ) +if (atype ==BPF_READ && value_regno >=0 ) {mark_btf_ld_reg (env, regs, value_regno, ret, reg->btf, btf_id, flag); +/* We've assigned a new type to regno, so don't undo masking. */ +if (regno == value_regno) + mask = false; + } +unmask_raw_tp_reg (reg, mask);return 0 ; }kernel/bpf/verifier.c static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · line 7178
Quoted 5 lines
kernel/bpf/verifier.c +1
-1@@ -7140,7 +7175,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (!err && t ==BPF_READ && value_regno >=0 )mark_reg_unknown (env, regs, value_regno); }else if (base_type (reg->type) ==PTR_TO_BTF_ID && - !type_may_be_null (reg->type)) { + (bpf_prog_is_raw_tp (env->prog) || !type_may_be_null (reg->type))) {While looking at this again, I'm wondering if this check is too relaxed. Since we're matching on
base_type, this will probably end up allowing anyPTR_TO_BTF_IDwithPTR_MAYBE_NULLinraw_tpprogs. So I think I need to narrow it down, unless I'm missing something.
Quoted 1 line
Will wait for a while for comments before respinning v3.
I verified that this is broken, a pointer from bpf_obj_new could be deref'd without NULL check.
I'll respin v3 fixing this.
Quoted 4 lines
Quoted 4 lines
kernel/bpf/verifier.c +0
-0err =check_ptr_to_btf_access (env, regs, regno, off, size, t, value_regno);
Kumar Kartikeya Dwivedi
maintainerPATCH v2 2/3 selftests/bpf: Clean up open-coded gettid syscall invocations
In reply to Kumar Kartikeya DwivediAvailability of the gettid definition across glibc versions supported by BPF selftests is not certain. Currently, all users in the tree open-code syscall to gettid. Convert them to a common macro definition.
--- tools/testing/selftests/bpf/benchs/bench_trigger.c | 3 ++- tools/testing/selftests/bpf/bpf_util.h | 9 +++++++++ .../testing/selftests/bpf/map_tests/task_storage_map.c | 3 ++- tools/testing/selftests/bpf/prog_tests/bpf_cookie.c | 2 +- tools/testing/selftests/bpf/prog_tests/bpf_iter.c | 6 +++--- .../selftests/bpf/prog_tests/cgrp_local_storage.c | 10 +++++----- tools/testing/selftests/bpf/prog_tests/core_reloc.c | 2 +- tools/testing/selftests/bpf/prog_tests/linked_funcs.c | 2 +- .../selftests/bpf/prog_tests/ns_current_pid_tgid.c | 2 +- tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c | 4 ++-- .../selftests/bpf/prog_tests/task_local_storage.c | 10 +++++----- .../selftests/bpf/prog_tests/uprobe_multi_test.c | 2 +- 12 files changed, 33 insertions(+), 22 deletions(-)
tools/testing/selftests/bpf/benchs/bench_trigger.c +2 -1
@@ -4,6 +4,7 @@#include <argp.h> #include <unistd.h> #include <stdint.h> +#include "bpf_util.h" #include "bench.h" #include "trigger_bench.skel.h" #include "trace_helpers.h"
@@ -72,7 +73,7 @@ static __always_inline void inc_counter(struct counter *counters)unsigned slot;if (unlikely (tid ==0 )) - tid =syscall (SYS_gettid); + tid =sys_gettid ();/* multiplicative hashing, it's fast */ slot =2654435769U * tid;
tools/testing/selftests/bpf/bpf_util.h +9 -0
@@ -6,6 +6,7 @@#include <stdlib.h> #include <string.h> #include <errno.h> +#include <syscall.h> #include <bpf/libbpf.h> /* libbpf_num_possible_cpus */ static inline unsigned int bpf_num_possible_cpus (void )
@@ -59,4 +60,12 @@ static inline void bpf_strlcpy(char *dst, const char *src, size_t sz) (offsetof(TYPE, MEMBER) + sizeof_field(TYPE, MEMBER))#endif +/* Availability of gettid across glibc versions is hit-and-miss, therefore +* fallback to syscall in this macro and use it everywhere. +*/ +#ifndef sys_gettid +#define sys_gettid () syscall(SYS_gettid) +#endif + +#endif /* __BPF_UTIL__ */
tools/testing/selftests/bpf/map_tests/task_storage_map.c +2 -1
@@ -12,6 +12,7 @@#include <bpf/bpf.h> #include <bpf/libbpf.h> +#include "bpf_util.h" #include "test_maps.h" #include "task_local_storage_helpers.h" #include "read_bpf_task_storage_busy.skel.h"
@@ -115,7 +116,7 @@ void test_task_storage_map_stress_lookup(void)CHECK (err,"attach" ,"error %d\n" , err);/* Trigger program */ -syscall (SYS_gettid); +sys_gettid (); skel->bss->pid =0 ;CHECK (skel->bss->busy !=0 ,"bad bpf_task_storage_busy" ,"got %d\n" , skel->bss->busy);
tools/testing/selftests/bpf/prog_tests/bpf_cookie.c +1 -1
@@ -690,7 +690,7 @@ void test_bpf_cookie(void)if (!ASSERT_OK_PTR (skel,"skel_open" ))return ; - skel->bss->my_tid =syscall (SYS_gettid); + skel->bss->my_tid =sys_gettid ();if (test__start_subtest ("kprobe" ))kprobe_subtest (skel);
tools/testing/selftests/bpf/prog_tests/bpf_iter.c +3 -3
@@ -226,7 +226,7 @@ static void test_task_common_nocheck(struct bpf_iter_attach_opts *opts,ASSERT_OK (pthread_create (&thread_id,NULL , &do_nothing_wait,NULL ),"pthread_create" ); - skel->bss->tid =syscall (SYS_gettid); + skel->bss->tid =sys_gettid ();do_dummy_read_opts (skel->progs.dump_task, opts);
@@ -255,10 +255,10 @@ static void *run_test_task_tid(void *arg)union bpf_iter_link_info linfo;int num_unknown_tid, num_known_tid; -ASSERT_NEQ (getpid (),syscall (SYS_gettid),"check_new_thread_id" ); +ASSERT_NEQ (getpid (),sys_gettid (),"check_new_thread_id" );memset (&linfo,0 ,sizeof (linfo)); - linfo.task.tid =syscall (SYS_gettid); + linfo.task.tid =sys_gettid (); opts.link_info = &linfo; opts.link_info_len =sizeof (linfo);test_task_common (&opts,0 ,1 );
tools/testing/selftests/bpf/prog_tests/cgrp_local_storage.c +5 -5
@@ -63,14 +63,14 @@ static void test_tp_btf(int cgroup_fd)if (!ASSERT_OK (err,"map_delete_elem" )) goto out; - skel->bss->target_pid =syscall (SYS_gettid); + skel->bss->target_pid =sys_gettid (); err =cgrp_ls_tp_btf__attach (skel);if (!ASSERT_OK (err,"skel_attach" )) goto out; -syscall (SYS_gettid); -syscall (SYS_gettid); +sys_gettid (); +sys_gettid (); skel->bss->target_pid =0 ;
@@ -154,7 +154,7 @@ static void test_recursion(int cgroup_fd) goto out;/* trigger sys_enter, make sure it does not cause deadlock */ -syscall (SYS_gettid); +sys_gettid (); out:cgrp_ls_recursion__destroy (skel);
@@ -224,7 +224,7 @@ static void test_yes_rcu_lock(__u64 cgroup_id)return ;CGROUP_MODE_SET (skel); - skel->bss->target_pid =syscall (SYS_gettid); + skel->bss->target_pid =sys_gettid ();bpf_program__set_autoload (skel->progs.yes_rcu_lock, true); err =cgrp_ls_sleepable__load (skel);
tools/testing/selftests/bpf/prog_tests/core_reloc.c +1 -1
@@ -1010,7 +1010,7 @@ static void run_core_reloc_tests(bool use_btfgen)struct data *data;void *mmap_data =NULL ; - my_pid_tgid =getpid () | ((uint64_t )syscall (SYS_gettid) <<32 ); + my_pid_tgid =getpid () | ((uint64_t )sys_gettid () <<32 );for (i =0 ; i <ARRAY_SIZE (test_cases); i++) {char btf_file[] ="/tmp/core_reloc.btf.XXXXXX" ;
tools/testing/selftests/bpf/prog_tests/linked_funcs.c +1 -1
@@ -20,7 +20,7 @@ void test_linked_funcs(void)bpf_program__set_autoload (skel->progs.handler1, true);bpf_program__set_autoload (skel->progs.handler2, true); - skel->rodata->my_tid =syscall (SYS_gettid); + skel->rodata->my_tid =sys_gettid (); skel->bss->syscall_id = SYS_getpgid; err =linked_funcs__load (skel);
tools/testing/selftests/bpf/prog_tests/ns_current_pid_tgid.c +1 -1
@@ -23,7 +23,7 @@ static int get_pid_tgid(pid_t *pid, pid_t *tgid,struct stat st;int err; - *pid =syscall (SYS_gettid); + *pid =sys_gettid (); *tgid =getpid (); err =stat ("/proc/self/ns/pid" , &st);
tools/testing/selftests/bpf/prog_tests/rcu_read_lock.c +2 -2
@@ -21,7 +21,7 @@ static void test_success(void)if (!ASSERT_OK_PTR (skel,"skel_open" ))return ; - skel->bss->target_pid =syscall (SYS_gettid); + skel->bss->target_pid =sys_gettid ();bpf_program__set_autoload (skel->progs.get_cgroup_id, true);bpf_program__set_autoload (skel->progs.task_succ, true);
@@ -58,7 +58,7 @@ static void test_rcuptr_acquire(void)if (!ASSERT_OK_PTR (skel,"skel_open" ))return ; - skel->bss->target_pid =syscall (SYS_gettid); + skel->bss->target_pid =sys_gettid ();bpf_program__set_autoload (skel->progs.task_acquire, true); err =rcu_read_lock__load (skel);
tools/testing/selftests/bpf/prog_tests/task_local_storage.c +5 -5
@@ -31,14 +31,14 @@ static void test_sys_enter_exit(void)if (!ASSERT_OK_PTR (skel,"skel_open_and_load" ))return ; - skel->bss->target_pid =syscall (SYS_gettid); + skel->bss->target_pid =sys_gettid (); err =task_local_storage__attach (skel);if (!ASSERT_OK (err,"skel_attach" )) goto out; -syscall (SYS_gettid); -syscall (SYS_gettid); +sys_gettid (); +sys_gettid ();/* 3x syscalls: 1x attach and 2x gettid */ ASSERT_EQ (skel->bss->enter_cnt,3 ,"enter_cnt" );
@@ -107,7 +107,7 @@ static void test_recursion(void)/* trigger sys_enter, make sure it does not cause deadlock */ skel->bss->test_pid =getpid (); -syscall (SYS_gettid); +sys_gettid (); skel->bss->test_pid =0 ;task_ls_recursion__detach (skel);
@@ -262,7 +262,7 @@ static void test_uptr_basic(void)__u64 ev_dummy_data =1 ;int err; - my_tid =syscall (SYS_gettid); + my_tid =sys_gettid (); parent_task_fd =sys_pidfd_open (my_tid,0 );if (!ASSERT_OK_FD (parent_task_fd,"parent_task_fd" ))return ;
tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c +1 -1
@@ -125,7 +125,7 @@ static void *child_thread(void *ctx)struct child *child = ctx;int c =0 , err; - child->tid =syscall (SYS_gettid); + child->tid =sys_gettid ();/* let parent know we are ready */ err =write (child->c2p[1 ], &c,1 );
Signature · 1 line
2.43.5
Kumar Kartikeya Dwivedi
maintainerPATCH v2 3/3 selftests/bpf: Add tests for raw_tp null handling
In reply to Kumar Kartikeya DwivediEnsure that trusted PTR_TO_BTF_ID accesses perform PROBE_MEM handling in raw_tp program. Without the previous fix, this selftest crashes the kernel due to a NULL-pointer dereference. Also ensure that dead code elimination does not kick in for checks on the pointer.
--- .../bpf/bpf_testmod/bpf_testmod-events.h | 8 +++++ .../selftests/bpf/bpf_testmod/bpf_testmod.c | 2 ++ .../selftests/bpf/prog_tests/raw_tp_null.c | 25 +++++++++++++++ .../testing/selftests/bpf/progs/raw_tp_null.c | 32 +++++++++++++++++++ 4 files changed, 67 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/raw_tp_null.c create mode 100644 tools/testing/selftests/bpf/progs/raw_tp_null.c
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod-events.h +8 -0
@@ -40,6 +40,14 @@ DECLARE_TRACE(bpf_testmod_test_nullable_bare,TP_ARGS (ctx__nullable) ); +struct sk_buff ; + +DECLARE_TRACE (bpf_testmod_test_raw_tp_null, + TP_PROTO(struct sk_buff *skb), +TP_ARGS (skb) +); + +#undef BPF_TESTMOD_DECLARE_TRACE#ifdef DECLARE_TRACE_WRITABLE #define BPF_TESTMOD_DECLARE_TRACE (call, proto, args, size) \
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +2 -0
@@ -380,6 +380,8 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, (void )bpf_testmod_test_arg_ptr_to_struct (&struct_arg1_2); + (void )trace_bpf_testmod_test_raw_tp_null (NULL ); + struct_arg3 =kmalloc ((sizeof (struct bpf_testmod_struct_arg_3 ) +sizeof (int )),GFP_KERNEL );if (struct_arg3 !=NULL ) {
tools/testing/selftests/bpf/prog_tests/raw_tp_null.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <test_progs.h> +#include "raw_tp_null.skel.h" + +void test_raw_tp_null (void ) +{ +struct raw_tp_null *skel; + + skel =raw_tp_null__open_and_load (); +if (!ASSERT_OK_PTR (skel,"raw_tp_null__open_and_load" )) +return ; + + skel->bss->tid =sys_gettid (); + +if (!ASSERT_OK (raw_tp_null__attach (skel),"raw_tp_null__attach" )) + goto end; + +ASSERT_OK (trigger_module_test_read (2 ),"trigger testmod read" ); +ASSERT_EQ (skel->bss->i,3 ,"invocations" ); + +end: +raw_tp_null__destroy (skel); +}
tools/testing/selftests/bpf/progs/raw_tp_null.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include <vmlinux.h> +#include <bpf/bpf_tracing.h> + +char _license[] SEC("license") = "GPL"; + +int tid; +int i; + +SEC("tp_btf/bpf_testmod_test_raw_tp_null") +int BPF_PROG (test_raw_tp_null ,struct sk_buff *skb) +{ +struct task_struct *task =bpf_get_current_task_btf (); + +if (task->pid != tid) +return 0 ; + + i = i + skb->mark +1 ; +/* The compiler may move the NULL check before this deref, which causes +* the load to fail as deref of scalar. Prevent that by using a barrier. +*/ +barrier (); +/* If dead code elimination kicks in, the increment below will +* be removed. For raw_tp programs, we mark input arguments as +* PTR_MAYBE_NULL, so branch prediction should never kick in. +*/ +if (!skb) + i +=2 ; +return 0 ; +}
Signature · 1 line
2.43.5
Quoted 38 lines · Kumar Kartikeya Dwivedi
More context is available in [0], but the TLDR; is that the verifier incorrectly assumes that any raw tracepoint argument will always be non-NULL. This means that even when users correctly check possible NULL arguments, the verifier can remove the NULL check due to incorrect knowledge of the NULL-ness of the pointer. Secondly, kernel helpers or kfuncs taking these trusted tracepoint arguments incorrectly assume that all arguments will always be valid non-NULL.
In this set, we mark
raw_tparguments asPTR_MAYBE_NULLon top ofPTR_TRUSTED, but special case their behavior when dereferencing them or pointer arithmetic over them is involved. When passing trusted args to helpers or kfuncs,raw_tpprograms are permitted to pass possibly NULL pointers in such cases.Any loads into such maybe NULL trusted
PTR_TO_BTF_IDis promoted to aPROBE_MEMload to handle emanating page faults. The verifier will ensure NULL checks on such pointers are preserved and do not lead to dead code elimination.This new behavior is not applied when
ref_obj_idis non-zero, as those pointers do not belong toraw_tparguments, but instead acquired objects.Since helpers and kfuncs already require attention for
PTR_TO_BTF_ID(non-trusted) pointers, we do not implement any protection for such cases in this patch set, and leave it as future work for an upcoming series.A selftest is included with this patch set to verify the new behavior, and it crashes the kernel without the first patch.
[0]: https://lore.kernel.org/bpf/CAADnVQLMPPavJQR6JFsi3dtaaLHB816JN4HCV_TFWohJ61D+wQ@mail.gmail.com
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]
Quoted 8 lines
* Add patch to clean up users of gettid (Andrii)
* Avoid nested blocks in sefltest (Andrii)
* Prevent code motion optimization in selftest usingbarrier()Kumar Kartikeya Dwivedi (3):
bpf: Markraw_tparguments withPTR_MAYBE_NULLselftests/bpf: Clean up open-coded gettid syscall invocations selftests/bpf: Add tests forraw_tpnull handling
thanks a lot for fixing this! lgtm
jirka
Quoted 30 lines
include/linux/bpf.h | 6 ++ kernel/bpf/btf.c | 5 +- kernel/bpf/verifier.c | 75 +++++++++++++++++-- .../selftests/bpf/benchs/bench_trigger.c | 3 +- .../bpf/bpf_testmod/bpf_testmod-events.h | 8 ++ .../selftests/bpf/bpf_testmod/bpf_testmod.c | 2 + tools/testing/selftests/bpf/bpf_util.h | 9 +++ .../bpf/map_tests/task_storage_map.c | 3 +- .../selftests/bpf/prog_tests/bpf_cookie.c | 2 +- .../selftests/bpf/prog_tests/bpf_iter.c | 6 +- .../bpf/prog_tests/cgrp_local_storage.c | 10 +-- .../selftests/bpf/prog_tests/core_reloc.c | 2 +- .../selftests/bpf/prog_tests/linked_funcs.c | 2 +- .../bpf/prog_tests/ns_current_pid_tgid.c | 2 +- .../selftests/bpf/prog_tests/raw_tp_null.c | 25 +++++++ .../selftests/bpf/prog_tests/rcu_read_lock.c | 4 +- .../bpf/prog_tests/task_local_storage.c | 10 +-- .../bpf/prog_tests/uprobe_multi_test.c | 2 +- .../testing/selftests/bpf/progs/raw_tp_null.c | 32 ++++++++ .../bpf/progs/test_tp_btf_nullable.c | 6 +- 20 files changed, 183 insertions(+), 31 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/raw_tp_null.c create mode 100644 tools/testing/selftests/bpf/progs/raw_tp_null.cbase-commit:
e626a13f6fbb4697f8734333432dca577628d09aSignature · 1 line
2.43.5
Hello,
Quoted 28 lines · Kumar Kartikeya Dwivedi
More context is available in [0], but the TLDR; is that the verifier incorrectly assumes that any raw tracepoint argument will always be non-NULL. This means that even when users correctly check possible NULL arguments, the verifier can remove the NULL check due to incorrect knowledge of the NULL-ness of the pointer. Secondly, kernel helpers or kfuncs taking these trusted tracepoint arguments incorrectly assume that all arguments will always be valid non-NULL.
In this set, we mark
raw_tparguments asPTR_MAYBE_NULLon top ofPTR_TRUSTED, but special case their behavior when dereferencing them or pointer arithmetic over them is involved. When passing trusted args to helpers or kfuncs,raw_tpprograms are permitted to pass possibly NULL pointers in such cases.Any loads into such maybe NULL trusted
PTR_TO_BTF_IDis promoted to aPROBE_MEMload to handle emanating page faults. The verifier will ensure NULL checks on such pointers are preserved and do not lead to dead code elimination.This new behavior is not applied when
ref_obj_idis non-zero, as those pointers do not belong toraw_tparguments, but instead acquired objects.Since helpers and kfuncs already require attention for
PTR_TO_BTF_ID(non-trusted) pointers, we do not implement any protection for such cases in this patch set, and leave it as future work for an upcoming series.
Quoted 4 lines
A selftest is included with this patch set to verify the new behavior, and it crashes the kernel without the first patch.
[0]: https://lore.kernel.org/bpf/CAADnVQLMPPavJQR6JFsi3dtaaLHB816JN4HCV_TFWohJ61D+wQ@mail.gmail.com
This indeed cures the issue for me! Thanks a lot for fixing it.
Best,
Juri