Support eliding map lookup nullness
Subsystems (7)
v5 cover Support eliding map lookup nullness
This patch allows progs to elide a null check on statically known map lookup keys. In other words, if the verifier can statically prove that the lookup will be in-bounds, allow the prog to drop the null check.
This is useful for two reasons:
1. Large numbers of nullness checks (especially when they cannot fail) unnecessarily pushes prog towards BPF_COMPLEXITY_LIMIT_JMP_SEQ.
2. It forms a tighter contract between programmer and verifier.
For (1), bpftrace is starting to make heavier use of percpu scratch maps. As a result, for user scripts with large number of unrolled loops, we are starting to hit jump complexity verification errors. These percpu lookups cannot fail anyways, as we only use static key values. Eliding nullness probably results in less work for verifier as well.
For (2), percpu scratch maps are often used as a larger stack, as the currrent stack is limited to 512 bytes. In these situations, it is desirable for the programmer to express: "this lookup should never fail, and if it does, it means I messed up the code". By omitting the null check, the programmer can "ask" the verifier to double check the logic.
Changes in v5:
* Dropped all acks
* Use s64 instead of long for const_map_key
* Ensure stack slot contains spilled reg before accessing spilled_ptr
* Ensure spilled reg is a scalar before accessing tnum const value
* Fix verifier selftest for 32-bit write to write at 8 byte alignment to ensure spill is tracked
* Introduce more precise tracking of helper stack accesses
* Do constant map key extraction as part of helper argument processing and then remove duplicated stack checks
* Use ret_flag instead of regs[BPF_REG_0].type
* Handle STACK_ZERO
* Fix bug in bpf_load_hdr_opt() arg annotation
Changes in v4:
* Only allow for CAP_BPF
* Add test for stack growing upwards
* Improve comment about stack growing upwards
Changes in v3:
* Check if stack is (erroneously) growing upwards
* Mention in commit message why existing tests needed change
Changes in v2:
* Added a check for when R2 is not a ptr to stack
* Added a check for when stack is uninitialized (no stack slot yet)
* Updated existing tests to account for null elision
* Added test case for when R2 can be both const and non-const
Daniel Xu (5):
bpf: verifier: Add missing newline on verbose() call
bpf: tcp: Mark bpf_load_hdr_opt() arg2 as read-write
bpf: verifier: Refactor helper access type tracking
bpf: verifier: Support eliding map lookup nullness
bpf: selftests: verifier: Add nullness elision tests
kernel/bpf/verifier.c | 127 ++++++++--- net/core/filter.c | 2 +- .../testing/selftests/bpf/progs/dynptr_fail.c | 6 +- tools/testing/selftests/bpf/progs/iters.c | 14 +- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/test_global_func10.c | 2 +- .../selftests/bpf/progs/uninit_stack.c | 29 --- .../bpf/progs/verifier_array_access.c | 214 ++++++++++++++++++ .../bpf/progs/verifier_basic_stack.c | 2 +- .../selftests/bpf/progs/verifier_const_or.c | 4 +- .../progs/verifier_helper_access_var_len.c | 12 +- .../selftests/bpf/progs/verifier_int_ptr.c | 2 +- .../selftests/bpf/progs/verifier_map_in_map.c | 2 +- .../selftests/bpf/progs/verifier_mtu.c | 2 +- .../selftests/bpf/progs/verifier_raw_stack.c | 4 +- .../selftests/bpf/progs/verifier_unpriv.c | 2 +- .../selftests/bpf/progs/verifier_var_off.c | 8 +- tools/testing/selftests/bpf/verifier/calls.c | 2 +- .../testing/selftests/bpf/verifier/map_kptr.c | 2 +- 19 files changed, 342 insertions(+), 96 deletions(-)
Signature · 1 line
2.46.0
PATCH v5 1/5 bpf: verifier: Add missing newline on verbose() call
The print was missing a newline.
--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/verifier.c +1 -1
@@ -7754,7 +7754,7 @@ static int check_stack_range_initialized( slot = -i -1 ; spi = slot /BPF_REG_SIZE ;if (state->allocated_stack <= slot) { -verbose (env,"verifier bug: allocated_stack too small" ); +verbose (env,"verifier bug: allocated_stack too small\n" );return -EFAULT ; }
Signature · 1 line
2.46.0
PATCH v5 2/5 bpf: tcp: Mark bpf_load_hdr_opt() arg2 as read-write
In reply to Daniel XuMEM_WRITE attribute is defined as: "Non-presence of MEM_WRITE means that MEM is only being read". bpf_load_hdr_opt() both reads and writes from its arg2 - void *search_res.
This matters a lot for the next commit where we more precisely track stack accesses. Without this annotation, the verifier will make false assumptions about the contents of memory written to by helpers and possibly prune valid branches.
--- net/core/filter.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
net/core/filter.c +1 -1
@@ -7643,7 +7643,7 @@ static const struct bpf_func_proto bpf_sock_ops_load_hdr_opt_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_PTR_TO_MEM, + .arg2_type = ARG_PTR_TO_MEM | MEM_WRITE, .arg3_type = ARG_CONST_SIZE, .arg4_type = ARG_ANYTHING, };
Signature · 1 line
2.46.0
Re: [PATCH bpf-next v5 2/5] bpf: tcp: Mark bpf_load_hdr_opt() arg2 as read-write
Quoted 8 lines · Daniel Xu
MEM_WRITEattribute is defined as: "Non-presence ofMEM_WRITEmeans that MEM is only being read".bpf_load_hdr_opt()both reads and writes from its arg2 - void*search_res.This matters a lot for the next commit where we more precisely track stack accesses. Without this annotation, the verifier will make false assumptions about the contents of memory written to by helpers and possibly prune valid branches.
PATCH v5 3/5 bpf: verifier: Refactor helper access type tracking
In reply to Daniel XuPreviously, the verifier was treating all PTR_TO_STACK registers passed to a helper call as potentially written to by the helper. However, all calls to check_stack_range_initialized() already have precise access type information available.
Rather than treat ACCESS_HELPER as a proxy for BPF_WRITE, pass enum bpf_access_type to check_stack_range_initialized() to more precisely track helper arguments.
One benefit from this precision is that registers tracked as valid spills and passed as a read-only helper argument remain tracked after the call. Rather than being marked STACK_MISC afterwards.
An additional benefit is the verifier logs are also more precise. For this particular error, users will enjoy a slightly clearer message. See included selftest updates for examples.
--- kernel/bpf/verifier.c | 45 +++++++------------ .../testing/selftests/bpf/progs/dynptr_fail.c | 6 +-- .../selftests/bpf/progs/test_global_func10.c | 2 +- .../selftests/bpf/progs/uninit_stack.c | 29 ------------ .../bpf/progs/verifier_basic_stack.c | 2 +- .../selftests/bpf/progs/verifier_const_or.c | 4 +- .../progs/verifier_helper_access_var_len.c | 12 ++--- .../selftests/bpf/progs/verifier_int_ptr.c | 2 +- .../selftests/bpf/progs/verifier_mtu.c | 2 +- .../selftests/bpf/progs/verifier_raw_stack.c | 4 +- .../selftests/bpf/progs/verifier_unpriv.c | 2 +- .../selftests/bpf/progs/verifier_var_off.c | 8 ++-- tools/testing/selftests/bpf/verifier/calls.c | 2 +- 13 files changed, 39 insertions(+), 81 deletions(-)
kernel/bpf/verifier.c +16 -29
@@ -5302,7 +5302,7 @@ enum bpf_access_src {static int check_stack_range_initialized (struct bpf_verifier_env *env,int regno,int off,int access_size,bool zero_size_allowed, -enum bpf_access_src type, +enum bpf_access_type type,struct bpf_call_arg_meta *meta);static struct bpf_reg_state *reg_state (struct bpf_verifier_env *env,int regno)
@@ -5335,7 +5335,7 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env,/* Note that we pass a NULL meta, so raw access will not be permitted. */ err =check_stack_range_initialized (env, ptr_regno, off, size, - false,ACCESS_DIRECT ,NULL ); + false,BPF_READ ,NULL );if (err)return err;
@@ -7205,7 +7205,7 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,static int check_stack_access_within_bounds (struct bpf_verifier_env *env,int regno,int off,int access_size, -enum bpf_access_src src,enum bpf_access_type type) +enum bpf_access_type type) {struct bpf_reg_state *regs =cur_regs (env);struct bpf_reg_state *reg = regs + regno;
@@ -7214,10 +7214,7 @@ static int check_stack_access_within_bounds(int err;char *err_extra; -if (src ==ACCESS_HELPER ) -/* We don't know if helpers are reading or writing (or both). */ - err_extra =" indirect access to" ; -else if (type ==BPF_READ ) +if (type ==BPF_READ ) err_extra =" read from" ;else err_extra =" write to" ;
@@ -7435,7 +7432,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn }else if (reg->type ==PTR_TO_STACK ) {/* Basic bounds checks. */ - err =check_stack_access_within_bounds (env, regno, off, size,ACCESS_DIRECT , t); + err =check_stack_access_within_bounds (env, regno, off, size, t);if (err)return err;
@@ -7655,13 +7652,11 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_istatic int check_stack_range_initialized (struct bpf_verifier_env *env,int regno,int off,int access_size,bool zero_size_allowed, -enum bpf_access_src type,struct bpf_call_arg_meta *meta) +enum bpf_access_type type,struct bpf_call_arg_meta *meta) {struct bpf_reg_state *reg =reg_state (env, regno);struct bpf_func_state *state =func (env, reg);int err, min_off, max_off, i, j, slot, spi; -char *err_extra = type ==ACCESS_HELPER ?" indirect" :"" ; -enum bpf_access_type bounds_check_type;/* Some accesses can write anything into the stack, others are * read-only. */
@@ -7672,18 +7667,10 @@ static int check_stack_range_initialized(return -EACCES ; } -if (type ==ACCESS_HELPER ) { -/* The bounds checks for writes are more permissive than for -* reads. However, if raw_mode is not set, we'll do extra -* checks below. -*/ - bounds_check_type =BPF_WRITE ; +if (type ==BPF_WRITE ) clobber = true; - }else { - bounds_check_type =BPF_READ ; - } - err =check_stack_access_within_bounds (env, regno, off, access_size, - type, bounds_check_type); + + err =check_stack_access_within_bounds (env, regno, off, access_size, type);if (err)return err;
@@ -7700,8 +7687,8 @@ static int check_stack_range_initialized(char tn_buf[48 ];tnum_strn (tn_buf,sizeof (tn_buf), reg->var_off); -verbose (env,"R%d%s variable offset stack access prohibited for !root, var_off=%s\n" , - regno, err_extra, tn_buf); +verbose (env,"R%d variable offset stack access prohibited for !root, var_off=%s\n" , + regno, tn_buf);return -EACCES ; }/* Only initialized buffer on stack is allowed to be accessed
@@ -7782,14 +7769,14 @@ static int check_stack_range_initialized( }if (tnum_is_const (reg->var_off)) { -verbose (env,"invalid%s read from stack R%d off %d+%d size %d\n" , - err_extra, regno, min_off, i - min_off, access_size); +verbose (env,"invalid read from stack R%d off %d+%d size %d\n" , + regno, min_off, i - min_off, access_size); }else {char tn_buf[48 ];tnum_strn (tn_buf,sizeof (tn_buf), reg->var_off); -verbose (env,"invalid%s read from stack R%d var_off %s+%d size %d\n" , - err_extra, regno, tn_buf, i - min_off, access_size); +verbose (env,"invalid read from stack R%d var_off %s+%d size %d\n" , + regno, tn_buf, i - min_off, access_size); }return -EACCES ; mark:
@@ -7864,7 +7851,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,return check_stack_range_initialized ( env, regno, reg->off, access_size, - zero_size_allowed,ACCESS_HELPER , meta); + zero_size_allowed, access_type, meta);case PTR_TO_BTF_ID :return check_ptr_to_btf_access (env, regs, regno, reg->off, access_size,BPF_READ ,-1 );
tools/testing/selftests/bpf/progs/dynptr_fail.c +3 -3
@@ -192,7 +192,7 @@ int ringbuf_invalid_api(void *ctx)/* Can't add a dynptr to a map */ SEC("?raw_tp") -__failure __msg("invalid indirect read from stack") +__failure __msg("invalid read from stack")int add_dynptr_to_map1 (void *ctx) {struct bpf_dynptr ptr;
@@ -210,7 +210,7 @@ int add_dynptr_to_map1(void *ctx)/* Can't add a struct with an embedded dynptr to a map */ SEC("?raw_tp") -__failure __msg("invalid indirect read from stack") +__failure __msg("invalid read from stack")int add_dynptr_to_map2 (void *ctx) {struct test_info x;
@@ -398,7 +398,7 @@ int data_slice_missing_null_check2(void *ctx)* dynptr argument */ SEC("?raw_tp") -__failure __msg("invalid indirect read from stack") +__failure __msg("invalid read from stack")int invalid_helper1 (void *ctx) {struct bpf_dynptr ptr;
tools/testing/selftests/bpf/progs/test_global_func10.c +1 -1
@@ -26,7 +26,7 @@ __noinline int foo(const struct Big *big) } SEC("cgroup_skb/ingress") -__failure __msg("invalid indirect access to stack") +__failure __msg("invalid read from stack") int global_func10(struct __sk_buff *skb) {const struct Small small = {.x = skb->len };
tools/testing/selftests/bpf/progs/uninit_stack.c +0 -29
@@ -55,33 +55,4 @@ exit_%=: r0 = 0; \ : __clobber_all); } -static __noinline void dummy(void) {} - -/* Pass a pointer to uninitialized stack memory to a helper. -* Passed memory block should be marked as STACK_MISC after helper call. -*/ -SEC("socket") -__log_level(7) __msg("fp-104=mmmmmmmm") -__naked int helper_uninit_to_misc (void *ctx) -{ - asmvolatile (" \ -/* force stack depth to be 128 */ \ -*(u64*)(r10 - 128) = r1; \ -r1 = r10; \ -r1 += -128; \ -r2 = 32; \ -call %[bpf_trace_printk]; \ -/* Call to dummy() forces print_verifier_state(..., true), \ -* thus showing the stack state, matched by __msg(). \ -*/ \ -call %[dummy]; \ -r0 = 0; \ -exit; \ -" - : - : __imm(bpf_trace_printk), -__imm (dummy) - : __clobber_all); -} - char _license[] SEC("license") = "GPL";
tools/testing/selftests/bpf/progs/verifier_basic_stack.c +1 -1
@@ -28,7 +28,7 @@ __naked void stack_out_of_bounds(void) SEC("socket") __description("uninitialized stack1") __success __log_level(4) __msg("stack depth 8") -__failure_unpriv __msg_unpriv("invalid indirect read from stack") +__failure_unpriv __msg_unpriv("invalid read from stack") __naked void uninitialized_stack1(void) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_const_or.c +2 -2
@@ -25,7 +25,7 @@ __naked void constant_should_keep_constant_type(void) SEC("tracepoint") __description("constant register |= constant should not bypass stack boundary checks") -__failure __msg("invalid indirect access to stack R1 off=-48 size=58") +__failure __msg("invalid write to stack R1 off=-48 size=58") __naked void not_bypass_stack_boundary_checks_1(void) { asm volatile (" \
@@ -62,7 +62,7 @@ __naked void register_should_keep_constant_type(void) SEC("tracepoint") __description("constant register |= constant register should not bypass stack boundary checks") -__failure __msg("invalid indirect access to stack R1 off=-48 size=58") +__failure __msg("invalid write to stack R1 off=-48 size=58") __naked void not_bypass_stack_boundary_checks_2(void) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_helper_access_var_len.c +6 -6
@@ -67,7 +67,7 @@ SEC("socket") __description("helper access to variable memory: stack, bitwise AND, zero included") /* in privileged mode reads from uninitialized stack locations are permitted */ __success __failure_unpriv -__msg_unpriv("invalid indirect read from stack R2 off -64+0 size 64") +__msg_unpriv("invalid read from stack R2 off -64+0 size 64") __retval(0)__naked void stack_bitwise_and_zero_included (void ) {
@@ -100,7 +100,7 @@ __naked void stack_bitwise_and_zero_included(void) SEC("tracepoint") __description("helper access to variable memory: stack, bitwise AND + JMP, wrong max") -__failure __msg("invalid indirect access to stack R1 off=-64 size=65") +__failure __msg("invalid write to stack R1 off=-64 size=65") __naked void bitwise_and_jmp_wrong_max(void) { asm volatile (" \
@@ -187,7 +187,7 @@ l0_%=: r0 = 0; \ SEC("tracepoint") __description("helper access to variable memory: stack, JMP, bounds + offset") -__failure __msg("invalid indirect access to stack R1 off=-64 size=65") +__failure __msg("invalid write to stack R1 off=-64 size=65") __naked void memory_stack_jmp_bounds_offset(void) { asm volatile (" \
@@ -211,7 +211,7 @@ l0_%=: r0 = 0; \ SEC("tracepoint") __description("helper access to variable memory: stack, JMP, wrong max") -__failure __msg("invalid indirect access to stack R1 off=-64 size=65") +__failure __msg("invalid write to stack R1 off=-64 size=65") __naked void memory_stack_jmp_wrong_max(void) { asm volatile (" \
@@ -260,7 +260,7 @@ SEC("socket") __description("helper access to variable memory: stack, JMP, no min check") /* in privileged mode reads from uninitialized stack locations are permitted */ __success __failure_unpriv -__msg_unpriv("invalid indirect read from stack R2 off -64+0 size 64") +__msg_unpriv("invalid read from stack R2 off -64+0 size 64") __retval(0)__naked void stack_jmp_no_min_check (void ) {
@@ -750,7 +750,7 @@ SEC("socket") __description("helper access to variable memory: 8 bytes leak") /* in privileged mode reads from uninitialized stack locations are permitted */ __success __failure_unpriv -__msg_unpriv("invalid indirect read from stack R2 off -64+32 size 64") +__msg_unpriv("invalid read from stack R2 off -64+32 size 64") __retval(0)__naked void variable_memory_8_bytes_leak (void ) {
tools/testing/selftests/bpf/progs/verifier_int_ptr.c +1 -1
@@ -96,7 +96,7 @@ __naked void arg_ptr_to_long_misaligned(void) SEC("cgroup/sysctl") __description("arg pointer to long size < sizeof(long)") -__failure __msg("invalid indirect access to stack R4 off=-4 size=8") +__failure __msg("invalid write to stack R4 off=-4 size=8") __naked void to_long_size_sizeof_long(void) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_mtu.c +1 -1
@@ -8,7 +8,7 @@ SEC("tc/ingress") __description("uninit/mtu: write rejected") __success __caps_unpriv(CAP_BPF|CAP_NET_ADMIN) -__failure_unpriv __msg_unpriv("invalid indirect read from stack") +__failure_unpriv __msg_unpriv("invalid read from stack")int tc_uninit_mtu (struct __sk_buff *ctx) {__u32 mtu;
tools/testing/selftests/bpf/progs/verifier_raw_stack.c +2 -2
@@ -236,7 +236,7 @@ __naked void load_bytes_spilled_regs_data(void) SEC("tc") __description("raw_stack: skb_load_bytes, invalid access 1") -__failure __msg("invalid indirect access to stack R3 off=-513 size=8") +__failure __msg("invalid write to stack R3 off=-513 size=8")__naked void load_bytes_invalid_access_1 (void ) { asm volatile (" \
@@ -255,7 +255,7 @@ __naked void load_bytes_invalid_access_1(void) SEC("tc") __description("raw_stack: skb_load_bytes, invalid access 2") -__failure __msg("invalid indirect access to stack R3 off=-1 size=8") +__failure __msg("invalid write to stack R3 off=-1 size=8")__naked void load_bytes_invalid_access_2 (void ) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_unpriv.c +1 -1
@@ -199,7 +199,7 @@ __naked void pass_pointer_to_helper_function(void) SEC("socket") __description("unpriv: indirectly pass pointer on stack to helper function") __success __failure_unpriv -__msg_unpriv("invalid indirect read from stack R2 off -8+0 size 8") +__msg_unpriv("invalid read from stack R2 off -8+0 size 8") __retval(0) __naked void on_stack_to_helper_function(void) {
tools/testing/selftests/bpf/progs/verifier_var_off.c +4 -4
@@ -203,7 +203,7 @@ __naked void stack_write_clobbers_spilled_regs(void) SEC("sockops") __description("indirect variable-offset stack access, unbounded") -__failure __msg("invalid unbounded variable-offset indirect access to stack R4") +__failure __msg("invalid unbounded variable-offset write to stack R4") __naked void variable_offset_stack_access_unbounded(void) { asm volatile (" \
@@ -236,7 +236,7 @@ l0_%=: r0 = 0; \ SEC("lwt_in") __description("indirect variable-offset stack access, max out of bound") -__failure __msg("invalid variable-offset indirect access to stack R2") +__failure __msg("invalid variable-offset read from stack R2") __naked void access_max_out_of_bound(void) { asm volatile (" \
@@ -269,7 +269,7 @@ __naked void access_max_out_of_bound(void)*/ SEC("socket") __description("indirect variable-offset stack access, zero-sized, max out of bound") -__failure __msg("invalid variable-offset indirect access to stack R1") +__failure __msg("invalid variable-offset write to stack R1") __naked void zero_sized_access_max_out_of_bound(void) { asm volatile (" \
@@ -294,7 +294,7 @@ __naked void zero_sized_access_max_out_of_bound(void) SEC("lwt_in") __description("indirect variable-offset stack access, min out of bound") -__failure __msg("invalid variable-offset indirect access to stack R2") +__failure __msg("invalid variable-offset read from stack R2") __naked void access_min_out_of_bound(void) { asm volatile (" \
tools/testing/selftests/bpf/verifier/calls.c +1 -1
@@ -2252,7 +2252,7 @@ BPF_EXIT_INSN(), }, .fixup_map_hash_48b = { 7 }, - .errstr_unpriv = "invalid indirect read from stack R2 off -8+0 size 8", + .errstr_unpriv = "invalid read from stack R2 off -8+0 size 8", .result_unpriv = REJECT,/* in privileged mode reads from uninitialized stack locations are permitted */ .result = ACCEPT,
Signature · 1 line
2.46.0
Re: [PATCH bpf-next v5 3/5] bpf: verifier: Refactor helper access type tracking
Quoted 13 lines · Daniel Xu
Previously, the verifier was treating all
PTR_TO_STACKregisters passed to a helper call as potentially written to by the helper. However, all calls tocheck_stack_range_initialized()already have precise access type information available.Rather than treat
ACCESS_HELPERas a proxy forBPF_WRITE, pass enumbpf_access_typetocheck_stack_range_initialized()to more precisely track helper arguments.One benefit from this precision is that registers tracked as valid spills and passed as a read-only helper argument remain tracked after the call. Rather than being marked
STACK_MISCafterwards.
Quoted 6 lines
An additional benefit is the verifier logs are also more precise. For this particular error, users will enjoy a slightly clearer message. See included selftest updates for examples.
Signed-off-by Daniel Xu <[email protected]>---
I think this change is ok.
With it there is only one use of 'enum bpf_access_src' remains, but it doesn't look like it could be removed.
[...]
Quoted 4 lines
tools/testing/selftests/bpf/progs/uninit_stack.c +0
-0@@ -55,33 +55,4 @@ exit_%=: r0 = 0; \ : __clobber_all);
tools/testing/selftests/bpf/progs/uninit_stack.c exit_%=: r0 = 0; \ · line 65
Quoted 10 lines
tools/testing/selftests/bpf/progs/uninit_stack.c +0
-8} -static __noinline void dummy(void) {} - -/* Pass a pointer to uninitialized stack memory to a helper. -* Passed memory block should be marked as STACK_MISC after helper call. -*/ -SEC("socket") -__log_level(7) __msg("fp-104=mmmmmmmm") -__naked int helper_uninit_to_misc (void *ctx)
Is it possible to peek a helper that writes into memory and not delete this test?
Quoted 21 lines
tools/testing/selftests/bpf/progs/uninit_stack.c +0
-21-{ - asmvolatile (" \ -/* force stack depth to be 128 */ \ -*(u64*)(r10 - 128) = r1; \ -r1 = r10; \ -r1 += -128; \ -r2 = 32; \ -call %[bpf_trace_printk]; \ -/* Call to dummy() forces print_verifier_state(..., true), \ -* thus showing the stack state, matched by __msg(). \ -*/ \ -call %[dummy]; \ -r0 = 0; \ -exit; \ -" - : - : __imm(bpf_trace_printk), -__imm (dummy) - : __clobber_all); -} -
[...]
Re: [PATCH bpf-next v5 3/5] bpf: verifier: Refactor helper access type tracking
Quoted 35 lines · Eduard Zingerman
Quoted 13 lines · Daniel Xu
Previously, the verifier was treating all
PTR_TO_STACKregisters passed to a helper call as potentially written to by the helper. However, all calls tocheck_stack_range_initialized()already have precise access type information available.Rather than treat
ACCESS_HELPERas a proxy forBPF_WRITE, pass enumbpf_access_typetocheck_stack_range_initialized()to more precisely track helper arguments.One benefit from this precision is that registers tracked as valid spills and passed as a read-only helper argument remain tracked after the call. Rather than being marked
STACK_MISCafterwards.Quoted 6 lines
An additional benefit is the verifier logs are also more precise. For this particular error, users will enjoy a slightly clearer message. See included selftest updates for examples.
Signed-off-by Daniel Xu <[email protected]>---
I think this change is ok.
With it there is only one use of 'enumbpf_access_src' remains, but it doesn't look like it could be removed.Acked-by Eduard Zingerman <[email protected]>[...]
Quoted 6 lines
tools/testing/selftests/bpf/progs/uninit_stack.c +0
-0@@ -55,33 +55,4 @@ exit_%=: r0 = 0; \ : __clobber_all); }
Quoted 11 lines
tools/testing/selftests/bpf/progs/uninit_stack.c exit_%=: r0 = 0; \ · line 65
Quoted 8 lines
tools/testing/selftests/bpf/progs/uninit_stack.c +0
-8-static __noinline void dummy(void) {} - -/* Pass a pointer to uninitialized stack memory to a helper. -* Passed memory block should be marked as STACK_MISC after helper call. -*/ -SEC("socket") -__log_level(7) __msg("fp-104=mmmmmmmm") -__naked int helper_uninit_to_misc (void *ctx)Is it possible to peek a helper that writes into memory and not delete this test?
Yeah, good idea. Will do.
PATCH v5 4/5 bpf: verifier: Support eliding map lookup nullness
In reply to Daniel XuThis commit allows progs to elide a null check on statically known map lookup keys. In other words, if the verifier can statically prove that the lookup will be in-bounds, allow the prog to drop the null check.
This is useful for two reasons:
1. Large numbers of nullness checks (especially when they cannot fail) unnecessarily pushes prog towards BPF_COMPLEXITY_LIMIT_JMP_SEQ.
2. It forms a tighter contract between programmer and verifier.
For (1), bpftrace is starting to make heavier use of percpu scratch maps. As a result, for user scripts with large number of unrolled loops, we are starting to hit jump complexity verification errors. These percpu lookups cannot fail anyways, as we only use static key values. Eliding nullness probably results in less work for verifier as well.
For (2), percpu scratch maps are often used as a larger stack, as the currrent stack is limited to 512 bytes. In these situations, it is desirable for the programmer to express: "this lookup should never fail, and if it does, it means I messed up the code". By omitting the null check, the programmer can "ask" the verifier to double check the logic.
Tests also have to be updated in sync with these changes, as the verifier is more efficient with this change. Notable, iters.c tests had to be changed to use a map type that still requires null checks, as it's exercising verifier tracking logic w.r.t iterators.
--- kernel/bpf/verifier.c | 80 ++++++++++++++++++- tools/testing/selftests/bpf/progs/iters.c | 14 ++-- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/verifier_map_in_map.c | 2 +- .../testing/selftests/bpf/verifier/map_kptr.c | 2 +- 5 files changed, 87 insertions(+), 13 deletions(-)
kernel/bpf/verifier.c +77 -3
@@ -287,6 +287,7 @@ struct bpf_call_arg_meta {u32 ret_btf_id;u32 subprogno;struct btf_field *kptr_field; +s64 const_map_key; };struct bpf_kfunc_call_arg_meta {
@@ -9163,6 +9164,53 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return 0 ; } +/* Returns constant key value if possible, else -1 */ +static s64 get_constant_map_key (struct bpf_verifier_env *env, +struct bpf_reg_state *key, +u32 key_size) +{ +struct bpf_func_state *state =func (env, key); +struct bpf_reg_state *reg; +int zero_size =0 ; +int stack_off; +u8 *stype; +int slot; +int spi; +int i; + +if (!env->bpf_capable) +return -1 ; +if (key->type !=PTR_TO_STACK ) +return -1 ; +if (!tnum_is_const (key->var_off)) +return -1 ; + + stack_off = key->off + key->var_off.value; + slot = -stack_off -1 ; + spi = slot /BPF_REG_SIZE ; + +/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++) + zero_size++; +if (zero_size == key_size) +return 0 ; + +if (!is_spilled_reg (&state->stack[spi])) +/* Not pointer to stack */ +return -1 ; + + reg = &state->stack[spi].spilled_ptr; +if (reg->type !=SCALAR_VALUE ) +/* Only scalars are valid array map keys */ +return -1 ; +else if (!tnum_is_const (reg->var_off)) +/* Stack value not statically known */ +return -1 ; + +return reg->var_off.value; +} + static int check_func_arg(struct bpf_verifier_env *env, u32 arg, struct bpf_call_arg_meta *meta, const struct bpf_func_proto *fn,
@@ -9173,6 +9221,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,enum bpf_arg_type arg_type = fn->arg_type[arg];enum bpf_reg_type type = reg->type;u32 *arg_btf_id =NULL ; +u32 key_size;int err =0 ;bool mask;
@@ -9307,8 +9356,11 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,verbose (env,"invalid map_ptr to access map->key\n" );return -EACCES ; } - err =check_helper_mem_access (env, regno, meta->map_ptr->key_size, -BPF_READ , false,NULL ); + key_size = meta->map_ptr->key_size; + err =check_helper_mem_access (env, regno, key_size,BPF_READ , false,NULL ); +if (err) +return err; + meta->const_map_key =get_constant_map_key (env, reg, key_size);break ;case ARG_PTR_TO_MAP_VALUE :if (type_may_be_null (arg_type) &®ister_is_null (reg))
@@ -10833,6 +10885,21 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno state->callback_subprogno == subprogno); } +/* Returns whether or not the given map type can potentially elide +* lookup return value nullness check. This is possible if the key +* is statically known. +*/ +static bool can_elide_value_nullness (enum bpf_map_type type) +{ +switch (type) { +case BPF_MAP_TYPE_ARRAY : +case BPF_MAP_TYPE_PERCPU_ARRAY : +return true; +default : +return false; + } +} +static int get_helper_proto (struct bpf_verifier_env *env,int func_id,const struct bpf_func_proto **ptr) {
@@ -11199,10 +11266,17 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn "kernel subsystem misconfigured verifier\n");return -EINVAL ; } + +if (func_id == BPF_FUNC_map_lookup_elem && +can_elide_value_nullness (meta.map_ptr->map_type) && + meta.const_map_key >=0 && + meta.const_map_key < meta.map_ptr->max_entries) + ret_flag &= ~PTR_MAYBE_NULL ; + regs[BPF_REG_0 ].map_ptr = meta.map_ptr; regs[BPF_REG_0 ].map_uid = meta.map_uid; regs[BPF_REG_0 ].type =PTR_TO_MAP_VALUE | ret_flag; -if (!type_may_be_null (ret_type) && +if (!type_may_be_null (ret_flag) &&btf_record_has_field (meta.map_ptr->record,BPF_SPIN_LOCK )) { regs[BPF_REG_0 ].id = ++env->id_gen; }
tools/testing/selftests/bpf/progs/iters.c +7 -7
@@ -524,11 +524,11 @@ int iter_subprog_iters(const void *ctx) }struct { - __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(type, BPF_MAP_TYPE_HASH); __type(key, int); __type(value, int); __uint(max_entries, 1000); -} arr_map SEC(".maps"); +} hash_map SEC(".maps"); SEC("?raw_tp") __failure __msg("invalid mem access 'scalar'")
@@ -539,7 +539,7 @@ int iter_err_too_permissive1(const void *ctx)MY_PID_GUARD (); - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key);if (!map_val)return 0 ;
@@ -561,12 +561,12 @@ int iter_err_too_permissive2(const void *ctx)MY_PID_GUARD (); - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key);if (!map_val)return 0 ; bpf_repeat(1000000) { - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key); } *map_val =123 ;
@@ -585,7 +585,7 @@ int iter_err_too_permissive3(const void *ctx)MY_PID_GUARD (); bpf_repeat(1000000) { - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key); found = true; }
@@ -606,7 +606,7 @@ int iter_tricky_but_fine(const void *ctx)MY_PID_GUARD (); bpf_repeat(1000000) { - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key);if (map_val) { found = true;break ;
tools/testing/selftests/bpf/progs/map_kptr_fail.c +1 -1
@@ -345,7 +345,7 @@ int reject_indirect_global_func_access(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("Unreleased reference id=5 alloc_insn=") +__failure __msg("Unreleased reference id=4 alloc_insn=") int kptr_xchg_ref_state(struct __sk_buff *ctx) {struct prog_test_ref_kfunc *p;
tools/testing/selftests/bpf/progs/verifier_map_in_map.c +1 -1
@@ -47,7 +47,7 @@ l0_%=: r0 = 0; \ SEC("xdp") __description("map in map state pruning") -__success __msg("processed 26 insns") +__success __msg("processed 15 insns") __log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ) __naked void map_in_map_state_pruning(void) {
tools/testing/selftests/bpf/verifier/map_kptr.c +1 -1
@@ -373,7 +373,7 @@ .prog_type = BPF_PROG_TYPE_SCHED_CLS, .fixup_map_kptr = { 1 }, .result = REJECT, - .errstr = "Unreleased reference id=5 alloc_insn=20", + .errstr = "Unreleased reference id=4 alloc_insn=20", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_acquire", 15 }, }
Signature · 1 line
2.46.0
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
On Thu, 2024-12-12 at 16:22 -0700, Daniel Xu wrote:
I think these changes are fine in general, but see below.
Quoted 13 lines
kernel/bpf/verifier.c +1
-0@@ -287,6 +287,7 @@ struct bpf_call_arg_meta {u32 ret_btf_id;u32 subprogno;struct btf_field *kptr_field; +s64 const_map_key; };struct bpf_kfunc_call_arg_meta {
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9170
Quoted 7 lines
kernel/bpf/verifier.c +4
-0@@ -9163,6 +9164,53 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return 0 ; } +/* Returns constant key value if possible, else -1 */ +static s64 get_constant_map_key (struct bpf_verifier_env *env, +struct bpf_reg_state *key, +u32 key_size)
I understand that this is not your use case, but maybe generalize this a bit by checking maximal register value instead of a constant?
Quoted 21 lines
kernel/bpf/verifier.c +21
-0+{ +struct bpf_func_state *state =func (env, key); +struct bpf_reg_state *reg; +int zero_size =0 ; +int stack_off; +u8 *stype; +int slot; +int spi; +int i; + +if (!env->bpf_capable) +return -1 ; +if (key->type !=PTR_TO_STACK ) +return -1 ; +if (!tnum_is_const (key->var_off)) +return -1 ; + + stack_off = key->off + key->var_off.value; + slot = -stack_off -1 ; + spi = slot /BPF_REG_SIZE ; +
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9201
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++) + zero_size++; +if (zero_size == key_size) +return 0 ; + +if (!is_spilled_reg (&state->stack[spi])) +/* Not pointer to stack */ +return -1 ;
Nit: there is a 'is_spilled_scalar_reg' utility function.
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9209
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+ + reg = &state->stack[spi].spilled_ptr; +if (reg->type !=SCALAR_VALUE ) +/* Only scalars are valid array map keys */ +return -1 ; +else if (!tnum_is_const (reg->var_off)) +/* Stack value not statically known */ +return -1 ;
I think you need to check if size of the spill matches the size of the key. The mismatch would be unsafe when spill size is smaller than key size. E.g. consider 1-byte spill with mask 'mmmmmmrr' and a 4-byte key, at runtime the 'mmmmmm' part might be non-zero, rendering key to be out of range.
Quoted 7 lines
kernel/bpf/verifier.c +4
-0+ +return reg->var_off.value; +} + static int check_func_arg(struct bpf_verifier_env *env, u32 arg, struct bpf_call_arg_meta *meta, const struct bpf_func_proto *fn,
[...]
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 20 lines · Eduard Zingerman
On Thu, 2024-12-12 at 16:22 -0700, Daniel Xu wrote:
I think these changes are fine in general, but see below.
Quoted 16 lines
kernel/bpf/verifier.c +1
-0@@ -287,6 +287,7 @@ struct bpf_call_arg_meta {u32 ret_btf_id;u32 subprogno;struct btf_field *kptr_field; +s64 const_map_key; };struct bpf_kfunc_call_arg_meta {@@ -9163,6 +9164,53 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return 0 ; }
Quoted 7 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9170
Quoted 4 lines
kernel/bpf/verifier.c +4
-0+/* Returns constant key value if possible, else -1 */ +static s64 get_constant_map_key (struct bpf_verifier_env *env, +struct bpf_reg_state *key, +u32 key_size)I understand that this is not your use case, but maybe generalize this a bit by checking maximal register value instead of a constant?
I'll check on this. If it works I think you're right - it allows more flexibility while retaining safety. User could define max_entries to be a power of two and then mask key with with 0xFFFF.. to guarantee null free codepaths.
Quoted 26 lines
Quoted 26 lines
kernel/bpf/verifier.c +25
-0+{ +struct bpf_func_state *state =func (env, key); +struct bpf_reg_state *reg; +int zero_size =0 ; +int stack_off; +u8 *stype; +int slot; +int spi; +int i; + +if (!env->bpf_capable) +return -1 ; +if (key->type !=PTR_TO_STACK ) +return -1 ; +if (!tnum_is_const (key->var_off)) +return -1 ; + + stack_off = key->off + key->var_off.value; + slot = -stack_off -1 ; + spi = slot /BPF_REG_SIZE ; + +/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++) + zero_size++;
Quoted 8 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9201
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+if (zero_size == key_size) +return 0 ; + +if (!is_spilled_reg (&state->stack[spi])) +/* Not pointer to stack */ +return -1 ;Nit: there is a '
is_spilled_scalar_reg' utility function.
Ack.
Quoted 15 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9209
Quoted 9 lines
kernel/bpf/verifier.c +8
-0+ + reg = &state->stack[spi].spilled_ptr; +if (reg->type !=SCALAR_VALUE ) +/* Only scalars are valid array map keys */ +return -1 ; +else if (!tnum_is_const (reg->var_off)) +/* Stack value not statically known */ +return -1 ;I think you need to check if size of the spill matches the size of the key. The mismatch would be unsafe when spill size is smaller than key size. E.g. consider 1-byte spill with mask 'mmmmmmrr' and a 4-byte key, at runtime the 'mmmmmm' part might be non-zero, rendering key to be out of range.
Ah great catch. I think you're right.
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
In reply to Daniel XuQuoted 28 lines · Daniel Xu
This commit allows progs to elide a null check on statically known map lookup keys. In other words, if the verifier can statically prove that the lookup will be in-bounds, allow the prog to drop the null check.
This is useful for two reasons:
1. Large numbers of nullness checks (especially when they cannot fail) unnecessarily pushes prog towards
BPF_COMPLEXITY_LIMIT_JMP_SEQ.
2. It forms a tighter contract between programmer and verifier.For (1), bpftrace is starting to make heavier use of percpu scratch maps. As a result, for user scripts with large number of unrolled loops, we are starting to hit jump complexity verification errors. These percpu lookups cannot fail anyways, as we only use static key values. Eliding nullness probably results in less work for verifier as well.
For (2), percpu scratch maps are often used as a larger stack, as the currrent stack is limited to 512 bytes. In these situations, it is desirable for the programmer to express: "this lookup should never fail, and if it does, it means I messed up the code". By omitting the null check, the programmer can "ask" the verifier to double check the logic.
Tests also have to be updated in sync with these changes, as the verifier is more efficient with this change. Notable,
iters.ctests had to be changed to use a map type that still requires null checks, as it's exercising verifier tracking logic w.r.t iterators.
Quoted 9 lines
Signed-off-by Daniel Xu <[email protected]>--- kernel/bpf/verifier.c | 80 ++++++++++++++++++- tools/testing/selftests/bpf/progs/iters.c | 14 ++-- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/verifier_map_in_map.c | 2 +- .../testing/selftests/bpf/verifier/map_kptr.c | 2 +- 5 files changed, 87 insertions(+), 13 deletions(-)
Eduard has great points. I've added a few more comments below.
pw-bot: cr
Quoted 34 lines
kernel/bpf/verifier.c +19
-0@@ -287,6 +287,7 @@ struct bpf_call_arg_meta {u32 ret_btf_id;u32 subprogno;struct btf_field *kptr_field; +s64 const_map_key; };struct bpf_kfunc_call_arg_meta {@@ -9163,6 +9164,53 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return 0 ; } +/* Returns constant key value if possible, else -1 */ +static s64 get_constant_map_key (struct bpf_verifier_env *env, +struct bpf_reg_state *key, +u32 key_size) +{ +struct bpf_func_state *state =func (env, key); +struct bpf_reg_state *reg; +int zero_size =0 ; +int stack_off; +u8 *stype; +int slot; +int spi; +int i; + +if (!env->bpf_capable) +return -1 ; +if (key->type !=PTR_TO_STACK ) +return -1 ;
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9194
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+if (!tnum_is_const (key->var_off)) +return -1 ; + + stack_off = key->off + key->var_off.value; + slot = -stack_off -1 ; + spi = slot /BPF_REG_SIZE ; + +/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++)
it's Friday and I'm lazy, but please double-check that this works for both big-endian and little-endian :)
with Eduard's suggestion this also becomes interesting when you have 000mmm mix (as one example), because that gives you a small range, and all values might be valid keys for arrays
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9200
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+ zero_size++; +if (zero_size == key_size) +return 0 ; + +if (!is_spilled_reg (&state->stack[spi])) +/* Not pointer to stack */
!is_spilled_reg and "Not pointer to stack" seem to be not exactly the same things?
btw, we also have is_spilled_scalar_reg() which you can use here instead of two separate checks?
Quoted 13 lines
kernel/bpf/verifier.c +13
-0+return -1 ; + + reg = &state->stack[spi].spilled_ptr; +if (reg->type !=SCALAR_VALUE ) +/* Only scalars are valid array map keys */ +return -1 ; +else if (!tnum_is_const (reg->var_off)) +/* Stack value not statically known */ +return -1 ; + +return reg->var_off.value; +} +
[...]
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 82 lines · Andrii Nakryiko
Quoted 28 lines · Daniel Xu
This commit allows progs to elide a null check on statically known map lookup keys. In other words, if the verifier can statically prove that the lookup will be in-bounds, allow the prog to drop the null check.
This is useful for two reasons:
1. Large numbers of nullness checks (especially when they cannot fail) unnecessarily pushes prog towards
BPF_COMPLEXITY_LIMIT_JMP_SEQ.
2. It forms a tighter contract between programmer and verifier.For (1), bpftrace is starting to make heavier use of percpu scratch maps. As a result, for user scripts with large number of unrolled loops, we are starting to hit jump complexity verification errors. These percpu lookups cannot fail anyways, as we only use static key values. Eliding nullness probably results in less work for verifier as well.
For (2), percpu scratch maps are often used as a larger stack, as the currrent stack is limited to 512 bytes. In these situations, it is desirable for the programmer to express: "this lookup should never fail, and if it does, it means I messed up the code". By omitting the null check, the programmer can "ask" the verifier to double check the logic.
Tests also have to be updated in sync with these changes, as the verifier is more efficient with this change. Notable,
iters.ctests had to be changed to use a map type that still requires null checks, as it's exercising verifier tracking logic w.r.t iterators.Quoted 9 lines
Signed-off-by Daniel Xu <[email protected]>--- kernel/bpf/verifier.c | 80 ++++++++++++++++++- tools/testing/selftests/bpf/progs/iters.c | 14 ++-- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/verifier_map_in_map.c | 2 +- .../testing/selftests/bpf/verifier/map_kptr.c | 2 +- 5 files changed, 87 insertions(+), 13 deletions(-)Eduard has great points. I've added a few more comments below.
pw-bot: cr
Quoted 39 lines
kernel/bpf/verifier.c +24
-0@@ -287,6 +287,7 @@ struct bpf_call_arg_meta {u32 ret_btf_id;u32 subprogno;struct btf_field *kptr_field; +s64 const_map_key; };struct bpf_kfunc_call_arg_meta {@@ -9163,6 +9164,53 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return 0 ; } +/* Returns constant key value if possible, else -1 */ +static s64 get_constant_map_key (struct bpf_verifier_env *env, +struct bpf_reg_state *key, +u32 key_size) +{ +struct bpf_func_state *state =func (env, key); +struct bpf_reg_state *reg; +int zero_size =0 ; +int stack_off; +u8 *stype; +int slot; +int spi; +int i; + +if (!env->bpf_capable) +return -1 ; +if (key->type !=PTR_TO_STACK ) +return -1 ; +if (!tnum_is_const (key->var_off)) +return -1 ; + + stack_off = key->off + key->var_off.value; + slot = -stack_off -1 ;
Quoted 8 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9194
Quoted 5 lines
kernel/bpf/verifier.c +5
-0+ spi = slot /BPF_REG_SIZE ; + +/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++)it's Friday and I'm lazy, but please double-check that this works for both big-endian and little-endian :)
Any tips? Are the existing tests running thru s390x hosts in CI sufficient or should I add some tests writen in C (and not BPF assembler)? I can never think about endianness correctly...
Quoted 4 lines
with Eduard's suggestion this also becomes interesting when you have 000mmm mix (as one example), because that gives you a small range, and all values might be valid keys for arrays
Can you define what "small range" means? What range is there with 0's? Any pointers would be helpful.
Quoted 10 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9200
Quoted 7 lines
kernel/bpf/verifier.c +6
-0+ zero_size++; +if (zero_size == key_size) +return 0 ; + +if (!is_spilled_reg (&state->stack[spi])) +/* Not pointer to stack */ !is_spilled_reg and "Not pointer to stack" seem to be not exactly the same things?
You're right - comment is not helpful. I'll make the change to use is_spilled_scalar_reg() which is probably as clear as it gets.
[..]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
On Fri, 2024-12-13 at 19:44 -0700, Daniel Xu wrote:
[...]
Quoted 10 lines
Quoted 6 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9194
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++)it's Friday and I'm lazy, but please double-check that this works for both big-endian and little-endian :)
Any tips? Are the existing tests running thru s390x hosts in CI sufficient or should I add some tests writen in C (and not BPF assembler)? I can never think about endianness correctly...
I think that if test operates on a key like:
valid key 15
v
0000000f <-- written to stack as a single u64 value
^^^^^^^
stack zero marks
and is executed (e.g. using __retval annotation),
then CI passing for s390 should be enough.
There is a guide on how to gen a s390 environment locally:
https://docs.kernel.org/bpf/s390.html
I used it recently to build a vmlinux for s390 with no or minimal issues. Used it to boot long time ago, but don't remember if there were any surprises.
Quoted 6 lines
Quoted 3 lines
with Eduard's suggestion this also becomes interesting when you have 000mmm mix (as one example), because that gives you a small range, and all values might be valid keys for arrays
Can you define what "small range" means? What range is there with 0's? Any pointers would be helpful.
I think Andrii means that each 'm' adds 8 bits of range. E.g. range for 0000_000m is 0-255, range for 0000_00mm is 0-65535, etc.
[...]
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 18 lines · Eduard Zingerman
On Fri, 2024-12-13 at 19:44 -0700, Daniel Xu wrote:
[...]
Quoted 10 lines
Quoted 6 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9194
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++)it's Friday and I'm lazy, but please double-check that this works for both big-endian and little-endian :)
Any tips? Are the existing tests running thru s390x hosts in CI sufficient or should I add some tests writen in C (and not BPF assembler)? I can never think about endianness correctly...
I think that if test operates on a key like:
Quoted 8 lines
valid key 15
v
0000000f<-- written to stack as a single u64 value
^^^^^^^
stack zero marksand is executed (e.g. using
__retvalannotation),
then CI passing for s390 should be enough.
+1, something like that where for big-endian it will be all zero while for little endian it would be 0xf (and then make sure that the test should *fail* by making sure that 0xf is not a valid index, so NULL check is necessary)
Quoted 11 lines
There is a guide on how to gen a s390 environment locally:
https://docs.kernel.org/bpf/s390.html
I used it recently to build a vmlinux for s390 with no or minimal issues. Used it to boot long time ago, but don't remember if there were any surprises.Quoted 4 lines
Quoted 3 lines
with Eduard's suggestion this also becomes interesting when you have 000mmm mix (as one example), because that gives you a small range, and all values might be valid keys for arrays
Quoted 5 lines
Quoted 2 lines
Can you define what "small range" means? What range is there with 0's? Any pointers would be helpful.
I think Andrii means that each 'm' adds 8 bits of range. E.g. range for 0000_000m is 0-255, range for 0000_00mm is 0-65535, etc.
yes, exactly, thank you, Eduard!
Quoted 3 lines
[...]
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 4 lines · Andrii Nakryiko
Quoted 3 lines · Eduard Zingerman
On Fri, 2024-12-13 at 19:44 -0700, Daniel Xu wrote:
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
with Eduard's suggestion this also becomes interesting when you have 000mmm mix (as one example), because that gives you a small range, and all values might be valid keys for arrays
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Can you define what "small range" means? What range is there with 0's? Any pointers would be helpful.
I think Andrii means that each 'm' adds 8 bits of range. E.g. range for 0000_000m is 0-255, range for 0000_00mm is 0-65535, etc.
yes, exactly, thank you, Eduard!
Gave it some thought. Still seems like a good idea, but I'd prefer to leave this extension for a separate patchset. Mostly b/c I'm running out of space in my head to grok everything :P. Probably higher likelihood of me getting the existing stuff correct if I don't add more scope.
Thanks,
Daniel
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
In reply to Andrii NakryikoQuoted 25 lines · Andrii Nakryiko
Quoted 24 lines · Eduard Zingerman
On Fri, 2024-12-13 at 19:44 -0700, Daniel Xu wrote:
[...]
Quoted 10 lines
Quoted 6 lines
kernel/bpf/verifier.c static int check_reg_const_str(struct bpf_verifier_env *env, · line 9194
Quoted 3 lines
kernel/bpf/verifier.c +3
-0+/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++)it's Friday and I'm lazy, but please double-check that this works for both big-endian and little-endian :)
Any tips? Are the existing tests running thru s390x hosts in CI sufficient or should I add some tests writen in C (and not BPF assembler)? I can never think about endianness correctly...
I think that if test operates on a key like:
valid key 15
v
0000000f<-- written to stack as a single u64 value
^^^^^^^
stack zero marks
Quoted 7 lines
Quoted 2 lines
and is executed (e.g. using
__retvalannotation),
then CI passing for s390 should be enough.+1, something like that where for big-endian it will be all zero while for little endian it would be
0xf(and then make sure that the test should *fail* by making sure that0xfis not a valid index, so NULL check is necessary)
How would it work for LE to be 0xF but BE to be 0x0?
The prog passes a pointer to the beginning of the u32 to bpf_map_lookup_elem(). The kernel does a 4 byte read starting from that address. On both BE and LE all 4 bytes will be interpreted. So set bits cannot just go away.
Am I missing something?
Thanks,
Daniel
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
On Thu, 2024-12-19 at 14:41 -0700, Daniel Xu wrote:
[...]
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
I think that if test operates on a key like:
valid key 15
v
0000000f<-- written to stack as a single u64 value
^^^^^^^
stack zero marksQuoted 7 lines
Quoted 2 lines
and is executed (e.g. using
__retvalannotation),
then CI passing for s390 should be enough.+1, something like that where for big-endian it will be all zero while for little endian it would be
0xf(and then make sure that the test should *fail* by making sure that0xfis not a valid index, so NULL check is necessary)
Quoted 8 lines
How would it work for LE to be
0xFbut BE to be0x0?The prog passes a pointer to the beginning of the u32 to
bpf_map_lookup_elem(). The kernel does a 4 byte read starting from that address. On both BE and LE all 4 bytes will be interpreted. So set bits cannot just go away.Am I missing something?
Ok, thinking a bit more, the best test I can come up with is:
u8 vals[8]; vals[0] = 0; ... vals[6] = 0; vals[7] = 0xf; p = bpf_map_lookup_elem(... vals ...); *p = 42;
For LE vals as u32 should be 0x0f;
For BE vals as u32 should be 0xf000_0000.
Hence, it is not safe to remove null check for this program. What would verifier think about the value of such key?
As far as I understand, there would be stack zero for for vals[0-6] and u8 stack spill for vals[7].
You were going to add a check for the spill size, which should help here. So, a negative test like above that checks that verifier complains that 'p' should be checked for nullness first?
If anyone has better test in mind, please speak-up.
[...]
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 39 lines · Eduard Zingerman
On Thu, 2024-12-19 at 14:41 -0700, Daniel Xu wrote:
[...]
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
I think that if test operates on a key like:
valid key 15
v
0000000f<-- written to stack as a single u64 value
^^^^^^^
stack zero marksQuoted 7 lines
Quoted 2 lines
and is executed (e.g. using
__retvalannotation),
then CI passing for s390 should be enough.+1, something like that where for big-endian it will be all zero while for little endian it would be
0xf(and then make sure that the test should *fail* by making sure that0xfis not a valid index, so NULL check is necessary)Quoted 8 lines
How would it work for LE to be
0xFbut BE to be0x0?The prog passes a pointer to the beginning of the u32 to
bpf_map_lookup_elem(). The kernel does a 4 byte read starting from that address. On both BE and LE all 4 bytes will be interpreted. So set bits cannot just go away.Am I missing something?
Ok, thinking a bit more, the best test I can come up with is:
u8 vals[8]; vals[0] = 0; ... vals[6] = 0; vals[7] = 0xf; p = bpf_map_lookup_elem(... vals ...); *p = 42;
Quoted 6 lines
For LE vals as u32 should be
0x0f;
For BE vals as u32 should be 0xf000_0000.
Hence, it is not safe to remove null check for this program. What would verifier think about the value of such key? As far as I understand, there would be stack zero for for vals[0-6] and u8 stack spill for vals[7].
Right. By checking that spill size is same as key size, we stay endian neutral, as constant values are tracked in native endianness.
However, if we were to start interpreting combinations of STACK_ZERO, STACK_MISC, and STACK_SPILL, the verifier would have to be endian aware (IIUC). Which makes it a somewhat interesting problem but also requires some thought to correctly handle the state space.
Quoted 5 lines
You were going to add a check for the spill size, which should help here. So, a negative test like above that checks that verifier complains that 'p' should be checked for nullness first?
If anyone has better test in mind, please speak-up.
I think this case reduces down to a spill_size != key_size test. As long as the sizes match, we don't have to worry about endianness.
Thanks,
Daniel
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
On Thu, 2024-12-19 at 17:40 -0700, Daniel Xu wrote:
[...]
Quoted 17 lines
Quoted 10 lines
Ok, thinking a bit more, the best test I can come up with is:
u8 vals[8]; vals[0] = 0; ... vals[6] = 0; vals[7] = 0xf; p = bpf_map_lookup_elem(... vals ...); *p = 42;Quoted 6 lines
For LE vals as u32 should be
0x0f;
For BE vals as u32 should be 0xf000_0000.
Hence, it is not safe to remove null check for this program. What would verifier think about the value of such key? As far as I understand, there would be stack zero for for vals[0-6] and u8 stack spill for vals[7].
Quoted 7 lines
Right. By checking that spill size is same as key size, we stay endian neutral, as constant values are tracked in native endianness.
However, if we were to start interpreting combinations of
STACK_ZERO,STACK_MISC, andSTACK_SPILL, the verifier would have to be endian aware (IIUC). Which makes it a somewhat interesting problem but also requires some thought to correctly handle the state space.
Right.
Quoted 8 lines
Quoted 5 lines
You were going to add a check for the spill size, which should help here. So, a negative test like above that checks that verifier complains that 'p' should be checked for nullness first?
If anyone has better test in mind, please speak-up.
I think this case reduces down to a
spill_size!=key_sizetest. As long as the sizes match, we don't have to worry about endianness.
Agree.
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 36 lines · Eduard Zingerman
On Thu, 2024-12-19 at 17:40 -0700, Daniel Xu wrote:
[...]
Quoted 17 lines
Quoted 10 lines
Ok, thinking a bit more, the best test I can come up with is:
u8 vals[8]; vals[0] = 0; ... vals[6] = 0; vals[7] = 0xf; p = bpf_map_lookup_elem(... vals ...); *p = 42;Quoted 6 lines
For LE vals as u32 should be
0x0f;
For BE vals as u32 should be 0xf000_0000.
Hence, it is not safe to remove null check for this program. What would verifier think about the value of such key? As far as I understand, there would be stack zero for for vals[0-6] and u8 stack spill for vals[7].Quoted 7 lines
Right. By checking that spill size is same as key size, we stay endian neutral, as constant values are tracked in native endianness.
However, if we were to start interpreting combinations of
STACK_ZERO,STACK_MISC, andSTACK_SPILL, the verifier would have to be endian aware (IIUC). Which makes it a somewhat interesting problem but also requires some thought to correctly handle the state space.Right.
Quoted 4 lines
Quoted 4 lines
You were going to add a check for the spill size, which should help here. So, a negative test like above that checks that verifier complains that 'p' should be checked for nullness first?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
If anyone has better test in mind, please speak-up.
I think this case reduces down to a
spill_size!=key_sizetest. As long as the sizes match, we don't have to worry about endianness.Agree.
Earlier I suggested to generalize this zero/misc/spill counting into a helper and reuse here and in check_stack_read_fixed_off().
We do very similar checks there with a similar purpose.
It sounds there are ideas to make this particular feature smarter than what we have in check_stack_read_fixed_off(). Let's not overdo it.
Even if a common helper is not possible, keep things consistent. The simpler the better.
Re: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
Quoted 42 lines · Alexei Starovoitov
Quoted 41 lines · Eduard Zingerman
On Thu, 2024-12-19 at 17:40 -0700, Daniel Xu wrote:
[...]
Quoted 17 lines
Quoted 10 lines
Ok, thinking a bit more, the best test I can come up with is:
u8 vals[8]; vals[0] = 0; ... vals[6] = 0; vals[7] = 0xf; p = bpf_map_lookup_elem(... vals ...); *p = 42;Quoted 6 lines
For LE vals as u32 should be
0x0f;
For BE vals as u32 should be 0xf000_0000.
Hence, it is not safe to remove null check for this program. What would verifier think about the value of such key? As far as I understand, there would be stack zero for for vals[0-6] and u8 stack spill for vals[7].Quoted 7 lines
Right. By checking that spill size is same as key size, we stay endian neutral, as constant values are tracked in native endianness.
However, if we were to start interpreting combinations of
STACK_ZERO,STACK_MISC, andSTACK_SPILL, the verifier would have to be endian aware (IIUC). Which makes it a somewhat interesting problem but also requires some thought to correctly handle the state space.Right.
Quoted 8 lines
Quoted 5 lines
You were going to add a check for the spill size, which should help here. So, a negative test like above that checks that verifier complains that 'p' should be checked for nullness first?
If anyone has better test in mind, please speak-up.
I think this case reduces down to a
spill_size!=key_sizetest. As long as the sizes match, we don't have to worry about endianness.
Quoted 6 lines
Quoted 1 line
Agree.
Earlier I suggested to generalize this zero/misc/spill counting into a helper and reuse here and in
check_stack_read_fixed_off().We do very similar checks there with a similar purpose.
Looked again, didn't see any obvious way to share code that doesn't make it more confusing. Let me post v6 without this particular refactor. If I missed something I'll fix it up in v7.
Quoted 6 lines
It sounds there are ideas to make this particular feature smarter than what we have in
check_stack_read_fixed_off(). Let's not overdo it.
Even if a common helper is not possible, keep things consistent. The simpler the better.
Fair enough. We can keep it simple.
Thanks,
Daniel
Kumar Kartikeya Dwivedi
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
In reply to Daniel XuQuoted 148 lines · Daniel Xu
This commit allows progs to elide a null check on statically known map lookup keys. In other words, if the verifier can statically prove that the lookup will be in-bounds, allow the prog to drop the null check.
This is useful for two reasons:
1. Large numbers of nullness checks (especially when they cannot fail) unnecessarily pushes prog towards
BPF_COMPLEXITY_LIMIT_JMP_SEQ.
2. It forms a tighter contract between programmer and verifier.For (1), bpftrace is starting to make heavier use of percpu scratch maps. As a result, for user scripts with large number of unrolled loops, we are starting to hit jump complexity verification errors. These percpu lookups cannot fail anyways, as we only use static key values. Eliding nullness probably results in less work for verifier as well.
For (2), percpu scratch maps are often used as a larger stack, as the currrent stack is limited to 512 bytes. In these situations, it is desirable for the programmer to express: "this lookup should never fail, and if it does, it means I messed up the code". By omitting the null check, the programmer can "ask" the verifier to double check the logic.
Tests also have to be updated in sync with these changes, as the verifier is more efficient with this change. Notable,
iters.ctests had to be changed to use a map type that still requires null checks, as it's exercising verifier tracking logic w.r.t iterators.Signed-off-by Daniel Xu <[email protected]>--- kernel/bpf/verifier.c | 80 ++++++++++++++++++- tools/testing/selftests/bpf/progs/iters.c | 14 ++-- .../selftests/bpf/progs/map_kptr_fail.c | 2 +- .../selftests/bpf/progs/verifier_map_in_map.c | 2 +- .../testing/selftests/bpf/verifier/map_kptr.c | 2 +- 5 files changed, 87 insertions(+), 13 deletions(-)kernel/bpf/verifier.c +69
-2@@ -287,6 +287,7 @@ struct bpf_call_arg_meta {u32 ret_btf_id;u32 subprogno;struct btf_field *kptr_field; +s64 const_map_key; };struct bpf_kfunc_call_arg_meta {@@ -9163,6 +9164,53 @@ static int check_reg_const_str(struct bpf_verifier_env *env,return 0 ; } +/* Returns constant key value if possible, else -1 */ +static s64 get_constant_map_key (struct bpf_verifier_env *env, +struct bpf_reg_state *key, +u32 key_size) +{ +struct bpf_func_state *state =func (env, key); +struct bpf_reg_state *reg; +int zero_size =0 ; +int stack_off; +u8 *stype; +int slot; +int spi; +int i; + +if (!env->bpf_capable) +return -1 ; +if (key->type !=PTR_TO_STACK ) +return -1 ; +if (!tnum_is_const (key->var_off)) +return -1 ; + + stack_off = key->off + key->var_off.value; + slot = -stack_off -1 ; + spi = slot /BPF_REG_SIZE ; + +/* First handle precisely tracked STACK_ZERO, up to BPF_REG_SIZE */ + stype = state->stack[spi].slot_type; +for (i =0 ; i <BPF_REG_SIZE && stype[i] ==STACK_ZERO ; i++) + zero_size++; +if (zero_size == key_size) +return 0 ; + +if (!is_spilled_reg (&state->stack[spi])) +/* Not pointer to stack */ +return -1 ; + + reg = &state->stack[spi].spilled_ptr; +if (reg->type !=SCALAR_VALUE ) +/* Only scalars are valid array map keys */ +return -1 ; +else if (!tnum_is_const (reg->var_off)) +/* Stack value not statically known */ +return -1 ; + +return reg->var_off.value; +} + static int check_func_arg(struct bpf_verifier_env *env, u32 arg, struct bpf_call_arg_meta *meta, const struct bpf_func_proto *fn,@@ -9173,6 +9221,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,enum bpf_arg_type arg_type = fn->arg_type[arg];enum bpf_reg_type type = reg->type;u32 *arg_btf_id =NULL ; +u32 key_size;int err =0 ;bool mask;@@ -9307,8 +9356,11 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg,verbose (env,"invalid map_ptr to access map->key\n" );return -EACCES ; } - err =check_helper_mem_access (env, regno, meta->map_ptr->key_size, -BPF_READ , false,NULL ); + key_size = meta->map_ptr->key_size; + err =check_helper_mem_access (env, regno, key_size,BPF_READ , false,NULL ); +if (err) +return err; + meta->const_map_key =get_constant_map_key (env, reg, key_size);break ;case ARG_PTR_TO_MAP_VALUE :if (type_may_be_null (arg_type) &®ister_is_null (reg))@@ -10833,6 +10885,21 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno state->callback_subprogno == subprogno); } +/* Returns whether or not the given map type can potentially elide +* lookup return value nullness check. This is possible if the key +* is statically known. +*/ +static bool can_elide_value_nullness (enum bpf_map_type type) +{ +switch (type) { +case BPF_MAP_TYPE_ARRAY : +case BPF_MAP_TYPE_PERCPU_ARRAY : +return true; +default : +return false; + } +} +static int get_helper_proto (struct bpf_verifier_env *env,int func_id,const struct bpf_func_proto **ptr) {
kernel/bpf/verifier.c static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · line 11274
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -11199,10 +11266,17 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn "kernel subsystem misconfigured verifier\n");return -EINVAL ; } + +if (func_id == BPF_FUNC_map_lookup_elem && +can_elide_value_nullness (meta.map_ptr->map_type) && + meta.const_map_key >=0 && + meta.const_map_key < meta.map_ptr->max_entries) + ret_flag &= ~PTR_MAYBE_NULL ;
I think we probably need mark_chain_precision applied on the constant key since its concrete value is made use of here to prevent pruning on it. If it's already happening and I missed it, I think we should atleast add a comment.
For context of a similar case with tail calls, see commit cc52d9140aa9 ("bpf: Fix record_func_key to perform backtracking on r3") for what happens when it is missed.
Quoted 112 lines
kernel/bpf/verifier.c +2
-1+ regs[BPF_REG_0 ].map_ptr = meta.map_ptr; regs[BPF_REG_0 ].map_uid = meta.map_uid; regs[BPF_REG_0 ].type =PTR_TO_MAP_VALUE | ret_flag; -if (!type_may_be_null (ret_type) && +if (!type_may_be_null (ret_flag) &&btf_record_has_field (meta.map_ptr->record,BPF_SPIN_LOCK )) { regs[BPF_REG_0 ].id = ++env->id_gen; }tools/testing/selftests/bpf/progs/iters.c +7
-7@@ -524,11 +524,11 @@ int iter_subprog_iters(const void *ctx) }struct { - __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(type, BPF_MAP_TYPE_HASH); __type(key, int); __type(value, int); __uint(max_entries, 1000); -} arr_map SEC(".maps"); +} hash_map SEC(".maps"); SEC("?raw_tp") __failure __msg("invalid mem access 'scalar'")@@ -539,7 +539,7 @@ int iter_err_too_permissive1(const void *ctx)MY_PID_GUARD (); - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key);if (!map_val)return 0 ;@@ -561,12 +561,12 @@ int iter_err_too_permissive2(const void *ctx)MY_PID_GUARD (); - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key);if (!map_val)return 0 ; bpf_repeat(1000000) { - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key); } *map_val =123 ;@@ -585,7 +585,7 @@ int iter_err_too_permissive3(const void *ctx)MY_PID_GUARD (); bpf_repeat(1000000) { - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key); found = true; }@@ -606,7 +606,7 @@ int iter_tricky_but_fine(const void *ctx)MY_PID_GUARD (); bpf_repeat(1000000) { - map_val =bpf_map_lookup_elem (&arr_map, &key); + map_val =bpf_map_lookup_elem (&hash_map, &key);if (map_val) { found = true;break ;tools/testing/selftests/bpf/progs/map_kptr_fail.c +1
-1@@ -345,7 +345,7 @@ int reject_indirect_global_func_access(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("Unreleased reference id=5 alloc_insn=") +__failure __msg("Unreleased reference id=4 alloc_insn=") int kptr_xchg_ref_state(struct __sk_buff *ctx) {struct prog_test_ref_kfunc *p;tools/testing/selftests/bpf/progs/verifier_map_in_map.c +1
-1@@ -47,7 +47,7 @@ l0_%=: r0 = 0; \ SEC("xdp") __description("map in map state pruning") -__success __msg("processed 26 insns") +__success __msg("processed 15 insns") __log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ) __naked void map_in_map_state_pruning(void) {tools/testing/selftests/bpf/verifier/map_kptr.c +1
-2@@ -373,7 +373,7 @@ .prog_type = BPF_PROG_TYPE_SCHED_CLS, .fixup_map_kptr = { 1 }, .result = REJECT, - .errstr = "Unreleased reference id=5 alloc_insn=20", + .errstr = "Unreleased reference id=4 alloc_insn=20", .fixup_kfunc_btf_id = { { "bpf_kfunc_call_test_acquire", 15 }, } --2.46.0
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
On Sat, 2024-12-14 at 00:10 +0100, Kumar Kartikeya Dwivedi wrote:
[...]
Quoted 16 lines
kernel/bpf/verifier.c static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · line 11274
Quoted 10 lines
kernel/bpf/verifier.c +6
-0@@ -11199,10 +11266,17 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn "kernel subsystem misconfigured verifier\n");return -EINVAL ; } + +if (func_id == BPF_FUNC_map_lookup_elem && +can_elide_value_nullness (meta.map_ptr->map_type) && + meta.const_map_key >=0 && + meta.const_map_key < meta.map_ptr->max_entries) + ret_flag &= ~PTR_MAYBE_NULL ;I think we probably need
mark_chain_precisionapplied on the constant key since its concrete value is made use of here to prevent pruning on it. If it's already happening and I missed it, I think we should atleast add a comment.
Quoted 3 lines
For context of a similar case with tail calls, see commit
cc52d9140aa9("bpf: Fixrecord_func_keyto perform backtracking on r3") for what happens when it is missed.
Great point, I'm sure this does not happen.
[...]
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 4/5] bpf: verifier: Support eliding map lookup nullness
On Fri, 2024-12-13 at 15:14 -0800, Eduard Zingerman wrote:
[...]
Quoted 1 line
Great point, I'm sure this does not happen.
I mean, mark_chain_precision() does not happen at the moment.
PATCH v5 5/5 bpf: selftests: verifier: Add nullness elision tests
In reply to Daniel XuTest that nullness elision works for common use cases. For example, we want to check that both full and subreg stack slots are recognized. As well as when there's both const and non-const values of R2 leading up to a lookup. And obviously some bound checks.
--- .../bpf/progs/verifier_array_access.c | 214 ++++++++++++++++++ 1 file changed, 214 insertions(+)
tools/testing/selftests/bpf/progs/verifier_array_access.c +214 -0
@@ -28,6 +28,20 @@ struct {__uint (map_flags,BPF_F_WRONLY_PROG ); } map_array_wo SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, struct test_val); +} map_array_pcpu SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, struct test_val); +} map_array SEC(".maps"); +struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 1);
@@ -525,4 +539,204 @@ l0_%=: exit; \ : __clobber_all); } +SEC("socket") +__description("valid map access into an array using constant without nullness") +__success __retval(4) +__naked void an_array_with_a_constant_no_nullness(void) +{ + asmvolatile (" \ +r1 = 1; \ +*(u64*)(r10 - 8) = r1; \ +r2 = r10; \ +r2 += -8; \ +r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +r1 = %[test_val_foo]; \ +*(u64*)(r0 + 0) = r1; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array), +__imm_const (test_val_foo, offsetof(struct test_val , foo)) + : __clobber_all); +} + +SEC("socket") +__description("valid multiple map access into an array using constant without nullness") +__success __retval(8) +__naked void multiple_array_with_a_constant_no_nullness(void) +{ + asmvolatile (" \ +r1 = 1; \ +*(u64*)(r10 - 8) = r1; \ +r2 = r10; \ +r2 += -8; \ +r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +r6 = %[test_val_foo]; \ +*(u64*)(r0 + 0) = r6; \ +r7 = *(u64*)(r0 + 0); \ +r1 = 0; \ +*(u64*)(r10 - 16) = r1; \ +r2 = r10; \ +r2 += -16; \ +r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +*(u64*)(r0 + 0) = r6; \ +r1 = *(u64*)(r0 + 0); \ +r7 += r1; \ +r0 = r7; \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array), +__imm_const (test_val_foo, offsetof(struct test_val , foo)) + : __clobber_all); +} + +SEC("socket") +__description("valid map access into an array using 32-bit constant without nullness") +__success __retval(4) +__naked void an_array_with_a_32bit_constant_no_nullness(void) +{ +/* 32-bit write must be to stack address aligned to BPF_REG_SIZE +* so that the spill is tracked. Unaligned subreg writes are less +* precisely tracked. +*/ + asmvolatile (" \ +r1 = 1; \ +*(u32*)(r10 - 8) = r1; \ +r2 = r10; \ +r2 += -8; \ +r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +r1 = %[test_val_foo]; \ +*(u64*)(r0 + 0) = r1; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array), +__imm_const (test_val_foo, offsetof(struct test_val , foo)) + : __clobber_all); +} + +SEC("socket") +__description("valid map access into an array using 32-bit constant 0 without nullness") +__success __retval(4) +__naked void an_array_with_a_32bit_constant_0_no_nullness(void) +{ +/* Unlike the above test, 32-bit zeroing is precisely tracked even +* if writes are not aligned to BPF_REG_SIZE. This tests that our +* STACK_ZERO handling functions. +*/ + asmvolatile (" \ +r1 = 0; \ +*(u32*)(r10 - 4) = r1; \ +r2 = r10; \ +r2 += -4; \ +r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +r1 = %[test_val_foo]; \ +*(u64*)(r0 + 0) = r1; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array), +__imm_const (test_val_foo, offsetof(struct test_val , foo)) + : __clobber_all); +} + +SEC("socket") +__description("valid map access into a pcpu array using constant without nullness") +__success __retval(4) +__naked void a_pcpu_array_with_a_constant_no_nullness(void) +{ + asmvolatile (" \ +r1 = 1; \ +*(u64*)(r10 - 8) = r1; \ +r2 = r10; \ +r2 += -8; \ +r1 = %[map_array_pcpu] ll; \ +call %[bpf_map_lookup_elem]; \ +r1 = %[test_val_foo]; \ +*(u64*)(r0 + 0) = r1; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array_pcpu), +__imm_const (test_val_foo, offsetof(struct test_val , foo)) + : __clobber_all); +} + +SEC("socket") +__description("invalid map access into an array using constant without nullness") +__failure __msg("R0 invalid mem access 'map_value_or_null'") +__naked void an_array_with_a_constant_no_nullness_out_of_bounds(void) +{ + asmvolatile (" \ +r1 = 3; \ +*(u64*)(r10 - 8) = r1; \ +r2 = r10; \ +r2 += -8; \ +r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +r1 = %[test_val_foo]; \ +*(u64*)(r0 + 0) = r1; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array), +__imm_const (test_val_foo, offsetof(struct test_val , foo)) + : __clobber_all); +} + +SEC("socket") +__description("invalid elided lookup using const and non-const key") +__failure __msg("R0 invalid mem access 'map_value_or_null'") +__naked void mixed_const_and_non_const_key_lookup(void) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +if r0 > 42 goto l1_%=; \ +*(u64*)(r10 - 8) = r0; \ +r2 = r10; \ +r2 += -8; \ +goto l0_%=; \ +l1_%=: r1 = 1; \ +*(u64*)(r10 - 8) = r1; \ +r2 = r10; \ +r2 += -8; \ +l0_%=: r1 = %[map_array] ll; \ +call %[bpf_map_lookup_elem]; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_get_prandom_u32), +__imm (bpf_map_lookup_elem), +__imm_addr (map_array) + : __clobber_all); +} + +SEC("socket") +__failure __msg("invalid read from stack R2 off=4096 size=4") +__naked void key_lookup_at_invalid_fp(void) +{ + asmvolatile (" \ +r1 = %[map_array] ll; \ +r2 = r10; \ +r2 += 4096; \ +call %[bpf_map_lookup_elem]; \ +r0 = *(u64*)(r0 + 0); \ +exit; \ +" : + : __imm(bpf_map_lookup_elem), +__imm_addr (map_array) + : __clobber_all); +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.46.0
Eduard Zingerman
maintainerRe: [PATCH bpf-next v5 5/5] bpf: selftests: verifier: Add nullness elision tests
Quoted 7 lines · Daniel Xu
Test that nullness elision works for common use cases. For example, we want to check that both full and subreg stack slots are recognized. As well as when there's both const and non-const values of R2 leading up to a lookup. And obviously some bound checks.
Signed-off-by Daniel Xu <[email protected]>---
Daniel,
since there would be a respin of this patch-set, maybe consider using plain C for some of the tests?
[...]
Re: [PATCH bpf-next v5 5/5] bpf: selftests: verifier: Add nullness elision tests
Quoted 6 lines · Eduard Zingerman
Quoted 5 lines · Daniel Xu
Test that nullness elision works for common use cases. For example, we want to check that both full and subreg stack slots are recognized. As well as when there's both const and non-const values of R2 leading up to a lookup. And obviously some bound checks.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Daniel Xu <[email protected]>---
Daniel,
since there would be a respin of this patch-set, maybe consider using plain C for some of the tests?
Yeah, makes sense. Will do for v6.
Thanks,
Daniel