bpf: backport of iterator and callback handling fixes
cover bpf: backport of iterator and callback handling fixes
This is a backport of two upstream patch-sets:
1. "exact states comparison for iterator convergence checks"
https://lore.kernel.org/all/[email protected]/
2. "verify callbacks as if they are called unknown number of times"
https://lore.kernel.org/all/[email protected]/
Both patch-sets fix BPF verifier logic related to handling loops:
for bpf iterators, and for helper functions that accept callback functions.
The backport of (2) was requested as a response to bug report by Mateusz Gienieczko <[email protected]>.
The (1) is a dependency of (2).
The patch-set was tested by running BPF verifier selftests on my local qemu-based setup.
Most of the commits could be cherry-picked but three required merging:
| Action | Upstream commit |
|--------+-------------------------------------------------------------------------------------------------|
| pick | 3c4e420cb653 ("bpf: move explored_state() closer to the beginning of verifier.c ") |
| pick | 4c97259abc9b ("bpf: extract same_callsites() as utility function ") |
| merge | 2793a8b015f7 ("bpf: exact states comparison for iterator convergence checks ") |
| pick | 389ede06c297 ("selftests/bpf: tests with delayed read/precision makrs in loop body ") |
| pick | 2a0992829ea3 ("bpf: correct loop detection for iterators convergence ") |
| pick | 64870feebecb ("selftests/bpf: test if state loops are detected in a tricky case ") |
| pick | b4d8239534fd ("bpf: print full verifier states on infinite loop detection ") |
| drop | dedd6c894110 ("Merge branch 'exact-states-comparison-for-iterator-convergence-checks' ") |
|--------+-------------------------------------------------------------------------------------------------|
| pick | 977bc146d4eb ("selftests/bpf: track tcp payload offset as scalar in xdp_synproxy ") |
| pick | 87eb0152bcc1 ("selftests/bpf: track string payload offset as scalar in strobemeta ") |
| pick | 683b96f9606a ("bpf: extract __check_reg_arg() utility function ") |
| pick | 58124a98cb8e ("bpf: extract setup_func_entry() utility function ") |
| merge | ab5cfac139ab ("bpf: verify callbacks as if they are called unknown number of times ") |
| pick | 958465e217db ("selftests/bpf: tests for iterating callbacks ") |
| pick | cafe2c21508a ("bpf: widening for callback iterators ") |
| pick | 9f3330aa644d ("selftests/bpf: test widening for iterating callbacks ") |
| merge | bb124da69c47 ("bpf: keep track of max number of bpf_loop callback iterations ") |
| pick | 57e2a52deeb1 ("selftests/bpf: check if max number of bpf_loop iterations is tracked ") |
| drop | acb12c859ac7 ("Merge branch 'verify-callbacks-as-if-they-are-called-unknown-number-of-times' ") |Note:
I don't know how deal with merge commits, so I just dropped those. These commits are empty but contain cover letters for both series, so it might be useful to pick those (how?).
Eduard Zingerman (17):
bpf: move explored_state() closer to the beginning of verifier.c
bpf: extract same_callsites() as utility function
bpf: exact states comparison for iterator convergence checks selftests/bpf: tests with delayed read/precision makrs in loop body
bpf: correct loop detection for iterators convergence selftests/bpf: test if state loops are detected in a tricky case
bpf: print full verifier states on infinite loop detection selftests/bpf: track tcp payload offset as scalar in xdp_synproxy selftests/bpf: track string payload offset as scalar in strobemeta
bpf: extract __check_reg_arg() utility function
bpf: extract setup_func_entry() utility function
bpf: verify callbacks as if they are called unknown number of times selftests/bpf: tests for iterating callbacks
bpf: widening for callback iterators
selftests/bpf: test widening for iterating callbacks
bpf: keep track of max number of bpf_loop callback iterations selftests/bpf: check if max number of bpf_loop iterations is tracked
include/linux/bpf_verifier.h | 32 + kernel/bpf/verifier.c | 875 ++++++++++++++---- .../selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/cb_refs.c | 1 + tools/testing/selftests/bpf/progs/iters.c | 695 ++++++++++++++ .../testing/selftests/bpf/progs/strobemeta.h | 78 +- .../bpf/progs/verifier_iterating_callbacks.c | 242 +++++ .../bpf/progs/verifier_subprog_precision.c | 86 +- .../selftests/bpf/progs/xdp_synproxy_kern.c | 84 +- 9 files changed, 1830 insertions(+), 265 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
Signature · 1 line
2.43.0
PATCH 1/17 bpf: move explored_state() closer to the beginning of verifier.c
[ Upstream commit 3c4e420cb653 ]
Subsequent patches would make use of explored_state() function. Move it up to avoid adding unnecessary prototype.
--- kernel/bpf/verifier.c | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-)
kernel/bpf/verifier.c +13 -15
@@ -1778,6 +1778,19 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,return 0 ; } +static u32 state_htab_size (struct bpf_verifier_env *env) +{ +return env->prog->len; +} + +static struct bpf_verifier_state_list **explored_state (struct bpf_verifier_env *env,int idx) +{ +struct bpf_verifier_state *cur = env->cur_state; +struct bpf_func_state *state = cur->frame[cur->curframe]; + +return &env->explored_states[(idx ^ state->callsite) %state_htab_size (env)]; +} +static void update_branch_counts (struct bpf_verifier_env *env,struct bpf_verifier_state *st) {while (st) {
@@ -14710,21 +14723,6 @@ enum { BRANCH = 2, }; -static u32 state_htab_size (struct bpf_verifier_env *env) -{ -return env->prog->len; -} - -static struct bpf_verifier_state_list **explored_state ( -struct bpf_verifier_env *env, -int idx) -{ -struct bpf_verifier_state *cur = env->cur_state; -struct bpf_func_state *state = cur->frame[cur->curframe]; - -return &env->explored_states[(idx ^ state->callsite) %state_htab_size (env)]; -} -static void mark_prune_point (struct bpf_verifier_env *env,int idx) { env->insn_aux_data[idx].prune_point = true;
Signature · 1 line
2.43.0
PATCH 2/17 bpf: extract same_callsites() as utility function
In reply to Eduard Zingerman[ Upstream commit 4c97259abc9b ]
Extract same_callsites() from clean_live_states() as a utility function. This function would be used by the next patch in the set.
--- kernel/bpf/verifier.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
kernel/bpf/verifier.c +15 -5
@@ -1791,6 +1791,20 @@ static struct bpf_verifier_state_list **explored_state(struct bpf_verifier_env *return &env->explored_states[(idx ^ state->callsite) %state_htab_size (env)]; } +static bool same_callsites (struct bpf_verifier_state *a,struct bpf_verifier_state *b) +{ +int fr; + +if (a->curframe != b->curframe) +return false; + +for (fr = a->curframe; fr >=0 ; fr--) +if (a->frame[fr]->callsite != b->frame[fr]->callsite) +return false; + +return true; +} +static void update_branch_counts (struct bpf_verifier_env *env,struct bpf_verifier_state *st) {while (st) {
@@ -15529,18 +15543,14 @@ static void clean_live_states(struct bpf_verifier_env *env, int insn, struct bpf_verifier_state *cur) {struct bpf_verifier_state_list *sl; -int i; sl = *explored_state (env, insn);while (sl) {if (sl->state.branches) goto next;if (sl->state.insn_idx != insn || - sl->state.curframe != cur->curframe) + !same_callsites (&sl->state, cur)) goto next; -for (i =0 ; i <= cur->curframe; i++) -if (sl->state.frame[i]->callsite != cur->frame[i]->callsite) - goto next;clean_verifier_state (env, &sl->state); next: sl = sl->next;
Signature · 1 line
2.43.0
PATCH 3/17 bpf: exact states comparison for iterator convergence checks
In reply to Eduard Zingerman[ Upstream commit 2793a8b015f7 ]
Convergence for open coded iterators is computed in is_state_visited() by examining states with branches count > 1 and using states_equal(). states_equal() computes sub-state relation using read and precision marks. Read and precision marks are propagated from children states, thus are not guaranteed to be complete inside a loop when branches count > 1. This could be demonstrated using the following unsafe program:
1. r7 = -16
2. r6 = bpf_get_prandom_u32()
3. while (bpf_iter_num_next(&fp[-8])) {
4. if (r6 != 42) {
5. r7 = -32
6. r6 = bpf_get_prandom_u32()
7. continue
8. }
9. r0 = r10
10. r0 += r7
11. r8 = *(u64 *)(r0 + 0)
12. r6 = bpf_get_prandom_u32()
13. }
Here verifier would first visit path 1-3, create a checkpoint at 3 with r7=-16, continue to 4-7,3 with r7=-32.
Because instructions at 9-12 had not been visitied yet existing checkpoint at 3 does not have read or precision mark for r7. Thus states_equal() would return true and verifier would discard current state, thus unsafe memory access at 11 would not be caught.
This commit fixes this loophole by introducing exact state comparisons for iterator convergence logic:
- registers are compared using regs_exact() regardless of read or precision marks;
- stack slots have to have identical type.
Unfortunately, this is too strict even for simple programs like below:
i = 0;
while(iter_next(&it))
i++;At each iteration step i++ would produce a new distinct state and eventually instruction processing limit would be reached.
To avoid such behavior speculatively forget (widen) range for imprecise scalar registers, if those registers were not precise at the end of the previous iteration and do not match exactly.
This a conservative heuristic that allows to verify wide range of programs, however it precludes verification of programs that conjure an imprecise value on the first loop iteration and use it as precise on the second.
Test case iter_task_vma_for_each() presents one of such cases:
unsigned int seen = 0;
...
bpf_for_each(task_vma, vma, task, 0) {
if (seen >= 1000)
break;
...
seen++;
}Here clang generates the following code:
<LBB0_4>:
24: r8 = r6 ; stash current value of
... body ... 'seen'
29: r1 = r10
30: r1 += -0x8
31: call bpf_iter_task_vma_next
32: r6 += 0x1 ; seen++;
33: if r0 == 0x0 goto +0x2 <LBB0_6> ; exit on next() == NULL
34: r7 += 0x10
35: if r8 < 0x3e7 goto -0xc <LBB0_4> ; loop on seen < 1000<LBB0_6>:
... exit ...
Note that counter in r6 is copied to r8 and then incremented, conditional jump is done using r8. Because of this precision mark for r6 lags one state behind of precision mark on r8 and widening logic kicks in.
Adding barrier_var(seen) after conditional is sufficient to force clang use the same register for both counting and conditional jump.
This issue was discussed in the thread [1] which was started by Andrew Werner <[email protected]> demonstrating a similar bug in callback functions handling. The callbacks would be addressed in a followup patch.
[1] https://lore.kernel.org/bpf/[email protected]/
[ Summary of the conflicts and their resolutions ]
Original patch contained a change to test case tools/testing/selftests/bpf/progs/iters_task_vma.c this change was dropped.
The test case was added by the following commit:e0e1a7a5fc37 ("selftests/bpf: Add tests for open-coded task_vma iter") Which is not a selected for porting to 6.6.y at the moment.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 218 ++++++++++++++++++++++++++++++----- 2 files changed, 188 insertions(+), 31 deletions(-)
include/linux/bpf_verifier.h +1 -0
@@ -383,6 +383,7 @@ struct bpf_verifier_state {*/ struct bpf_idx_pair *jmp_history;u32 jmp_history_cnt; +u32 dfs_depth; };#define bpf_get_spilled_reg (slot, frame) \
kernel/bpf/verifier.c +187 -31
349 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH 4/17 selftests/bpf: tests with delayed read/precision makrs in loop body
In reply to Eduard Zingerman[ Upstream commit 389ede06c297 ]
These test cases try to hide read and precision marks from loop convergence logic: marks would only be assigned on subsequent loop iterations or after exploring states pushed to env->head stack first. Without verifier fix to use exact states comparison logic for iterators convergence these tests (except 'triple_continue') would be errorneously marked as safe.
--- tools/testing/selftests/bpf/progs/iters.c | 518 ++++++++++++++++++++++ 1 file changed, 518 insertions(+)
tools/testing/selftests/bpf/progs/iters.c +518 -0
@@ -14,6 +14,13 @@ int my_pid;int arr[256 ]; int small_arr[16] SEC(".data.small_arr"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 10); + __type(key, int); + __type(value, int); +} amap SEC(".maps"); +#ifdef REAL_TEST #define MY_PID_GUARD () if (my_pid != (bpf_get_current_pid_tgid() >> 32)) return 0#else
@@ -716,4 +723,515 @@ int iter_pass_iter_ptr_to_subprog(const void *ctx)return 0 ; } +SEC("?raw_tp") +__failure +__msg("R1 type=scalar expected=fp") +__naked int delayed_read_mark(void) +{ +/* This is equivalent to C program below. +* The call to bpf_iter_num_next() is reachable with r7 values &fp[-16] and 0xdead. +* State with r7=&fp[-16] is visited first and follows r6 != 42 ... continue branch. +* At this point iterator next() call is reached with r7 that has no read mark. +* Loop body with r7=0xdead would only be visited if verifier would decide to continue +* with second loop iteration. Absence of read mark on r7 might affect state +* equivalent logic used for iterator convergence tracking. +* +* r7 = &fp[-16] +* fp[-16] = 0 +* r6 = bpf_get_prandom_u32() +* bpf_iter_num_new(&fp[-8], 0, 10) +* while (bpf_iter_num_next(&fp[-8])) { +* r6++ +* if (r6 != 42) { +* r7 = 0xdead +* continue; +* } +* bpf_probe_read_user(r7, 8, 0xdeadbeef); // this is not safe +* } +* bpf_iter_num_destroy(&fp[-8]) +* return 0 +*/ + asmvolatile ( +"r7 = r10;" +"r7 += -16;" +"r0 = 0;" +"*(u64 *)(r7 + 0) = r0;" +"call %[bpf_get_prandom_u32];" +"r6 = r0;" +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"1:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto 2f;" +"r6 += 1;" +"if r6 != 42 goto 3f;" +"r7 = 0xdead;" +"goto 1b;" +"3:" +"r1 = r7;" +"r2 = 8;" +"r3 = 0xdeadbeef;" +"call %[bpf_probe_read_user];" +"goto 1b;" +"2:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy), +__imm (bpf_probe_read_user) + : __clobber_all + ); +} + +SEC("?raw_tp") +__failure +__msg("math between fp pointer and register with unbounded") +__naked int delayed_precision_mark(void) +{ +/* This is equivalent to C program below. +* The test is similar to delayed_iter_mark but verifies that incomplete +* precision don't fool verifier. +* The call to bpf_iter_num_next() is reachable with r7 values -16 and -32. +* State with r7=-16 is visited first and follows r6 != 42 ... continue branch. +* At this point iterator next() call is reached with r7 that has no read +* and precision marks. +* Loop body with r7=-32 would only be visited if verifier would decide to continue +* with second loop iteration. Absence of precision mark on r7 might affect state +* equivalent logic used for iterator convergence tracking. +* +* r8 = 0 +* fp[-16] = 0 +* r7 = -16 +* r6 = bpf_get_prandom_u32() +* bpf_iter_num_new(&fp[-8], 0, 10) +* while (bpf_iter_num_next(&fp[-8])) { +* if (r6 != 42) { +* r7 = -32 +* r6 = bpf_get_prandom_u32() +* continue; +* } +* r0 = r10 +* r0 += r7 +* r8 = *(u64 *)(r0 + 0) // this is not safe +* r6 = bpf_get_prandom_u32() +* } +* bpf_iter_num_destroy(&fp[-8]) +* return r8 +*/ + asmvolatile ( +"r8 = 0;" +"*(u64 *)(r10 - 16) = r8;" +"r7 = -16;" +"call %[bpf_get_prandom_u32];" +"r6 = r0;" +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"1:" +"r1 = r10;" +"r1 += -8;\n" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto 2f;" +"if r6 != 42 goto 3f;" +"r7 = -32;" +"call %[bpf_get_prandom_u32];" +"r6 = r0;" +"goto 1b;\n" +"3:" +"r0 = r10;" +"r0 += r7;" +"r8 = *(u64 *)(r0 + 0);" +"call %[bpf_get_prandom_u32];" +"r6 = r0;" +"goto 1b;\n" +"2:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r0 = r8;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy), +__imm (bpf_probe_read_user) + : __clobber_all + ); +} + +SEC("?raw_tp") +__failure +__msg("math between fp pointer and register with unbounded") +__flag (BPF_F_TEST_STATE_FREQ) +__naked int loop_state_deps1(void) +{ +/* This is equivalent to C program below. +* +* The case turns out to be tricky in a sense that: +* - states with c=-25 are explored only on a second iteration +* of the outer loop; +* - states with read+precise mark on c are explored only on +* second iteration of the inner loop and in a state which +* is pushed to states stack first. +* +* Depending on the details of iterator convergence logic +* verifier might stop states traversal too early and miss +* unsafe c=-25 memory access. +* +* j = iter_new(); // fp[-16] +* a = 0; // r6 +* b = 0; // r7 +* c = -24; // r8 +* while (iter_next(j)) { +* i = iter_new(); // fp[-8] +* a = 0; // r6 +* b = 0; // r7 +* while (iter_next(i)) { +* if (a == 1) { +* a = 0; +* b = 1; +* } else if (a == 0) { +* a = 1; +* if (random() == 42) +* continue; +* if (b == 1) { +* *(r10 + c) = 7; // this is not safe +* iter_destroy(i); +* iter_destroy(j); +* return; +* } +* } +* } +* iter_destroy(i); +* a = 0; +* b = 0; +* c = -25; +* } +* iter_destroy(j); +* return; +*/ + asmvolatile ( +"r1 = r10;" +"r1 += -16;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"r6 = 0;" +"r7 = 0;" +"r8 = -24;" +"j_loop_%=:" +"r1 = r10;" +"r1 += -16;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto j_loop_end_%=;" +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"r6 = 0;" +"r7 = 0;" +"i_loop_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto i_loop_end_%=;" +"check_one_r6_%=:" +"if r6 != 1 goto check_zero_r6_%=;" +"r6 = 0;" +"r7 = 1;" +"goto i_loop_%=;" +"check_zero_r6_%=:" +"if r6 != 0 goto i_loop_%=;" +"r6 = 1;" +"call %[bpf_get_prandom_u32];" +"if r0 != 42 goto check_one_r7_%=;" +"goto i_loop_%=;" +"check_one_r7_%=:" +"if r7 != 1 goto i_loop_%=;" +"r0 = r10;" +"r0 += r8;" +"r1 = 7;" +"*(u64 *)(r0 + 0) = r1;" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r1 = r10;" +"r1 += -16;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" +"i_loop_end_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r6 = 0;" +"r7 = 0;" +"r8 = -25;" +"goto j_loop_%=;" +"j_loop_end_%=:" +"r1 = r10;" +"r1 += -16;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy) + : __clobber_all + ); +} + +SEC("?raw_tp") +__success +__naked int triple_continue (void ) +{ +/* This is equivalent to C program below. +* High branching factor of the loop body turned out to be +* problematic for one of the iterator convergence tracking +* algorithms explored. +* +* r6 = bpf_get_prandom_u32() +* bpf_iter_num_new(&fp[-8], 0, 10) +* while (bpf_iter_num_next(&fp[-8])) { +* if (bpf_get_prandom_u32() != 42) +* continue; +* if (bpf_get_prandom_u32() != 42) +* continue; +* if (bpf_get_prandom_u32() != 42) +* continue; +* r0 += 0; +* } +* bpf_iter_num_destroy(&fp[-8]) +* return 0 +*/ + asmvolatile ( +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"loop_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto loop_end_%=;" +"call %[bpf_get_prandom_u32];" +"if r0 != 42 goto loop_%=;" +"call %[bpf_get_prandom_u32];" +"if r0 != 42 goto loop_%=;" +"call %[bpf_get_prandom_u32];" +"if r0 != 42 goto loop_%=;" +"r0 += 0;" +"goto loop_%=;" +"loop_end_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy) + : __clobber_all + ); +} + +SEC("?raw_tp") +__success +__naked int widen_spill (void ) +{ +/* This is equivalent to C program below. +* The counter is stored in fp[-16], if this counter is not widened +* verifier states representing loop iterations would never converge. +* +* fp[-16] = 0 +* bpf_iter_num_new(&fp[-8], 0, 10) +* while (bpf_iter_num_next(&fp[-8])) { +* r0 = fp[-16]; +* r0 += 1; +* fp[-16] = r0; +* } +* bpf_iter_num_destroy(&fp[-8]) +* return 0 +*/ + asmvolatile ( +"r0 = 0;" +"*(u64 *)(r10 - 16) = r0;" +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"loop_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto loop_end_%=;" +"r0 = *(u64 *)(r10 - 16);" +"r0 += 1;" +"*(u64 *)(r10 - 16) = r0;" +"goto loop_%=;" +"loop_end_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy) + : __clobber_all + ); +} + +SEC("raw_tp") +__success +__naked int checkpoint_states_deletion (void ) +{ +/* This is equivalent to C program below. +* +* int *a, *b, *c, *d, *e, *f; +* int i, sum = 0; +* bpf_for(i, 0, 10) { +* a = bpf_map_lookup_elem(&amap, &i); +* b = bpf_map_lookup_elem(&amap, &i); +* c = bpf_map_lookup_elem(&amap, &i); +* d = bpf_map_lookup_elem(&amap, &i); +* e = bpf_map_lookup_elem(&amap, &i); +* f = bpf_map_lookup_elem(&amap, &i); +* if (a) sum += 1; +* if (b) sum += 1; +* if (c) sum += 1; +* if (d) sum += 1; +* if (e) sum += 1; +* if (f) sum += 1; +* } +* return 0; +* +* The body of the loop spawns multiple simulation paths +* with different combination of NULL/non-NULL information for a/b/c/d/e/f. +* Each combination is unique from states_equal() point of view. +* Explored states checkpoint is created after each iterator next call. +* Iterator convergence logic expects that eventually current state +* would get equal to one of the explored states and thus loop +* exploration would be finished (at-least for a specific path). +* Verifier evicts explored states with high miss to hit ratio +* to to avoid comparing current state with too many explored +* states per instruction. +* This test is designed to "stress test" eviction policy defined using formula: +* +* sl->miss_cnt > sl->hit_cnt * N + N // if true sl->state is evicted +* +* Currently N is set to 64, which allows for 6 variables in this test. +*/ + asmvolatile ( +"r6 = 0;" /* a */ +"r7 = 0;" /* b */ +"r8 = 0;" /* c */ +"*(u64 *)(r10 - 24) = r6;" /* d */ +"*(u64 *)(r10 - 32) = r6;" /* e */ +"*(u64 *)(r10 - 40) = r6;" /* f */ +"r9 = 0;" /* sum */ +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"loop_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto loop_end_%=;" + +"*(u64 *)(r10 - 16) = r0;" + +"r1 = %[amap] ll;" +"r2 = r10;" +"r2 += -16;" +"call %[bpf_map_lookup_elem];" +"r6 = r0;" + +"r1 = %[amap] ll;" +"r2 = r10;" +"r2 += -16;" +"call %[bpf_map_lookup_elem];" +"r7 = r0;" + +"r1 = %[amap] ll;" +"r2 = r10;" +"r2 += -16;" +"call %[bpf_map_lookup_elem];" +"r8 = r0;" + +"r1 = %[amap] ll;" +"r2 = r10;" +"r2 += -16;" +"call %[bpf_map_lookup_elem];" +"*(u64 *)(r10 - 24) = r0;" + +"r1 = %[amap] ll;" +"r2 = r10;" +"r2 += -16;" +"call %[bpf_map_lookup_elem];" +"*(u64 *)(r10 - 32) = r0;" + +"r1 = %[amap] ll;" +"r2 = r10;" +"r2 += -16;" +"call %[bpf_map_lookup_elem];" +"*(u64 *)(r10 - 40) = r0;" + +"if r6 == 0 goto +1;" +"r9 += 1;" +"if r7 == 0 goto +1;" +"r9 += 1;" +"if r8 == 0 goto +1;" +"r9 += 1;" +"r0 = *(u64 *)(r10 - 24);" +"if r0 == 0 goto +1;" +"r9 += 1;" +"r0 = *(u64 *)(r10 - 32);" +"if r0 == 0 goto +1;" +"r9 += 1;" +"r0 = *(u64 *)(r10 - 40);" +"if r0 == 0 goto +1;" +"r9 += 1;" + +"goto loop_%=;" +"loop_end_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm (bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy), +__imm_addr (amap) + : __clobber_all + ); +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.43.0
PATCH 5/17 bpf: correct loop detection for iterators convergence
In reply to Eduard Zingerman[ Upstream commit 2a0992829ea3 ]
It turns out that .branches > 0 in is_state_visited() is not a sufficient condition to identify if two verifier states form a loop when iterators convergence is computed. This commit adds logic to distinguish situations like below:
(I) initial (II) initial
| |
V V
.---------> hdr ..
| | |
| V V
| .------... .------..
| | | | |
| V V V V
| ... ... .-> hdr ..
| | | | | |
| V V | V V
| succ <- cur | succ <- cur
| | | |
| V | V
| ... | ...
| | | |
'----' '----'For both (I) and (II) successor 'succ' of the current state 'cur' was previously explored and has branches count at 0. However, loop entry 'hdr' corresponding to 'succ' might be a part of current DFS path. If that is the case 'succ' and 'cur' are members of the same loop and have to be compared exactly.
--- include/linux/bpf_verifier.h | 15 +++ kernel/bpf/verifier.c | 207 ++++++++++++++++++++++++++++++++++- 2 files changed, 218 insertions(+), 4 deletions(-)
include/linux/bpf_verifier.h +15 -0
@@ -372,10 +372,25 @@ struct bpf_verifier_state {struct bpf_active_lock active_lock;bool speculative;bool active_rcu_lock; +/* If this state was ever pointed-to by other state's loop_entry field +* this flag would be set to true. Used to avoid freeing such states +* while they are still in use. +*/ +bool used_as_loop_entry;/* first and last insn idx of this verifier state */ u32 first_insn_idx;u32 last_insn_idx; +/* If this state is a part of states loop this field points to some +* parent of this state such that: +* - it is also a member of the same states loop; +* - DFS states traversal starting from initial state visits loop_entry +* state before this state. +* Used to compute topmost loop entry for state loops. +* State loops might appear because of open coded iterators logic. +* See get_loop_entry() for more information. +*/ +struct bpf_verifier_state *loop_entry;/* jmp history recorded from first to last. * backtracking is using it to go from last to first. * For most states jmp_history_cnt is [0-3].
kernel/bpf/verifier.c +203 -4
252 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH 6/17 selftests/bpf: test if state loops are detected in a tricky case
In reply to Eduard Zingerman[ Upstream commit 64870feebecb ]
A convoluted test case for iterators convergence logic that demonstrates that states with branch count equal to 0 might still be a part of not completely explored loop.
E.g. consider the following state diagram:
initial Here state 'succ' was processed first,
| it was eventually tracked to produce a
V state identical to 'hdr'.
.---------> hdr All branches from 'succ' had been explored
| | and thus 'succ' has its .branches == 0.
| V
| .------... Suppose states 'cur' and 'succ' correspond
| | | to the same instruction + callsites.
| V V In such case it is necessary to check
| ... ... whether 'succ' and 'cur' are identical.
| | | If 'succ' and 'cur' are a part of the same loop
| V V they have to be compared exactly.
| succ <- cur
| |
| V
| ...
| |
'----'--- tools/testing/selftests/bpf/progs/iters.c | 177 ++++++++++++++++++++++ 1 file changed, 177 insertions(+)
tools/testing/selftests/bpf/progs/iters.c +177 -0
@@ -998,6 +998,183 @@ __naked int loop_state_deps1(void) ); } +SEC("?raw_tp") +__failure +__msg("math between fp pointer and register with unbounded") +__flag (BPF_F_TEST_STATE_FREQ) +__naked int loop_state_deps2(void) +{ +/* This is equivalent to C program below. +* +* The case turns out to be tricky in a sense that: +* - states with read+precise mark on c are explored only on a second +* iteration of the first inner loop and in a state which is pushed to +* states stack first. +* - states with c=-25 are explored only on a second iteration of the +* second inner loop and in a state which is pushed to states stack +* first. +* +* Depending on the details of iterator convergence logic +* verifier might stop states traversal too early and miss +* unsafe c=-25 memory access. +* +* j = iter_new(); // fp[-16] +* a = 0; // r6 +* b = 0; // r7 +* c = -24; // r8 +* while (iter_next(j)) { +* i = iter_new(); // fp[-8] +* a = 0; // r6 +* b = 0; // r7 +* while (iter_next(i)) { +* if (a == 1) { +* a = 0; +* b = 1; +* } else if (a == 0) { +* a = 1; +* if (random() == 42) +* continue; +* if (b == 1) { +* *(r10 + c) = 7; // this is not safe +* iter_destroy(i); +* iter_destroy(j); +* return; +* } +* } +* } +* iter_destroy(i); +* i = iter_new(); // fp[-8] +* a = 0; // r6 +* b = 0; // r7 +* while (iter_next(i)) { +* if (a == 1) { +* a = 0; +* b = 1; +* } else if (a == 0) { +* a = 1; +* if (random() == 42) +* continue; +* if (b == 1) { +* a = 0; +* c = -25; +* } +* } +* } +* iter_destroy(i); +* } +* iter_destroy(j); +* return; +*/ + asmvolatile ( +"r1 = r10;" +"r1 += -16;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"r6 = 0;" +"r7 = 0;" +"r8 = -24;" +"j_loop_%=:" +"r1 = r10;" +"r1 += -16;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto j_loop_end_%=;" + +/* first inner loop */ +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"r6 = 0;" +"r7 = 0;" +"i_loop_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto i_loop_end_%=;" +"check_one_r6_%=:" +"if r6 != 1 goto check_zero_r6_%=;" +"r6 = 0;" +"r7 = 1;" +"goto i_loop_%=;" +"check_zero_r6_%=:" +"if r6 != 0 goto i_loop_%=;" +"r6 = 1;" +"call %[bpf_get_prandom_u32];" +"if r0 != 42 goto check_one_r7_%=;" +"goto i_loop_%=;" +"check_one_r7_%=:" +"if r7 != 1 goto i_loop_%=;" +"r0 = r10;" +"r0 += r8;" +"r1 = 7;" +"*(u64 *)(r0 + 0) = r1;" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" +"r1 = r10;" +"r1 += -16;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" +"i_loop_end_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" + +/* second inner loop */ +"r1 = r10;" +"r1 += -8;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"r6 = 0;" +"r7 = 0;" +"i2_loop_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto i2_loop_end_%=;" +"check2_one_r6_%=:" +"if r6 != 1 goto check2_zero_r6_%=;" +"r6 = 0;" +"r7 = 1;" +"goto i2_loop_%=;" +"check2_zero_r6_%=:" +"if r6 != 0 goto i2_loop_%=;" +"r6 = 1;" +"call %[bpf_get_prandom_u32];" +"if r0 != 42 goto check2_one_r7_%=;" +"goto i2_loop_%=;" +"check2_one_r7_%=:" +"if r7 != 1 goto i2_loop_%=;" +"r6 = 0;" +"r8 = -25;" +"goto i2_loop_%=;" +"i2_loop_end_%=:" +"r1 = r10;" +"r1 += -8;" +"call %[bpf_iter_num_destroy];" + +"r6 = 0;" +"r7 = 0;" +"goto j_loop_%=;" +"j_loop_end_%=:" +"r1 = r10;" +"r1 += -16;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy) + : __clobber_all + ); +} + SEC("?raw_tp") __success__naked int triple_continue (void )
Signature · 1 line
2.43.0
PATCH 7/17 bpf: print full verifier states on infinite loop detection
In reply to Eduard Zingerman[ Upstream commit b4d8239534fd ]
Additional logging in is_state_visited(): if infinite loop is detected print full verifier state for both current and equivalent states.
--- kernel/bpf/verifier.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/bpf/verifier.c +4 -0
@@ -16548,6 +16548,10 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) !iter_active_depths_differ(&sl->state, cur)) {verbose_linfo (env, insn_idx,"; " );verbose (env,"infinite loop detected at insn %d\n" , insn_idx); +verbose (env,"cur state:" ); +print_verifier_state (env, cur->frame[cur->curframe], true); +verbose (env,"old state:" ); +print_verifier_state (env, sl->state.frame[cur->curframe], true);return -EINVAL ; }/* if the verifier is processing a loop, avoid adding new state
Signature · 1 line
2.43.0
PATCH 8/17 selftests/bpf: track tcp payload offset as scalar in xdp_synproxy
In reply to Eduard Zingerman[ Upstream commit 977bc146d4eb ]
This change prepares syncookie_{tc,xdp} for update in callbakcs verification logic. To allow bpf_loop() verification converge when multiple callback itreations are considered:
- track offset inside TCP payload explicitly, not as a part of the pointer;
- make sure that offset does not exceed MAX_PACKET_OFF enforced by verifier;
- make sure that offset is tracked as unbound scalar between iterations, otherwise verifier won't be able infer that bpf_loop callback reaches identical states.
--- .../selftests/bpf/progs/xdp_synproxy_kern.c | 84 ++++++++++++------- 1 file changed, 52 insertions(+), 32 deletions(-)
tools/testing/selftests/bpf/progs/xdp_synproxy_kern.c +52 -32
@@ -53,6 +53,8 @@#define DEFAULT_TTL 64#define MAX_ALLOWED_PORTS 8 +#define MAX_PACKET_OFF 0xffff +#define swap (a, b) \ do { typeof(a) __tmp = (a); (a) = (b); (b) = __tmp; } while (0)
@@ -183,63 +185,76 @@ static __always_inline __u32 tcp_time_stamp_raw(void) }struct tcpopt_context { -__u8 *ptr; -__u8 *end; +void *data;void *data_end;__be32 *tsecr;__u8 wscale;bool option_timestamp;bool option_sack; +__u32 off; }; -static int tscookie_tcpopt_parse (struct tcpopt_context *ctx) +static __always_inline u8 *next(struct tcpopt_context *ctx, __u32 sz) { -__u8 opcode, opsize; +__u64 off = ctx->off; +__u8 *data; -if (ctx->ptr >= ctx->end) -return 1 ; -if (ctx->ptr >= ctx->data_end) -return 1 ; +/* Verifier forbids access to packet when offset exceeds MAX_PACKET_OFF */ +if (off >MAX_PACKET_OFF - sz) +return NULL ; - opcode = ctx->ptr[0 ]; + data = ctx->data + off; +barrier_var (data); +if (data + sz >= ctx->data_end) +return NULL ; -if (opcode ==TCPOPT_EOL ) -return 1 ; -if (opcode ==TCPOPT_NOP ) { - ++ctx->ptr; -return 0 ; - } + ctx->off += sz; +return data; +} -if (ctx->ptr +1 >= ctx->end) -return 1 ; -if (ctx->ptr +1 >= ctx->data_end) +static int tscookie_tcpopt_parse (struct tcpopt_context *ctx) +{ +__u8 *opcode, *opsize, *wscale, *tsecr; +__u32 off = ctx->off; + + opcode =next (ctx,1 ); +if (!opcode)return 1 ; - opsize = ctx->ptr[1 ]; -if (opsize <2 ) + +if (*opcode ==TCPOPT_EOL )return 1 ; +if (*opcode ==TCPOPT_NOP ) +return 0 ; -if (ctx->ptr + opsize > ctx->end) + opsize =next (ctx,1 ); +if (!opsize || *opsize <2 )return 1 ; -switch (opcode) { +switch (*opcode) {case TCPOPT_WINDOW : -if (opsize ==TCPOLEN_WINDOW && ctx->ptr +TCPOLEN_WINDOW <= ctx->data_end) - ctx->wscale = ctx->ptr[2 ] <TCP_MAX_WSCALE ? ctx->ptr[2 ] :TCP_MAX_WSCALE ; + wscale =next (ctx,1 ); +if (!wscale) +return 1 ; +if (*opsize ==TCPOLEN_WINDOW ) + ctx->wscale = *wscale <TCP_MAX_WSCALE ? *wscale :TCP_MAX_WSCALE ;break ;case TCPOPT_TIMESTAMP : -if (opsize ==TCPOLEN_TIMESTAMP && ctx->ptr +TCPOLEN_TIMESTAMP <= ctx->data_end) { + tsecr =next (ctx,4 ); +if (!tsecr) +return 1 ; +if (*opsize ==TCPOLEN_TIMESTAMP ) { ctx->option_timestamp = true;/* Client's tsval becomes our tsecr. */ - *ctx->tsecr =get_unaligned ((__be32 *)(ctx->ptr +2 )); + *ctx->tsecr =get_unaligned ((__be32 *)tsecr); }break ;case TCPOPT_SACK_PERM : -if (opsize ==TCPOLEN_SACK_PERM ) +if (*opsize ==TCPOLEN_SACK_PERM ) ctx->option_sack = true;break ; } - ctx->ptr += opsize; + ctx->off = off + *opsize;return 0 ; }
@@ -256,16 +271,21 @@ static int tscookie_tcpopt_parse_batch(__u32 index, void *context) static __always_inline bool tscookie_init(struct tcphdr *tcp_header,__u16 tcp_len,__be32 *tsval, -__be32 *tsecr,void *data_end) +__be32 *tsecr,void *data,void *data_end) {struct tcpopt_context loop_ctx = { - .ptr = (__u8 *)(tcp_header +1 ), - .end = (__u8 *)tcp_header + tcp_len, + .data = data, .data_end = data_end, .tsecr = tsecr, .wscale =TS_OPT_WSCALE_MASK , .option_timestamp = false, .option_sack = false, +/* Note: currently verifier would track .off as unbound scalar. +* In case if verifier would at some point get smarter and +* compute bounded value for this var, beware that it might +* hinder bpf_loop() convergence validation. +*/ + .off = (__u8 *)(tcp_header +1 ) - (__u8 *)data, };u32 cookie;
@@ -635,7 +655,7 @@ static __always_inline int syncookie_handle_syn(struct header_pointers *hdr, cookie = (__u32 )value;if (tscookie_init ((void *)hdr->tcp, hdr->tcp_len, - &tsopt_buf[0 ], &tsopt_buf[1 ], data_end)) + &tsopt_buf[0 ], &tsopt_buf[1 ], data, data_end)) tsopt = tsopt_buf;/* Check that there is enough space for a SYNACK. It also covers
Signature · 1 line
2.43.0
PATCH 9/17 selftests/bpf: track string payload offset as scalar in strobemeta
In reply to Eduard Zingerman[ Upstream commit 87eb0152bcc1 ]
This change prepares strobemeta for update in callbacks verification logic. To allow bpf_loop() verification converge when multiple callback iterations are considered:
- track offset inside strobemeta_payload->payload directly as scalar value;
- at each iteration make sure that remaining strobemeta_payload->payload capacity is sufficient for execution of read_{map,str}_var functions;
- make sure that offset is tracked as unbound scalar between iterations, otherwise verifier won't be able infer that bpf_loop callback reaches identical states.
--- .../testing/selftests/bpf/progs/strobemeta.h | 78 ++++++++++++------- 1 file changed, 48 insertions(+), 30 deletions(-)
tools/testing/selftests/bpf/progs/strobemeta.h +48 -30
@@ -24,9 +24,11 @@ struct task_struct {};#define STACK_TABLE_EPOCH_SHIFT 20#define STROBE_MAX_STR_LEN 1#define STROBE_MAX_CFGS 32 +#define READ_MAP_VAR_PAYLOAD_CAP \ + ((1 + STROBE_MAX_MAP_ENTRIES * 2) * STROBE_MAX_STR_LEN)#define STROBE_MAX_PAYLOAD \ (STROBE_MAX_STRS * STROBE_MAX_STR_LEN + \ - STROBE_MAX_MAPS * (1 + STROBE_MAX_MAP_ENTRIES * 2) * STROBE_MAX_STR_LEN) + STROBE_MAX_MAPS * READ_MAP_VAR_PAYLOAD_CAP)struct strobe_value_header {/*
@@ -355,7 +357,7 @@ static __always_inline uint64_t read_str_var(struct strobemeta_cfg *cfg, size_t idx, void *tls_base, struct strobe_value_generic *value, struct strobemeta_payload *data, - void *payload) + size_t off) { void *location;uint64_t len;
@@ -366,7 +368,7 @@ static __always_inline uint64_t read_str_var(struct strobemeta_cfg *cfg,return 0 ;bpf_probe_read_user (value,sizeof (struct strobe_value_generic ), location); - len =bpf_probe_read_user_str (payload,STROBE_MAX_STR_LEN , value->ptr); + len =bpf_probe_read_user_str (&data->payload[off],STROBE_MAX_STR_LEN , value->ptr);/* * if bpf_probe_read_user_str returns error (<0), due to casting to * unsinged int, it will become big number, so next check is
@@ -378,14 +380,14 @@ static __always_inline uint64_t read_str_var(struct strobemeta_cfg *cfg,return 0 ; data->str_lens[idx] = len; -return len; +return off + len; } -static __always_inline void *read_map_var(struct strobemeta_cfg *cfg, -size_t idx,void *tls_base, -struct strobe_value_generic *value, -struct strobemeta_payload *data, -void *payload) +static __always_inline uint64_t read_map_var(struct strobemeta_cfg *cfg, +size_t idx,void *tls_base, +struct strobe_value_generic *value, +struct strobemeta_payload *data, +size_t off) {struct strobe_map_descr * descr = &data->map_descrs[idx];struct strobe_map_raw map;
@@ -397,11 +399,11 @@ static __always_inline void *read_map_var(struct strobemeta_cfg *cfg, location =calc_location (&cfg->map_locs[idx], tls_base);if (!location) -return payload; +return off;bpf_probe_read_user (value,sizeof (struct strobe_value_generic ), location);if (bpf_probe_read_user (&map,sizeof (struct strobe_map_raw ), value->ptr)) -return payload; +return off; descr->id = map.id; descr->cnt = map.cnt;
@@ -410,10 +412,10 @@ static __always_inline void *read_map_var(struct strobemeta_cfg *cfg, data->req_meta_valid =1 ; } - len =bpf_probe_read_user_str (payload,STROBE_MAX_STR_LEN , map.tag); + len =bpf_probe_read_user_str (&data->payload[off],STROBE_MAX_STR_LEN , map.tag);if (len <=STROBE_MAX_STR_LEN ) { descr->tag_len = len; - payload += len; + off += len; }#ifdef NO_UNROLL
@@ -426,22 +428,22 @@ static __always_inline void *read_map_var(struct strobemeta_cfg *cfg,break ; descr->key_lens[i] =0 ; - len =bpf_probe_read_user_str (payload,STROBE_MAX_STR_LEN , + len =bpf_probe_read_user_str (&data->payload[off],STROBE_MAX_STR_LEN , map.entries[i].key);if (len <=STROBE_MAX_STR_LEN ) { descr->key_lens[i] = len; - payload += len; + off += len; } descr->val_lens[i] =0 ; - len =bpf_probe_read_user_str (payload,STROBE_MAX_STR_LEN , + len =bpf_probe_read_user_str (&data->payload[off],STROBE_MAX_STR_LEN , map.entries[i].val);if (len <=STROBE_MAX_STR_LEN ) { descr->val_lens[i] = len; - payload += len; + off += len; } } -return payload; +return off; }#ifdef USE_BPF_LOOP
@@ -455,14 +457,20 @@ struct read_var_ctx {struct strobemeta_payload *data;void *tls_base;struct strobemeta_cfg *cfg; -void *payload; +size_t payload_off;/* value gets mutated */ struct strobe_value_generic *value;enum read_type type; }; -static int read_var_callback (__u32 index,struct read_var_ctx *ctx) +static int read_var_callback (__u64 index,struct read_var_ctx *ctx) { +/* lose precision info for ctx->payload_off, verifier won't track +* double xor, barrier_var() is needed to force clang keep both xors. +*/ + ctx->payload_off ^= index; +barrier_var (ctx->payload_off); + ctx->payload_off ^= index;switch (ctx->type) {case READ_INT_VAR :if (index >=STROBE_MAX_INTS )
@@ -472,14 +480,18 @@ static int read_var_callback(__u32 index, struct read_var_ctx *ctx)case READ_MAP_VAR :if (index >=STROBE_MAX_MAPS )return 1 ; - ctx->payload =read_map_var (ctx->cfg, index, ctx->tls_base, - ctx->value, ctx->data, ctx->payload); +if (ctx->payload_off >sizeof (ctx->data->payload) -READ_MAP_VAR_PAYLOAD_CAP ) +return 1 ; + ctx->payload_off =read_map_var (ctx->cfg, index, ctx->tls_base, + ctx->value, ctx->data, ctx->payload_off);break ;case READ_STR_VAR :if (index >=STROBE_MAX_STRS )return 1 ; - ctx->payload +=read_str_var (ctx->cfg, index, ctx->tls_base, - ctx->value, ctx->data, ctx->payload); +if (ctx->payload_off >sizeof (ctx->data->payload) -STROBE_MAX_STR_LEN ) +return 1 ; + ctx->payload_off =read_str_var (ctx->cfg, index, ctx->tls_base, + ctx->value, ctx->data, ctx->payload_off);break ; }return 0 ;
@@ -501,7 +513,8 @@ static void *read_strobe_meta(struct task_struct *task,pid_t pid =bpf_get_current_pid_tgid () >>32 ;struct strobe_value_generic value = {0 };struct strobemeta_cfg *cfg; -void *tls_base, *payload; +size_t payload_off; +void *tls_base; cfg =bpf_map_lookup_elem (&strobemeta_cfgs, &pid);if (!cfg)
@@ -509,7 +522,7 @@ static void *read_strobe_meta(struct task_struct *task, data->int_vals_set_mask =0 ; data->req_meta_valid =0 ; - payload = data->payload; + payload_off =0 ;/* * we don't have struct task_struct definition, it should be: * tls_base = (void *)task->thread.fsbase;
@@ -522,7 +535,7 @@ static void *read_strobe_meta(struct task_struct *task, .tls_base = tls_base, .value = &value, .data = data, - .payload = payload, + .payload_off = 0, };int err;
@@ -540,6 +553,11 @@ static void *read_strobe_meta(struct task_struct *task, err =bpf_loop (STROBE_MAX_MAPS , read_var_callback, &ctx,0 );if (err !=STROBE_MAX_MAPS )return NULL ; + + payload_off = ctx.payload_off; +/* this should not really happen, here only to satisfy verifer */ +if (payload_off >sizeof (data->payload)) + payload_off =sizeof (data->payload);#else #ifdef NO_UNROLL #pragma clang loop unroll(disable)
@@ -555,7 +573,7 @@ static void *read_strobe_meta(struct task_struct *task,#pragma unroll#endif /* NO_UNROLL */ for (int i =0 ; i <STROBE_MAX_STRS ; ++i) { - payload +=read_str_var (cfg, i, tls_base, &value, data, payload); + payload_off =read_str_var (cfg, i, tls_base, &value, data, payload_off); }#ifdef NO_UNROLL #pragma clang loop unroll(disable)
@@ -563,7 +581,7 @@ static void *read_strobe_meta(struct task_struct *task,#pragma unroll#endif /* NO_UNROLL */ for (int i =0 ; i <STROBE_MAX_MAPS ; ++i) { - payload =read_map_var (cfg, i, tls_base, &value, data, payload); + payload_off =read_map_var (cfg, i, tls_base, &value, data, payload_off); }#endif /* USE_BPF_LOOP */
@@ -571,7 +589,7 @@ static void *read_strobe_meta(struct task_struct *task,* return pointer right after end of payload, so it's possible to * calculate exact amount of useful data that needs to be sent */ -return payload; +return &data->payload[payload_off]; }SEC ("raw_tracepoint/kfree_skb" )
Signature · 1 line
2.43.0
PATCH 10/17 bpf: extract __check_reg_arg() utility function
In reply to Eduard Zingerman[ Upstream commit 683b96f9606a ]
Split check_reg_arg() into two utility functions:
- check_reg_arg() operating on registers from current verifier state;
- __check_reg_arg() operating on a specific set of registers passed as a parameter;
The __check_reg_arg() function would be used by a follow-up change for callbacks handling.
--- kernel/bpf/verifier.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +13 -6
@@ -3313,13 +3313,11 @@ static void mark_insn_zext(struct bpf_verifier_env *env, reg->subreg_def =DEF_NOT_SUBREG ; } -static int check_reg_arg (struct bpf_verifier_env *env,u32 regno, -enum reg_arg_type t) +static int __check_reg_arg (struct bpf_verifier_env *env,struct bpf_reg_state *regs,u32 regno, +enum reg_arg_type t) { -struct bpf_verifier_state *vstate = env->cur_state; -struct bpf_func_state *state = vstate->frame[vstate->curframe];struct bpf_insn *insn = env->prog->insnsi + env->insn_idx; -struct bpf_reg_state *reg, *regs = state->regs; +struct bpf_reg_state *reg;bool rw64;if (regno >=MAX_BPF_REG ) {
@@ -3360,6 +3358,15 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno,return 0 ; } +static int check_reg_arg (struct bpf_verifier_env *env,u32 regno, +enum reg_arg_type t) +{ +struct bpf_verifier_state *vstate = env->cur_state; +struct bpf_func_state *state = vstate->frame[vstate->curframe]; + +return __check_reg_arg (env, state->regs, regno, t); +} +static void mark_jmp_point (struct bpf_verifier_env *env,int idx) { env->insn_aux_data[idx].jmp_point = true;
@@ -9166,7 +9173,7 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env,/* after the call registers r0 - r5 were scratched */ for (i =0 ; i <CALLER_SAVED_REGS ; i++) {mark_reg_not_init (env, regs, caller_saved[i]); -check_reg_arg (env, caller_saved[i],DST_OP_NO_MARK ); +__check_reg_arg (env, regs, caller_saved[i],DST_OP_NO_MARK ); } }
Signature · 1 line
2.43.0
PATCH 11/17 bpf: extract setup_func_entry() utility function
In reply to Eduard Zingerman[ Upstream commit 58124a98cb8e ]
Move code for simulated stack frame creation to a separate utility function. This function would be used in the follow-up change for callbacks handling.
--- kernel/bpf/verifier.c | 84 ++++++++++++++++++++++++------------------- 1 file changed, 48 insertions(+), 36 deletions(-)
kernel/bpf/verifier.c +48 -36
@@ -9186,11 +9186,10 @@ static int set_callee_state(struct bpf_verifier_env *env,struct bpf_func_state *caller, struct bpf_func_state *callee, int insn_idx); -static int __check_func_call (struct bpf_verifier_env *env,struct bpf_insn *insn, -int *insn_idx,int subprog, -set_callee_state_fn set_callee_state_cb) +static int setup_func_entry (struct bpf_verifier_env *env,int subprog,int callsite, +set_callee_state_fn set_callee_state_cb, +struct bpf_verifier_state *state) { -struct bpf_verifier_state *state = env->cur_state;struct bpf_func_state *caller, *callee;int err;
@@ -9200,13 +9199,53 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insnreturn -E2BIG ; } - caller = state->frame[state->curframe];if (state->frame[state->curframe +1 ]) {verbose (env,"verifier bug. Frame %d already allocated\n" , state->curframe +1 );return -EFAULT ; } + caller = state->frame[state->curframe]; + callee =kzalloc (sizeof (*callee),GFP_KERNEL ); +if (!callee) +return -ENOMEM ; + state->frame[state->curframe +1 ] = callee; + +/* callee cannot access r0, r6 - r9 for reading and has to write +* into its own stack before reading from it. +* callee can read/write into caller's stack +*/ +init_func_state (env, callee, +/* remember the callsite, it will be used by bpf_exit */ + callsite, + state->curframe +1 /* frameno within this callchain */ , + subprog/* subprog number within this prog */ ); +/* Transfer references to the callee */ + err =copy_reference_state (callee, caller); + err = err ?:set_callee_state_cb (env, caller, callee, callsite); +if (err) + goto err_out; + +/* only increment it after check_reg_arg() finished */ + state->curframe++; + +return 0 ; + +err_out: +free_func_state (callee); + state->frame[state->curframe +1 ] =NULL ; +return err; +} + +static int __check_func_call (struct bpf_verifier_env *env,struct bpf_insn *insn, +int *insn_idx,int subprog, +set_callee_state_fn set_callee_state_cb) +{ +struct bpf_verifier_state *state = env->cur_state; +struct bpf_func_state *caller, *callee; +int err; + + caller = state->frame[state->curframe]; err =btf_check_subprog_call (env, subprog, caller->regs);if (err == -EFAULT )return err;
@@ -9275,35 +9314,12 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insnreturn 0 ; } - callee =kzalloc (sizeof (*callee),GFP_KERNEL ); -if (!callee) -return -ENOMEM ; - state->frame[state->curframe +1 ] = callee; - -/* callee cannot access r0, r6 - r9 for reading and has to write -* into its own stack before reading from it. -* callee can read/write into caller's stack -*/ -init_func_state (env, callee, -/* remember the callsite, it will be used by bpf_exit */ - *insn_idx/* callsite */ , - state->curframe +1 /* frameno within this callchain */ , - subprog/* subprog number within this prog */ ); - -/* Transfer references to the callee */ - err =copy_reference_state (callee, caller); + err =setup_func_entry (env, subprog, *insn_idx, set_callee_state_cb, state);if (err) - goto err_out; - - err =set_callee_state_cb (env, caller, callee, *insn_idx); -if (err) - goto err_out; +return err;clear_caller_saved_regs (env, caller->regs); -/* only increment it after check_reg_arg() finished */ - state->curframe++; -/* and go analyze first insn of the callee */ *insn_idx = env->subprog_info[subprog].start -1 ;
@@ -9311,14 +9327,10 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insnverbose (env,"caller:\n" );print_verifier_state (env, caller, true);verbose (env,"callee:\n" ); -print_verifier_state (env, callee, true); +print_verifier_state (env, state->frame[state->curframe], true); } -return 0 ; -err_out: -free_func_state (callee); - state->frame[state->curframe +1 ] =NULL ; -return err; +return 0 ; } int map_set_for_each_callback_args(struct bpf_verifier_env *env,
Signature · 1 line
2.43.0
PATCH 12/17 bpf: verify callbacks as if they are called unknown number of times
In reply to Eduard Zingerman[ Upstream commit ab5cfac139ab ]
Prior to this patch callbacks were handled as regular function calls, execution of callback body was modeled exactly once.
This patch updates callbacks handling logic as follows:
- introduces a function push_callback_call() that schedules callback body verification in env->head stack;
- updates prepare_func_exit() to reschedule callback body verification upon BPF_EXIT;
- as calls to bpf_*_iter_next(), calls to callback invoking functions are marked as checkpoints;
- is_state_visited() is updated to stop callback based iteration when some identical parent state is found.
Paths with callback function invoked zero times are now verified first, which leads to necessity to modify some selftests:
- the following negative tests required adding release/unlock/drop calls to avoid previously masked unrelated error reports:
- cb_refs.c:underflow_prog
- exceptions_fail.c:reject_rbtree_add_throw
- exceptions_fail.c:reject_with_cp_reference
- the following precision tracking selftests needed change in expected log trace:
- verifier_subprog_precision.c:callback_result_precise
(note: r0 precision is no longer propagated inside callback and
I think this is a correct behavior)
- verifier_subprog_precision.c:parent_callee_saved_reg_precise_with_callback
- verifier_subprog_precision.c:parent_stack_slot_precise_with_callback
[ Summary of the conflicts and their resolutions ]
There were three conflicts with this patch, all with code related to implementation of BPF exceptions: the upstream series was developed after BPF exceptions landed in bpf-next and BPF exceptions are not selected for porting to 6.6.y at the moment.
All conflicts were resolved by excluding exceptions related parts.
--- include/linux/bpf_verifier.h | 5 + kernel/bpf/verifier.c | 272 +++++++++++------- tools/testing/selftests/bpf/progs/cb_refs.c | 1 + .../bpf/progs/verifier_subprog_precision.c | 71 ++++- 4 files changed, 237 insertions(+), 112 deletions(-)
include/linux/bpf_verifier.h +5 -0
@@ -399,6 +399,7 @@ struct bpf_verifier_state {struct bpf_idx_pair *jmp_history;u32 jmp_history_cnt;u32 dfs_depth; +u32 callback_unroll_depth; };#define bpf_get_spilled_reg (slot, frame) \
@@ -506,6 +507,10 @@ struct bpf_insn_aux_data {* this instruction, regardless of any heuristics */ bool force_checkpoint; +/* true if instruction is a call to a helper function that +* accepts callback function as a parameter. +*/ +bool calls_callback; };#define MAX_USED_MAPS 64/* max number of maps accessed by one eBPF program */
kernel/bpf/verifier.c +174 -98
437 lines not on this page · raw message
tools/testing/selftests/bpf/progs/cb_refs.c +1 -0
@@ -33,6 +33,7 @@ int underflow_prog(void *ctx)if (!p)return 0 ;bpf_for_each_map_elem (&array_map, cb1, &p,0 ); +bpf_kfunc_call_test_release (p);return 0 ; }
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +57 -14
114 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH 13/17 selftests/bpf: tests for iterating callbacks
In reply to Eduard Zingerman[ Upstream commit 958465e217db ]
A set of test cases to check behavior of callback handling logic, check if verifier catches the following situations:
- program not safe on second callback iteration;
- program not safe on zero callback iterations;
- infinite loop inside a callback.
Verify that callback logic works for bpf_loop, bpf_for_each_map_elem, bpf_user_ringbuf_drain, bpf_find_vma.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_iterating_callbacks.c | 147 ++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -31,6 +31,7 @@#include "verifier_helper_restricted.skel.h" #include "verifier_helper_value_access.skel.h" #include "verifier_int_ptr.skel.h" +#include "verifier_iterating_callbacks.skel.h" #include "verifier_jeq_infer_not_null.skel.h" #include "verifier_ld_ind.skel.h" #include "verifier_ldsx.skel.h"
@@ -138,6 +139,7 @@ void test_verifier_helper_packet_access(void) { RUN(verifier_helper_packet_accesvoid test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }void test_verifier_helper_value_access (void ) {RUN (verifier_helper_value_access); }void test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); } +void test_verifier_iterating_callbacks (void ) {RUN (verifier_iterating_callbacks); }void test_verifier_jeq_infer_not_null (void ) {RUN (verifier_jeq_infer_not_null); }void test_verifier_ld_ind (void ) {RUN (verifier_ld_ind); }void test_verifier_ldsx (void ) {RUN (verifier_ldsx); }
tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c new file +147 -0
147 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH 14/17 bpf: widening for callback iterators
In reply to Eduard Zingerman[ Upstream commit cafe2c21508a ]
Callbacks are similar to open coded iterators, so add imprecise widening logic for callback body processing. This makes callback based loops behave identically to open coded iterators, e.g. allowing to verify programs like below:
struct ctx { u32 i; };
int cb(u32 idx, struct ctx* ctx)
{
++ctx->i;
return 0;
}
...
struct ctx ctx = { .i = 0 };
bpf_loop(100, cb, &ctx, 0);
...--- kernel/bpf/verifier.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
kernel/bpf/verifier.c +22 -2
@@ -9614,9 +9614,10 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env)static int prepare_func_exit (struct bpf_verifier_env *env,int *insn_idx) { -struct bpf_verifier_state *state = env->cur_state; +struct bpf_verifier_state *state = env->cur_state, *prev_st;struct bpf_func_state *caller, *callee;struct bpf_reg_state *r0; +bool in_callback_fn;int err; callee = state->frame[state->curframe];
@@ -9671,7 +9672,8 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)* there function call logic would reschedule callback visit. If iteration * converges is_state_visited() would prune that visit eventually. */ -if (callee->in_callback_fn) + in_callback_fn = callee->in_callback_fn; +if (in_callback_fn) *insn_idx = callee->callsite;else *insn_idx = callee->callsite +1 ;
@@ -9685,6 +9687,24 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)/* clear everything in the callee */ free_func_state (callee); state->frame[state->curframe--] =NULL ; + +/* for callbacks widen imprecise scalars to make programs like below verify: +* +* struct ctx { int i; } +* void cb(int idx, struct ctx *ctx) { ctx->i++; ... } +* ... +* struct ctx = { .i = 0; } +* bpf_loop(100, cb, &ctx, 0); +* +* This is similar to what is done in process_iter_next_call() for open +* coded iterators. +*/ + prev_st = in_callback_fn ?find_prev_entry (env, state, *insn_idx) :NULL ; +if (prev_st) { + err =widen_imprecise_scalars (env, prev_st, state); +if (err) +return err; + }return 0 ; }
Signature · 1 line
2.43.0
PATCH 15/17 selftests/bpf: test widening for iterating callbacks
In reply to Eduard Zingerman[ Upstream commit 9f3330aa644d ]
A test case to verify that imprecise scalars widening is applied to callback entering state, when callback call is simulated repeatedly.
--- .../bpf/progs/verifier_iterating_callbacks.c | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+)
tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c +20 -0
@@ -25,6 +25,7 @@ struct buf_context {struct num_context {__u64 i; +__u64 j; };__u8 choice_arr[2 ] = {0 ,1 };
@@ -69,6 +70,25 @@ int unsafe_on_zero_iter(void *unused)return choice_arr[loop_ctx.i]; } +static int widening_cb (__u32 idx,struct num_context *ctx) +{ + ++ctx->i; +return 0 ; +} + +SEC("?raw_tp") +__success +int widening (void *unused) +{ +struct num_context loop_ctx = { .i =0 , .j =1 }; + +bpf_loop (100 , widening_cb, &loop_ctx,0 ); +/* loop_ctx.j is not changed during callback iteration, +* verifier should not apply widening to it. +*/ +return choice_arr[loop_ctx.j]; +} +static int loop_detection_cb (__u32 idx,struct num_context *ctx) {for (;;) {}
Signature · 1 line
2.43.0
PATCH 16/17 bpf: keep track of max number of bpf_loop callback iterations
In reply to Eduard Zingerman[ Upstream commit bb124da69c47 ]
In some cases verifier can't infer convergence of the bpf_loop() iteration. E.g. for the following program:
static int cb(__u32 idx, struct num_context* ctx)
{
ctx->i++;
return 0;
}
SEC("?raw_tp")
int prog(void *_)
{
struct num_context ctx = { .i = 0 };
__u8 choice_arr[2] = { 0, 1 };
bpf_loop(2, cb, &ctx, 0);
return choice_arr[ctx.i];
}Each 'cb' simulation would eventually return to 'prog' and reach 'return choice_arr[ctx.i]' statement. At which point ctx.i would be marked precise, thus forcing verifier to track multitude of separate states with {.i=0}, {.i=1}, ... at bpf_loop() callback entry.
This commit allows "brute force" handling for such cases by limiting number of callback body simulations using 'umax' value of the first bpf_loop() parameter.
For this, extend bpf_func_state with 'callback_depth' field. Increment this field when callback visiting state is pushed to states traversal stack. For frame #N it's 'callback_depth' field counts how many times callback with frame depth N+1 had been executed. Use bpf_func_state specifically to allow independent tracking of callback depths when multiple nested bpf_loop() calls are present.
[ Summary of the conflicts and their resolutions ]
There was one conflict with this patch, related to implementation of BPF exceptions: the upstream series was developed after BPF exceptions landed in bpf-next and BPF exceptions are not selected for porting to 6.6.y at the moment.
Conflict was resolved by excluding exceptions related part.
--- include/linux/bpf_verifier.h | 11 ++++++ kernel/bpf/verifier.c | 19 ++++++++-- .../bpf/progs/verifier_subprog_precision.c | 35 +++++++++++++------ 3 files changed, 53 insertions(+), 12 deletions(-)
include/linux/bpf_verifier.h +11 -0
@@ -300,6 +300,17 @@ struct bpf_func_state {bool in_callback_fn;struct tnum callback_ret_range;bool in_async_callback_fn; +/* For callback calling functions that limit number of possible +* callback executions (e.g. bpf_loop) keeps track of current +* simulated iteration number. +* Value in frame N refers to number of times callback with frame +* N+1 was simulated, e.g. for the following call: +* +* bpf_loop(..., fn, ...); | suppose current frame is N +* | fn would be simulated in frame N+1 +* | number of simulations is tracked in frame N +*/ +u32 callback_depth;/* The following fields should be last. See copy_func_state() */ int acquired_refs;
kernel/bpf/verifier.c +17 -2
@@ -9320,6 +9320,8 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *insreturn err; callback_state->callback_unroll_depth++; + callback_state->frame[callback_state->curframe -1 ]->callback_depth++; + caller->callback_depth =0 ;return 0 ; }
@@ -10102,8 +10104,21 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insnbreak ;case BPF_FUNC_loop:update_loop_inline_state (env, meta.subprogno); - err =push_callback_call (env, insn, insn_idx, meta.subprogno, - set_loop_callback_state); +/* Verifier relies on R1 value to determine if bpf_loop() iteration +* is finished, thus mark it precise. +*/ + err =mark_chain_precision (env,BPF_REG_1 ); +if (err) +return err; +if (cur_func (env)->callback_depth < regs[BPF_REG_1 ].umax_value) { + err =push_callback_call (env, insn, insn_idx, meta.subprogno, + set_loop_callback_state); + }else { +cur_func (env)->callback_depth =0 ; +if (env->log.level &BPF_LOG_LEVEL2 ) +verbose (env,"frame%d bpf_loop iteration limit reached\n" , + env->cur_state->curframe); + }break ;case BPF_FUNC_dynptr_from_mem:if (regs[BPF_REG_1 ].type !=PTR_TO_MAP_VALUE ) {
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +25 -10
@@ -119,7 +119,23 @@ __naked int global_subprog_result_precise(void) SEC("?raw_tp") __success __log_level(2) -/* First simulated path does not include callback body */ +/* First simulated path does not include callback body, +* r1 and r4 are always precise for bpf_loop() calls. +*/ +__msg("9: (85) call bpf_loop#181") +__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") +__msg("mark_precise: frame0: parent state regs=r4 stack=:") +__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9") +__msg("mark_precise: frame0: regs=r4 stack= before 8: (b7) r4 = 0") +__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") +__msg("mark_precise: frame0: parent state regs=r1 stack=:") +__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9") +__msg("mark_precise: frame0: regs=r1 stack= before 8: (b7) r4 = 0") +__msg("mark_precise: frame0: regs=r1 stack= before 7: (b7) r3 = 0") +__msg("mark_precise: frame0: regs=r1 stack= before 6: (bf) r2 = r8") +__msg("mark_precise: frame0: regs=r1 stack= before 5: (bf) r1 = r6") +__msg ("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3" ) +/* r6 precision propagation */ __msg("14: (0f) r1 += r6") __msg("mark_precise: frame0: last_idx 14 first_idx 9")__msg ("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7" )
@@ -134,10 +150,9 @@ __msg("17: (b7) r0 = 0") __msg("18: (95) exit") __msg("returning from callee:") __msg("to caller at 9:") -/* r4 (flags) is always precise for bpf_loop() */ -__msg("frame 0: propagating r4") +__msg("frame 0: propagating r1,r4") __msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") -__msg("mark_precise: frame0: regs=r4 stack= before 18: (95) exit") +__msg("mark_precise: frame0: regs=r1,r4 stack= before 18: (95) exit") __msg("from 18 to 9: safe")__naked int callback_result_precise (void ) {
@@ -264,12 +279,12 @@ __msg("15: (b7) r0 = 0") __msg("16: (95) exit") __msg("returning from callee:")__msg ("to caller at 9:" ) -/* r4 (flags) is always precise for bpf_loop(), +/* r1, r4 are always precise for bpf_loop(), * r6 was marked before backtracking to callback body. */ -__msg("frame 0: propagating r4,r6") +__msg("frame 0: propagating r1,r4,r6") __msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") -__msg("mark_precise: frame0: regs=r4,r6 stack= before 16: (95) exit") +__msg("mark_precise: frame0: regs=r1,r4,r6 stack= before 16: (95) exit") __msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0") __msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop")__msg ("mark_precise: frame0: parent state regs= stack=:" )
@@ -422,12 +437,12 @@ __msg("17: (b7) r0 = 0") __msg("18: (95) exit") __msg("returning from callee:")__msg ("to caller at 10:" ) -/* r4 (flags) is always precise for bpf_loop(), +/* r1, r4 are always precise for bpf_loop(), * fp-8 was marked before backtracking to callback body. */ -__msg("frame 0: propagating r4,fp-8") +__msg("frame 0: propagating r1,r4,fp-8") __msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1") -__msg("mark_precise: frame0: regs=r4 stack=-8 before 18: (95) exit") +__msg("mark_precise: frame0: regs=r1,r4 stack=-8 before 18: (95) exit") __msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0") __msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181")__msg ("mark_precise: frame0: parent state regs= stack=:" )
Signature · 1 line
2.43.0
PATCH 17/17 selftests/bpf: check if max number of bpf_loop iterations is tracked
In reply to Eduard Zingerman[ Upstream commit 57e2a52deeb1 ]
Check that even if bpf_loop() callback simulation does not converge to a specific state, verification could proceed via "brute force" simulation of maximal number of callback calls.
--- .../bpf/progs/verifier_iterating_callbacks.c | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+)
tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c +75 -0
@@ -164,4 +164,79 @@ int unsafe_find_vma(void *unused)return choice_arr[loop_ctx.i]; } +static int iter_limit_cb (__u32 idx,struct num_context *ctx) +{ + ctx->i++; +return 0 ; +} + +SEC("?raw_tp") +__success +int bpf_loop_iter_limit_ok (void *unused) +{ +struct num_context ctx = { .i =0 }; + +bpf_loop (1 , iter_limit_cb, &ctx,0 ); +return choice_arr[ctx.i]; +} + +SEC("?raw_tp") +__failure __msg("invalid access to map value, value_size=2 off=2 size=1") +int bpf_loop_iter_limit_overflow(void *unused) +{ +struct num_context ctx = { .i =0 }; + +bpf_loop (2 , iter_limit_cb, &ctx,0 ); +return choice_arr[ctx.i]; +} + +static int iter_limit_level2a_cb (__u32 idx,struct num_context *ctx) +{ + ctx->i +=100 ; +return 0 ; +} + +static int iter_limit_level2b_cb (__u32 idx,struct num_context *ctx) +{ + ctx->i +=10 ; +return 0 ; +} + +static int iter_limit_level1_cb (__u32 idx,struct num_context *ctx) +{ + ctx->i +=1 ; +bpf_loop (1 , iter_limit_level2a_cb, ctx,0 ); +bpf_loop (1 , iter_limit_level2b_cb, ctx,0 ); +return 0 ; +} + +/* Check that path visiting every callback function once had been +* reached by verifier. Variables 'ctx{1,2}i' below serve as flags, +* with each decimal digit corresponding to a callback visit marker. +*/ +SEC("socket") +__success __retval(111111) +int bpf_loop_iter_limit_nested(void *unused) +{ +struct num_context ctx1 = { .i =0 }; +struct num_context ctx2 = { .i =0 }; +__u64 a, b, c; + +bpf_loop (1 , iter_limit_level1_cb, &ctx1,0 ); +bpf_loop (1 , iter_limit_level1_cb, &ctx2,0 ); + a = ctx1.i; + b = ctx2.i; +/* Force 'ctx1.i' and 'ctx2.i' precise. */ + c = choice_arr[(a + b) %2 ]; +/* This makes 'c' zero, but neither clang nor verifier know it. */ + c /=10 ; +/* Make sure that verifier does not visit 'impossible' states: +* enumerate all possible callback visit masks. +*/ +if (a !=0 && a !=1 && a !=11 && a !=101 && a !=111 && + b !=0 && b !=1 && b !=11 && b !=101 && b !=111 ) + asmvolatile ("r0 /= 0;" :::"r0" ); +return 1000 * a + b + c; +} + char _license[] SEC("license") = "GPL";
Signature · 1 line
2.43.0
Greg KH
maintainerQuoted 42 lines · Eduard Zingerman
This is a backport of two upstream patch-sets:
1. "exact states comparison for iterator convergence checks"
https://lore.kernel.org/all/[email protected]/
2. "verify callbacks as if they are called unknown number of times"
https://lore.kernel.org/all/[email protected]/Both patch-sets fix BPF verifier logic related to handling loops:
for bpf iterators, and for helper functions that accept callback functions.The backport of (2) was requested as a response to bug report by Mateusz Gienieczko <[email protected]>.
The (1) is a dependency of (2).The patch-set was tested by running BPF verifier selftests on my local qemu-based setup.
Most of the commits could be cherry-picked but three required merging:
| Action | Upstream commit | |--------+-------------------------------------------------------------------------------------------------| | pick | 3c4e420cb653 ("bpf: move explored_state() closer to the beginning of verifier.c ") | | pick | 4c97259abc9b ("bpf: extract same_callsites() as utility function ") | | merge | 2793a8b015f7 ("bpf: exact states comparison for iterator convergence checks ") | | pick | 389ede06c297 ("selftests/bpf: tests with delayed read/precision makrs in loop body ") | | pick | 2a0992829ea3 ("bpf: correct loop detection for iterators convergence ") | | pick | 64870feebecb ("selftests/bpf: test if state loops are detected in a tricky case ") | | pick | b4d8239534fd ("bpf: print full verifier states on infinite loop detection ") | | drop | dedd6c894110 ("Merge branch 'exact-states-comparison-for-iterator-convergence-checks' ") | |--------+-------------------------------------------------------------------------------------------------| | pick | 977bc146d4eb ("selftests/bpf: track tcp payload offset as scalar in xdp_synproxy ") | | pick | 87eb0152bcc1 ("selftests/bpf: track string payload offset as scalar in strobemeta ") | | pick | 683b96f9606a ("bpf: extract __check_reg_arg() utility function ") | | pick | 58124a98cb8e ("bpf: extract setup_func_entry() utility function ") | | merge | ab5cfac139ab ("bpf: verify callbacks as if they are called unknown number of times ") | | pick | 958465e217db ("selftests/bpf: tests for iterating callbacks ") | | pick | cafe2c21508a ("bpf: widening for callback iterators ") | | pick | 9f3330aa644d ("selftests/bpf: test widening for iterating callbacks ") | | merge | bb124da69c47 ("bpf: keep track of max number of bpf_loop callback iterations ") | | pick | 57e2a52deeb1 ("selftests/bpf: check if max number of bpf_loop iterations is tracked ") | | drop | acb12c859ac7 ("Merge branch 'verify-callbacks-as-if-they-are-called-unknown-number-of-times' ") |
Quoted 4 lines
Note:
I don't know how deal with merge commits, so I just dropped those. These commits are empty but contain cover letters for both series, so it might be useful to pick those (how?).
Nah, merge commits are not needed to be backported, that doesn't work :)
All now queued up, thanks!
greg k-h