bpf: replace path-sensitive with path-insensitive live stack analysis
Subsystems (6)
Eduard Zingerman
maintainercover bpf: replace path-sensitive with path-insensitive live stack analysis
Consider the following program, assuming checkpoint is created for a state at instruction (3):
1: call bpf_get_prandom_u32()
2: *(u64 *)(r10 - 8) = 42
-- checkpoint #1 --
3: if r0 != 0 goto +1
4: exit;
5: r0 = *(u64 *)(r10 - 8)
6: exit
The verifier processes this program by exploring two paths:
- 1 -> 2 -> 3 -> 4
- 1 -> 2 -> 3 -> 5 -> 6
When instruction (5) is processed, the current liveness tracking mechanism moves up the register parent links and records a "read" mark for stack slot -8 at checkpoint #1, stopping because of the "write" mark recorded at instruction (2).
This patch set replaces the existing liveness tracking mechanism with a path-insensitive data flow analysis. The program above is processed as follows:
- a data structure representing live stack slots for instructions 1-6 in frame #0 is allocated;
- when instruction (2) is processed, record that slot -8 is written at instruction (2) in frame #0;
- when instruction (5) is processed, record that slot -8 is read at instruction (5) in frame #0;
- when instruction (6) is processed, propagate read mark for slot -8 up the control flow graph to instructions 3 and 2.
The key difference is that the new mechanism operates on a control flow graph and associates read and write marks with pairs of (call chain, instruction index). In contrast, the old mechanism operates on verifier states and register parent links, associating read and write marks with verifier states.
Motivation
==========
As it stands, this patch set makes liveness tracking slightly less precise, as it no longer distinguishes individual program paths taken by the verifier during symbolic execution.
See the "Impact on verification performance" section for details.
However, this change is intended as a stepping stone toward the following goals:
- Short term, integrate precision tracking into liveness analysis and remove the following code:
- verifier backedge states accumulation in is_state_visited();
- most of the logic for precision tracking;
- jump history tracking.
- Long term, help with more efficient loop verification handling.
Why integrating precision tracking?
-----------------------------------
In a sense, precision tracking is very similar to liveness tracking. The data flow equations for liveness tracking look as follows:
live_after =
U [state[s].live_before for s in insn_successors(i)]
state[i].live_before =
(live_after / state[i].must_write) U state[i].may_read
While data flow equations for precision tracking look as follows:
precise_after =
U [state[s].precise_before for s in insn_successors(i)]
// if some of the instruction outputs are precise, // assume its inputs to be precise induced_precise =
⎧ state[i].may_read if (state[i].may_write ∩ precise_after) ≠ ∅
⎨
⎩ ∅ otherwise
state[i].precise_before =
(precise_after / state[i].must_write) ∩ induced_precise
Where:
- may_read set represents a union of all possibly read slots
(any slot in may_read set might be by the instruction);
- must_write set represents an intersection of all possibly written slots (any slot in must_write set is guaranteed to be written by the instruction).
- may_write set represents a union of all possibly written slots (any slot in may_write set might be written by the instruction).
This means that precision tracking can be implemented as a logical extension of liveness tracking:
- track registers as well as stack slots;
- add bit masks to represent precise_before and may_write;
- add above equations for precise_before computation;
- (linked registers require some additional consideration).
Such extension would allow removal of:
- precision propagation logic in verifier.c:
- backtrack_insn()
- mark_chain_precision()
- propagate_{precision,backedges}()
- push_jmp_history() and related data structures, which are only used by precision tracking;
- add_scc_backedge() and related backedge state accumulation in is_state_visited(), superseded by per-callchain function state accumulated by liveness analysis.
The hope here is that unifying liveness and precision tracking will reduce overall amount of code and make it easier to reason about.
How this helps with loops?
--------------------------
As it stands, this patch set shares the same deficiency as the current liveness tracking mechanism. Liveness marks on stack slots cannot be used to prune states when processing iterator-based loops:
- such states still have branches to be explored;
- meaning that not all stack slot reads have been discovered.
For example:
1: while(iter_next()) {
2: if (...)
3: r0 = *(u64 *)(r10 - 8)
4: if (...)
5: r0 = *(u64 *)(r10 - 16)
6: ...
7: }For any checkpoint state created at instruction (1), it is only possible to rely on read marks for slots fp[-8] and fp[-16] once all child states of (1) have been explored. Thus, when the verifier transitions from (7) to (1), it cannot rely on read marks.
However, sacrificing path-sensitivity makes it possible to run analysis defined in this patch set before main verification pass, if estimates for value ranges are available.
E.g. for the following program:
1: while(iter_next()) {
2: r0 = r10
3: r0 += r2
4: r0 = *(u64 *)(r2 + 0)
5: ...
6: }If an estimate for r2 range is available before the main verification pass, it can be used to populate read marks at instruction (4) and run the liveness analysis. Thus making conservative liveness information available during loops verification.
Such estimates can be provided by some form of value range analysis. Value range analysis is also necessary to address loop verification from another angle: computing boundaries for loop induction variables and iteration counts.
The hope here is that the new liveness tracking mechanism will support the broader goal of making loop verification more efficient.
Validation
==========
The change was tested on three program sets:
- bpf selftests
- sched_ext
- Meta's internal set of programs
Commit [#8] enables a special mode where both the current and new liveness analyses are enabled simultaneously. This mode signals an error if the new algorithm considers a stack slot dead while the current algorithm assumes it is alive. This mode was very useful for debugging. At the time of posting, no such errors have been reported for the above program sets.
[#8] "bpf: signal error if old liveness is more conservative than new"
Impact on memory consumption
============================
Debug patch [1] extends the kernel and veristat to count the amount of memory allocated for storing analysis data. This patch is not included in the submission. The maximal observed impact for the above program sets is 2.6Mb.
Data below is shown in bytes.
For bpf selftests top 5 consumers look as follows:
File Program liveness mem ----------------------- ---------------- ------------ pyperf180.bpf.o on_event 2629740 pyperf600.bpf.o on_event 2287662 pyperf100.bpf.o on_event 1427022 test_verif_scale3.bpf.o balancer_ingress 1121283 pyperf_subprogs.bpf.o on_event 756900
For sched_ext top 5 consumers loog as follows:
File Program liveness mem --------- ------------------------------- ------------ bpf.bpf.o lavd_enqueue 164686 bpf.bpf.o lavd_select_cpu 157393 bpf.bpf.o layered_enqueue 154817 bpf.bpf.o lavd_init 127865 bpf.bpf.o layered_dispatch 110129
For Meta's internal set of programs top consumer is 1Mb.
[1] https://github.com/kernel-patches/bpf/commit/085588e787b7998a296eb320666897d80bca7c08
Impact on verification performance
==================================
Veristat results below are reported using -f insns_pct>1 -f !insns<500 filter and -t option (BPF_F_TEST_STATE_FREQ flag).
master vs patch-set, selftests (out of ~4K programs)
----------------------------------------------------
File Program Insns (A) Insns (B) Insns (DIFF) -------------------------------- -------------------------------------- --------- --------- --------------- cpumask_success.bpf.o test_global_mask_nested_deep_array_rcu 1622 1655 +33 (+2.03%) strobemeta_bpf_loop.bpf.o on_event 2163 2684 +521 (+24.09%) test_cls_redirect.bpf.o cls_redirect 36001 42515 +6514 (+18.09%) test_cls_redirect_dynptr.bpf.o cls_redirect 2299 2339 +40 (+1.74%) test_cls_redirect_subprogs.bpf.o cls_redirect 69545 78497 +8952 (+12.87%) test_l4lb_noinline.bpf.o balancer_ingress 2993 3084 +91 (+3.04%) test_xdp_noinline.bpf.o balancer_ingress_v4 3539 3616 +77 (+2.18%) test_xdp_noinline.bpf.o balancer_ingress_v6 3608 3685 +77 (+2.13%)
master vs patch-set, sched_ext (out of 148 programs)
----------------------------------------------------
File Program Insns (A) Insns (B) Insns (DIFF) --------- ---------------- --------- --------- --------------- bpf.bpf.o chaos_dispatch 2257 2287 +30 (+1.33%) bpf.bpf.o lavd_enqueue 20735 22101 +1366 (+6.59%) bpf.bpf.o lavd_select_cpu 22100 24409 +2309 (+10.45%) bpf.bpf.o layered_dispatch 25051 25606 +555 (+2.22%) bpf.bpf.o p2dq_dispatch 961 990 +29 (+3.02%) bpf.bpf.o rusty_quiescent 526 534 +8 (+1.52%) bpf.bpf.o rusty_runnable 541 547 +6 (+1.11%)
Perf report
===========
In relative terms, the analysis does not consume much CPU time. For example, here is a perf report collected for pyperf180 selftest:
# Children Self Command Shared Object Symbol
# ........ ........ ........ .................... ........................................
...
1.22% 1.22% veristat [kernel.kallsyms] [k] bpf_update_live_stack
...Eduard Zingerman (10):
bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
bpf: use compute_live_registers() info in clean_func_state
bpf: remove redundant REG_LIVE_READ check in stacksafe()
bpf: declare a few utility functions as internal api
bpf: compute instructions postorder per subprogram
bpf: callchain sensitive stack liveness tracking using CFG
bpf: enable callchain sensitive stack liveness tracking
bpf: signal error if old liveness is more conservative than new
bpf: disable and remove registers chain based liveness
bpf: table based bpf_insn_successors()
Documentation/bpf/verifier.rst | 264 ------- include/linux/bpf_verifier.h | 52 +- kernel/bpf/Makefile | 2 +- kernel/bpf/liveness.c | 723 ++++++++++++++++++ kernel/bpf/log.c | 28 +- kernel/bpf/verifier.c | 564 ++++---------- .../testing/selftests/bpf/prog_tests/align.c | 178 ++--- .../selftests/bpf/prog_tests/spin_lock.c | 12 +- .../selftests/bpf/prog_tests/test_veristat.c | 44 +- .../selftests/bpf/progs/exceptions_assert.c | 34 +- .../selftests/bpf/progs/iters_state_safety.c | 4 +- .../selftests/bpf/progs/iters_testmod_seq.c | 6 +- .../bpf/progs/mem_rdonly_untrusted.c | 4 +- .../selftests/bpf/progs/verifier_bounds.c | 38 +- .../bpf/progs/verifier_global_ptr_args.c | 4 +- .../selftests/bpf/progs/verifier_ldsx.c | 2 +- .../selftests/bpf/progs/verifier_precision.c | 16 +- .../selftests/bpf/progs/verifier_scalar_ids.c | 10 +- .../selftests/bpf/progs/verifier_spill_fill.c | 40 +- .../bpf/progs/verifier_subprog_precision.c | 6 +- .../selftests/bpf/verifier/bpf_st_mem.c | 4 +- 21 files changed, 1107 insertions(+), 928 deletions(-) create mode 100644 kernel/bpf/liveness.c
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 1/10 bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
Prepare for bpf_reg_state->live field removal by introducing a separate flag to track if clean_verifier_state() had been applied to the state. No functional changes.
--- include/linux/bpf_verifier.h | 2 +- kernel/bpf/log.c | 6 ++---- kernel/bpf/verifier.c | 13 ++++--------- 3 files changed, 7 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +1 -1
@@ -45,7 +45,6 @@ enum bpf_reg_liveness {REG_LIVE_READ64 =0x2 ,/* likewise, but full 64-bit content matters */ REG_LIVE_READ =REG_LIVE_READ32 |REG_LIVE_READ64 , REG_LIVE_WRITTEN = 0x4, /* reg was written first, screening off later reads */ - REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */ };#define ITER_PREFIX "bpf_iter_"
@@ -445,6 +444,7 @@ struct bpf_verifier_state {bool speculative;bool in_sleepable; +bool cleaned;/* first and last insn idx of this verifier state */ u32 first_insn_idx;
kernel/bpf/log.c +2 -4
@@ -545,14 +545,12 @@ static char slot_type_char[] = {static void print_liveness (struct bpf_verifier_env *env,enum bpf_reg_liveness live) { -if (live & (REG_LIVE_READ |REG_LIVE_WRITTEN |REG_LIVE_DONE )) -verbose (env,"_" ); +if (live & (REG_LIVE_READ |REG_LIVE_WRITTEN )) +verbose (env,"_" );if (live ®_LIVE_READ )verbose (env,"r" );if (live ®_LIVE_WRITTEN )verbose (env,"w" ); -if (live ®_LIVE_DONE ) -verbose (env,"D" ); }#define UNUM_MAX_DECIMAL U16_MAX
kernel/bpf/verifier.c +4 -9
@@ -1758,6 +1758,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,return err; dst_state->speculative = src->speculative; dst_state->in_sleepable = src->in_sleepable; + dst_state->cleaned = src->cleaned; dst_state->curframe = src->curframe; dst_state->branches = src->branches; dst_state->parent = src->parent;
@@ -3574,11 +3575,6 @@ static int mark_reg_read(struct bpf_verifier_env *env,/* if read wasn't screened by an earlier write ... */ if (writes && state->live ®_LIVE_WRITTEN )break ; -if (verifier_bug_if (parent->live ®_LIVE_DONE , env, -"type %s var_off %lld off %d" , -reg_type_str (env, parent->type), - parent->var_off.value, parent->off)) -return -EFAULT ;/* The first condition is more likely to be true than the * second, checked it first. */
@@ -18472,7 +18468,6 @@ static void clean_func_state(struct bpf_verifier_env *env,for (i =0 ; i <BPF_REG_FP ; i++) { live = st->regs[i].live;/* liveness must not touch this register anymore */ - st->regs[i].live |=REG_LIVE_DONE ;if (!(live ®_LIVE_READ ))/* since the register is unused, clear its state * to make further comparison simpler
@@ -18483,7 +18478,6 @@ static void clean_func_state(struct bpf_verifier_env *env,for (i =0 ; i < st->allocated_stack /BPF_REG_SIZE ; i++) { live = st->stack[i].spilled_ptr.live;/* liveness must not touch this stack slot anymore */ - st->stack[i].spilled_ptr.live |=REG_LIVE_DONE ;if (!(live ®_LIVE_READ )) {__mark_reg_not_init (env, &st->stack[i].spilled_ptr);for (j =0 ; j <BPF_REG_SIZE ; j++)
@@ -18497,6 +18491,7 @@ static void clean_verifier_state(struct bpf_verifier_env *env, {int i; + st->cleaned = true;for (i =0 ; i <= st->curframe; i++)clean_func_state (env, st->frame[i]); }
@@ -18524,7 +18519,7 @@ static void clean_verifier_state(struct bpf_verifier_env *env,* their final liveness marks are already propagated. * Hence when the verifier completes the search of state list in is_state_visited() * we can call this clean_live_states() function to mark all liveness states -* as REG_LIVE_DONE to indicate that 'parent' pointers of 'struct bpf_reg_state' +* as st->cleaned to indicate that 'parent' pointers of 'struct bpf_reg_state' * will not be used. * This function also clears the registers and stack for states that !READ * to simplify state merging.
@@ -18547,7 +18542,7 @@ static void clean_live_states(struct bpf_verifier_env *env, int insn,if (sl->state.insn_idx != insn || !same_callsites (&sl->state, cur))continue ; -if (sl->state.frame[0 ]->regs[0 ].live ®_LIVE_DONE ) +if (sl->state.cleaned)/* all regs in this state in all frames were already marked */ continue ;if (incomplete_read_marks (env, &sl->state))
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 2/10 bpf: use compute_live_registers() info in clean_func_state
In reply to Eduard ZingermanPrepare for bpf_reg_state->live field removal by leveraging insn_aux_data->live_regs_before instead of bpf_reg_state->live in compute_live_registers(). This is similar to logic in func_states_equal(). No changes in verification performance for selftests or sched_ext.
--- kernel/bpf/verifier.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +9 -6
@@ -18460,15 +18460,16 @@ static bool check_scalar_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap) }static void clean_func_state (struct bpf_verifier_env *env, -struct bpf_func_state *st) +struct bpf_func_state *st, +u32 ip) { +u16 live_regs = env->insn_aux_data[ip].live_regs_before;enum bpf_reg_liveness live;int i, j;for (i =0 ; i <BPF_REG_FP ; i++) { - live = st->regs[i].live;/* liveness must not touch this register anymore */ -if (!(live ®_LIVE_READ )) +if (!(live_regs &BIT (i)))/* since the register is unused, clear its state * to make further comparison simpler */
@@ -18489,11 +18490,13 @@ static void clean_func_state(struct bpf_verifier_env *env,static void clean_verifier_state (struct bpf_verifier_env *env,struct bpf_verifier_state *st) { -int i; +int i, ip; st->cleaned = true; -for (i =0 ; i <= st->curframe; i++) -clean_func_state (env, st->frame[i]); +for (i =0 ; i <= st->curframe; i++) { + ip =frame_insn_idx (st, i); +clean_func_state (env, st->frame[i], ip); + } }/* the parentage chains form a tree.
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 3/10 bpf: remove redundant REG_LIVE_READ check in stacksafe()
In reply to Eduard Zingermanstacksafe() is called in exact == NOT_EXACT mode only for states that had been porcessed by clean_verifier_states(). The latter replaces dead stack spills with a series of STACK_INVALID masks. Such masks are already handled by stacksafe().
--- kernel/bpf/verifier.c | 7 ------- 1 file changed, 7 deletions(-)
kernel/bpf/verifier.c +0 -7
@@ -18756,13 +18756,6 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, cur->stack[spi].slot_type[i % BPF_REG_SIZE]))return false; -if (!(old->stack[spi].spilled_ptr.live ®_LIVE_READ ) - && exact ==NOT_EXACT ) { - i +=BPF_REG_SIZE -1 ; -/* explored state didn't use this */ -continue ; - } -if (old->stack[spi].slot_type[i %BPF_REG_SIZE ] ==STACK_INVALID )continue ;
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 4/10 bpf: declare a few utility functions as internal api
In reply to Eduard ZingermanNamely, rename the following functions and add prototypes to bpf_verifier.h:
- find_containing_subprog -> bpf_find_containing_subprog
- insn_successors -> bpf_insn_successors - calls_callback -> bpf_calls_callback - fmt_stack_mask -> bpf_fmt_stack_mask
--- include/linux/bpf_verifier.h | 5 +++++ kernel/bpf/verifier.c | 34 ++++++++++++++++------------------ 2 files changed, 21 insertions(+), 18 deletions(-)
include/linux/bpf_verifier.h +5 -0
@@ -1065,4 +1065,9 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifievoid print_insn_state (struct bpf_verifier_env *env,const struct bpf_verifier_state *vstate,u32 frameno); +struct bpf_subprog_info *bpf_find_containing_subprog (struct bpf_verifier_env *env,int off); +int bpf_insn_successors (struct bpf_prog *prog,u32 idx,u32 succ[2 ]); +void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask); +bool bpf_calls_callback (struct bpf_verifier_env *env,int insn_idx); +#endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/verifier.c +16 -18
@@ -2964,7 +2964,7 @@ static int cmp_subprogs(const void *a, const void *b) }/* Find subprogram that contains instruction at 'off' */ -static struct bpf_subprog_info *find_containing_subprog (struct bpf_verifier_env *env,int off) +struct bpf_subprog_info *bpf_find_containing_subprog (struct bpf_verifier_env *env,int off) {struct bpf_subprog_info *vals = env->subprog_info;int l, r, m;
@@ -2989,7 +2989,7 @@ static int find_subprog(struct bpf_verifier_env *env, int off) {struct bpf_subprog_info *p; - p =find_containing_subprog (env, off); + p =bpf_find_containing_subprog (env, off);if (!p || p->start != off)return -ENOENT ;return p - env->subprog_info;
@@ -4196,7 +4196,7 @@ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask) } }/* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */ -static void fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask) +void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask) {DECLARE_BITMAP (mask,64 );bool first = true;
@@ -4251,8 +4251,6 @@ static void bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_histo } } -static bool calls_callback (struct bpf_verifier_env *env,int insn_idx); -/* For given verifier state backtrack_insn() is called from the last insn to * the first insn. Its purpose is to compute a bitmask of registers and * stack slots that needs precision in the parent verifier state.
@@ -4279,7 +4277,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,fmt_reg_mask (env->tmp_str_buf,TMP_STR_BUF_LEN ,bt_reg_mask (bt));verbose (env,"mark_precise: frame%d: regs=%s " , bt->frame, env->tmp_str_buf); -fmt_stack_mask (env->tmp_str_buf,TMP_STR_BUF_LEN ,bt_stack_mask (bt)); +bpf_fmt_stack_mask (env->tmp_str_buf,TMP_STR_BUF_LEN ,bt_stack_mask (bt));verbose (env,"stack=%s before " , env->tmp_str_buf);verbose (env,"%d: " , idx);verbose_insn (env, insn);
@@ -4480,7 +4478,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,* backtracking, as these registers are set by the function * invoking callback. */ -if (subseq_idx >=0 &&calls_callback (env, subseq_idx)) +if (subseq_idx >=0 &&bpf_calls_callback (env, subseq_idx))for (i =BPF_REG_1 ; i <=BPF_REG_5 ; i++)bt_clear_reg (bt, i);if (bt_reg_mask (bt) &BPF_REGMASK_ARGS ) {
@@ -4919,7 +4917,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env, bt_frame_reg_mask(bt, fr));verbose (env,"mark_precise: frame%d: parent state regs=%s " , fr, env->tmp_str_buf); -fmt_stack_mask (env->tmp_str_buf,TMP_STR_BUF_LEN , +bpf_fmt_stack_mask (env->tmp_str_buf,TMP_STR_BUF_LEN ,bt_frame_stack_mask (bt, fr));verbose (env,"stack=%s: " , env->tmp_str_buf);print_verifier_state (env, st, fr, true);
@@ -11004,7 +11002,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) "At callback return", "R0");return -EINVAL ; } -if (!calls_callback (env, callee->callsite)) { +if (!bpf_calls_callback (env, callee->callsite)) {verifier_bug (env,"in callback at %d, callsite %d !calls_callback" , *insn_idx, callee->callsite);return -EFAULT ;
@@ -17269,7 +17267,7 @@ static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off) {struct bpf_subprog_info *subprog; - subprog =find_containing_subprog (env, off); + subprog =bpf_find_containing_subprog (env, off); subprog->changes_pkt_data = true; }
@@ -17277,7 +17275,7 @@ static void mark_subprog_might_sleep(struct bpf_verifier_env *env, int off) {struct bpf_subprog_info *subprog; - subprog =find_containing_subprog (env, off); + subprog =bpf_find_containing_subprog (env, off); subprog->might_sleep = true; }
@@ -17291,8 +17289,8 @@ static void merge_callee_effects(struct bpf_verifier_env *env, int t, int w) {struct bpf_subprog_info *caller, *callee; - caller =find_containing_subprog (env, t); - callee =find_containing_subprog (env, w); + caller =bpf_find_containing_subprog (env, t); + callee =bpf_find_containing_subprog (env, w); caller->changes_pkt_data |= callee->changes_pkt_data; caller->might_sleep |= callee->might_sleep; }
@@ -17362,7 +17360,7 @@ static void mark_calls_callback(struct bpf_verifier_env *env, int idx) env->insn_aux_data[idx].calls_callback = true; } -static bool calls_callback (struct bpf_verifier_env *env,int insn_idx) +bool bpf_calls_callback (struct bpf_verifier_env *env,int insn_idx) {return env->insn_aux_data[insn_idx].calls_callback; }
@@ -19410,7 +19408,7 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) goto hit; } } -if (calls_callback (env, insn_idx)) { +if (bpf_calls_callback (env, insn_idx)) {if (states_equal (env, &sl->state, cur,RANGE_WITHIN )) goto hit; goto skip_inf_loop_check;
@@ -24136,7 +24134,7 @@ static bool can_jump(struct bpf_insn *insn)return false; } -static int insn_successors (struct bpf_prog *prog,u32 idx,u32 succ[2 ]) +int bpf_insn_successors (struct bpf_prog *prog,u32 idx,u32 succ[2 ]) {struct bpf_insn *insn = &prog->insnsi[idx];int i =0 , insn_sz;
@@ -24352,7 +24350,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)u16 new_out =0 ;u16 new_in =0 ; - succ_num =insn_successors (env->prog, insn_idx, succ); + succ_num =bpf_insn_successors (env->prog, insn_idx, succ);for (int s =0 ; s < succ_num; ++s) new_out |= state[succ[s]].in; new_in = (new_out & ~live->def) | live->use;
@@ -24521,7 +24519,7 @@ static int compute_scc(struct bpf_verifier_env *env) stack[stack_sz++] = w; }/* Visit 'w' successors */ - succ_cnt =insn_successors (env->prog, w, succ); + succ_cnt =bpf_insn_successors (env->prog, w, succ);for (j =0 ; j < succ_cnt; ++j) {if (pre[succ[j]]) { low[w] =min (low[w], low[succ[j]]);
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 5/10 bpf: compute instructions postorder per subprogram
In reply to Eduard ZingermanThe next patch would require doing postorder traversal of individual subprograms. Facilitate this by moving env->cfg.insn_postorder computation from check_cfg() to a separate pass, as check_cfg() descends into called subprograms (and it needs to, because of merge_callee_effects() logic).
env->cfg.insn_postorder is used only by compute_live_registers(), this function does not track cross subprogram dependencies, thus the change does not affect it's operation.
--- include/linux/bpf_verifier.h | 6 +++- kernel/bpf/verifier.c | 67 +++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +5 -1
@@ -665,6 +665,7 @@ struct bpf_subprog_info {/* 'start' has to be the first field otherwise find_subprog() won't work */ u32 start;/* insn idx of function entry point */ u32 linfo_idx;/* The idx to the main_prog->aux->linfo */ +u32 postorder_start;/* The idx to the env->cfg.insn_postorder */ u16 stack_depth;/* max. stack depth used by this function */ u16 stack_extra;/* offsets in range [stack_depth .. fastcall_stack_off)
@@ -794,7 +795,10 @@ struct bpf_verifier_env {struct {int *insn_state;int *insn_stack; -/* vector of instruction indexes sorted in post-order */ +/* +* vector of instruction indexes sorted in post-order, grouped by subprogram, +* see bpf_subprog_info->postorder_start. +*/ int *insn_postorder;int cur_stack;/* current position in the insn_postorder vector */
kernel/bpf/verifier.c +54 -13
103 lines not on this page · raw message
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 6/10 bpf: callchain sensitive stack liveness tracking using CFG
In reply to Eduard ZingermanThis commit adds a flow-sensitive, context-sensitive, path-insensitive data flow analysis for live stack slots:
- flow-sensitive: uses program control flow graph to compute data flow values;
- context-sensitive: collects data flow values for each possible call chain in a program;
- path-insensitive: does not distinguish between separate control flow graph paths reaching the same instruction.
Compared to the current path-sensitive analysis, this approach trades some precision for not having to enumerate every path in the program. This gives a theoretical capability to run the analysis before main verification pass. See cover letter for motivation.
The basic idea is as follows:
- Data flow values indicate stack slots that might be read and stack slots that are definitely written.
- Data flow values are collected for each
(call chain, instruction number) combination in the program.
- Within a subprogram, data flow values are propagated using control flow graph.
- Data flow values are transferred from entry instructions of callee subprograms to call sites in caller subprograms.
In other words, a tree of all possible call chains is constructed. Each node of this tree represents a subprogram. Read and write marks are collected for each instruction of each node. Live stack slots are first computed for lower level nodes. Then, information about outer stack slots that might be read or are definitely written by a subprogram is propagated one level up, to the corresponding call instructions of the upper nodes. Procedure repeats until root node is processed.
In the absence of value range analysis, stack read/write marks are collected during main verification pass, and data flow computation is triggered each time verifier.c:states_equal() needs to query the information.
Implementation details are documented in kernel/bpf/liveness.c. Quantitative data about verification performance changes and memory consumption is in the cover letter.
--- include/linux/bpf_verifier.h | 14 + kernel/bpf/Makefile | 2 +- kernel/bpf/liveness.c | 672 +++++++++++++++++++++++++++++++++++ 3 files changed, 687 insertions(+), 1 deletion(-) create mode 100644 kernel/bpf/liveness.c
include/linux/bpf_verifier.h +14 -0
@@ -745,6 +745,8 @@ struct bpf_scc_info {struct bpf_scc_visit visits[]; }; +struct bpf_liveness ; +/* single container for all structs * one verifier_env per bpf_check() call */
@@ -846,6 +848,7 @@ struct bpf_verifier_env {struct bpf_insn insn_buf[INSN_BUF_SIZE ];struct bpf_insn epilogue_buf[INSN_BUF_SIZE ];struct bpf_scc_callchain callchain_buf; +struct bpf_liveness *liveness;/* array of pointers to bpf_scc_info indexed by SCC id */ struct bpf_scc_info **scc_info;u32 scc_cnt;
@@ -1074,4 +1077,15 @@ int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]);void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask);bool bpf_calls_callback (struct bpf_verifier_env *env,int insn_idx); +int bpf_stack_liveness_init (struct bpf_verifier_env *env); +void bpf_stack_liveness_free (struct bpf_verifier_env *env); +int bpf_update_live_stack (struct bpf_verifier_env *env); +int bpf_mark_stack_read (struct bpf_verifier_env *env,u32 frameno,u32 insn_idx,u64 mask); +void bpf_mark_stack_write (struct bpf_verifier_env *env,u32 frameno,u64 mask); +int bpf_reset_stack_write_marks (struct bpf_verifier_env *env,u32 insn_idx); +int bpf_commit_stack_write_marks (struct bpf_verifier_env *env); +int bpf_live_stack_query_init (struct bpf_verifier_env *env,struct bpf_verifier_state *st); +bool bpf_stack_slot_alive (struct bpf_verifier_env *env,u32 frameno,u32 spi); +void bpf_reset_live_stack_callchain (struct bpf_verifier_env *env); +#endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/Makefile +1 -1
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse endif CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy) -obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o +obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
kernel/bpf/liveness.c new file +672 -0
672 lines not on this page · raw message
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 7/10 bpf: enable callchain sensitive stack liveness tracking
In reply to Eduard ZingermanAllocate analysis instance:
- Add bpf_stack_liveness_{init,free}() calls to bpf_check().
Notify the instance about any stack reads and writes:
- Add bpf_mark_stack_write() call at every location where REG_LIVE_WRITTEN is recorded for a stack slot.
- Add bpf_mark_stack_read() call at every location mark_reg_read() is called.
- Both bpf_mark_stack_{read,write}() rely on env->liveness->cur_instance callchain being in sync with env->cur_state. It is possible to update env->liveness->cur_instance every time a mark read/write is called, but that costs a hash table lookup and is noticeable in the performance profile. Hence, manually reset env->liveness->cur_instance whenever the verifier changes env->cur_state call stack:
- call bpf_reset_live_stack_callchain() when the verifier enters a
subprogram;
- call bpf_update_live_stack() when the verifier exits a subprogram (it implies the reset).
Make sure bpf_update_live_stack() is called for a callchain before issuing liveness queries. And make sure that bpf_update_live_stack() is called for any callee callchain first:
- Add bpf_update_live_stack() call at every location that processes BPF_EXIT:
- exit from a subprogram;
- before pop_stack() call.
This makes sure that bpf_update_live_stack() is called for callee callchains before caller callchains.
Make sure must_write marks are set to zero for instructions that do not always access the stack:
- Wrap do_check_insn() with bpf_reset_stack_write_marks() / bpf_commit_stack_write_marks() calls.
Any calls to bpf_mark_stack_write() are accumulated between this pair of calls. If no bpf_mark_stack_write() calls were made it means that the instruction does not access stack (at-least on the current verification path) and it is important to record this fact.
Finally, use bpf_live_stack_query_init() / bpf_stack_slot_alive() to query stack liveness info.
The manual tracking of the correct order for callee/caller bpf_update_live_stack() calls is a bit convoluted and may warrant some automation in future revisions.
--- kernel/bpf/verifier.c | 61 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 8 deletions(-)
kernel/bpf/verifier.c +53 -8
@@ -789,6 +789,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ state->stack[spi].spilled_ptr.live |=REG_LIVE_WRITTEN ; state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; +bpf_mark_stack_write (env, state->frameno,BIT (spi -1 ) |BIT (spi));return 0 ; }
@@ -828,6 +829,7 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat*/ state->stack[spi].spilled_ptr.live |=REG_LIVE_WRITTEN ; state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; +bpf_mark_stack_write (env, state->frameno,BIT (spi -1 ) |BIT (spi)); }static int unmark_stack_slots_dynptr (struct bpf_verifier_env *env,struct bpf_reg_state *reg)
@@ -939,6 +941,7 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,/* Same reason as unmark_stack_slots_dynptr above */ state->stack[spi].spilled_ptr.live |=REG_LIVE_WRITTEN ; state->stack[spi -1 ].spilled_ptr.live |=REG_LIVE_WRITTEN ; +bpf_mark_stack_write (env, state->frameno,BIT (spi -1 ) |BIT (spi));return 0 ; }
@@ -1066,6 +1069,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,for (j =0 ; j <BPF_REG_SIZE ; j++) slot->slot_type[j] =STACK_ITER ; +bpf_mark_stack_write (env, state->frameno,BIT (spi - i));mark_stack_slot_scratched (env, spi - i); }
@@ -1097,6 +1101,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,for (j =0 ; j <BPF_REG_SIZE ; j++) slot->slot_type[j] =STACK_INVALID ; +bpf_mark_stack_write (env, state->frameno,BIT (spi - i));mark_stack_slot_scratched (env, spi - i); }
@@ -1186,6 +1191,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, slot = &state->stack[spi]; st = &slot->spilled_ptr; +bpf_mark_stack_write (env, reg->frameno,BIT (spi));__mark_reg_known_zero (st); st->type =PTR_TO_STACK ;/* we don't have dedicated reg type */ st->live |=REG_LIVE_WRITTEN ;
@@ -1244,6 +1250,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r/* see unmark_stack_slots_dynptr() for why we need to set REG_LIVE_WRITTEN */ st->live |=REG_LIVE_WRITTEN ; +bpf_mark_stack_write (env, reg->frameno,BIT (spi));for (i =0 ; i <BPF_REG_SIZE ; i++) slot->slot_type[i] =STACK_INVALID ;
@@ -3619,6 +3626,9 @@ static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_regif (err)return err; + err =bpf_mark_stack_read (env, reg->frameno, env->insn_idx,BIT (spi - i)); +if (err) +return err;mark_stack_slot_scratched (env, spi - i); }return 0 ;
@@ -5151,6 +5161,18 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,if (err)return err; +if (!(off %BPF_REG_SIZE ) && size ==BPF_REG_SIZE ) { +/* only mark the slot as written if all 8 bytes were written +* otherwise read propagation may incorrectly stop too soon +* when stack slots are partially written. +* This heuristic means that read propagation will be +* conservative, since it will add reg_live_read marks +* to stack slots all the way to first state when programs +* writes+reads less than 8 bytes +*/ +bpf_mark_stack_write (env, state->frameno,BIT (spi)); + } +check_fastcall_stack_contract (env, state, insn_idx, off);mark_stack_slot_scratched (env, spi);if (reg && !(off %BPF_REG_SIZE ) && reg->type ==SCALAR_VALUE && env->bpf_capable) {
@@ -5420,12 +5442,16 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,struct bpf_reg_state *reg;u8 *stype, type;int insn_flags =insn_stack_access_flags (reg_state->frameno, spi); +int err; stype = reg_state->stack[spi].slot_type; reg = ®_state->stack[spi].spilled_ptr;mark_stack_slot_scratched (env, spi);check_fastcall_stack_contract (env, state, env->insn_idx, off); + err =bpf_mark_stack_read (env, reg_state->frameno, env->insn_idx,BIT (spi)); +if (err) +return err;if (is_spilled_reg (®_state->stack[spi])) {u8 spill_size =1 ;
@@ -8159,6 +8185,9 @@ static int check_stack_range_initialized(mark_reg_read (env, &state->stack[spi].spilled_ptr, state->stack[spi].spilled_ptr.parent,REG_LIVE_READ64 ); + err =bpf_mark_stack_read (env, reg->frameno, env->insn_idx,BIT (spi)); +if (err) +return err;/* We do not set REG_LIVE_WRITTEN for stack slot, as we can not * be sure that whether stack slot is written to or not. Hence, * we must still conservatively propagate reads upwards even if
@@ -10716,6 +10745,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,/* and go analyze first insn of the callee */ *insn_idx = env->subprog_info[subprog].start -1 ; +bpf_reset_live_stack_callchain (env); +if (env->log.level &BPF_LOG_LEVEL ) {verbose (env,"caller:\n" );print_verifier_state (env, state, caller->frameno, true);
@@ -18502,7 +18533,6 @@ static void clean_func_state(struct bpf_verifier_env *env, u32 ip) {u16 live_regs = env->insn_aux_data[ip].live_regs_before; -enum bpf_reg_liveness live;int i, j;for (i =0 ; i <BPF_REG_FP ; i++) {
@@ -18515,9 +18545,7 @@ static void clean_func_state(struct bpf_verifier_env *env, }for (i =0 ; i < st->allocated_stack /BPF_REG_SIZE ; i++) { - live = st->stack[i].spilled_ptr.live; -/* liveness must not touch this stack slot anymore */ -if (!(live ®_LIVE_READ )) { +if (!bpf_stack_slot_alive (env, st->frameno, i)) {__mark_reg_not_init (env, &st->stack[i].spilled_ptr);for (j =0 ; j <BPF_REG_SIZE ; j++) st->stack[i].slot_type[j] =STACK_INVALID ;
@@ -18530,6 +18558,7 @@ static void clean_verifier_state(struct bpf_verifier_env *env, {int i, ip; +bpf_live_stack_query_init (env, st); st->cleaned = true;for (i =0 ; i <= st->curframe; i++) { ip =frame_insn_idx (st, i);
@@ -18615,9 +18644,6 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,if (exact ==EXACT )return regs_exact (rold, rcur, idmap); -if (!(rold->live ®_LIVE_READ ) && exact ==NOT_EXACT ) -/* explored state didn't use this */ -return true;if (rold->type ==NOT_INIT ) {if (exact ==NOT_EXACT || rcur->type ==NOT_INIT )/* explored state can't have used this */
@@ -19856,6 +19882,9 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,return PROCESS_BPF_EXIT ;if (env->cur_state->curframe) { + err =bpf_update_live_stack (env); +if (err) +return err;/* exit from nested function */ err =prepare_func_exit (env, &env->insn_idx);if (err)
@@ -20041,7 +20070,7 @@ static int do_check(struct bpf_verifier_env *env)for (;;) {struct bpf_insn *insn;struct bpf_insn_aux_data *insn_aux; -int err; +int err, marks_err;/* reset current history entry on each new instruction */ env->cur_hist_ent =NULL ;
@@ -20134,7 +20163,15 @@ static int do_check(struct bpf_verifier_env *env)if (state->speculative && insn_aux->nospec) goto process_bpf_exit; + err =bpf_reset_stack_write_marks (env, env->insn_idx); +if (err) +return err; err =do_check_insn (env, &do_print_state); +if (err >=0 ||error_recoverable_with_nospec (err)) { + marks_err =bpf_commit_stack_write_marks (env); +if (marks_err) +return marks_err; + }if (error_recoverable_with_nospec (err) && state->speculative) {/* Prevent this speculative path from ever reaching the * insn that would have been unsafe to execute.
@@ -20173,6 +20210,9 @@ static int do_check(struct bpf_verifier_env *env) process_bpf_exit:mark_verifier_state_scratched (env); err =update_branch_counts (env, env->cur_state); +if (err) +return err; + err =bpf_update_live_stack (env);if (err)return err; err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
@@ -24733,6 +24773,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3if (ret <0 ) goto skip_full_check; + ret =bpf_stack_liveness_init (env); +if (ret) + goto skip_full_check; + ret =check_attach_btf_id (env);if (ret) goto skip_full_check;
@@ -24882,6 +24926,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3mutex_unlock (&bpf_verifier_lock);vfree (env->insn_aux_data); err_free_env: +bpf_stack_liveness_free (env);kvfree (env->cfg.insn_postorder);kvfree (env->scc_info);kvfree (env);
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 8/10 bpf: signal error if old liveness is more conservative than new
In reply to Eduard ZingermanUnlike the new algorithm, register chain based liveness tracking is fully path sensitive, and thus should be strictly more accurate. Validate the new algorithm by signaling an error whenever it considers a stack slot dead while the old algorithm considers it alive.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 9 +++++++++ 2 files changed, 10 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -852,6 +852,7 @@ struct bpf_verifier_env {/* array of pointers to bpf_scc_info indexed by SCC id */ struct bpf_scc_info **scc_info;u32 scc_cnt; +bool internal_error; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
kernel/bpf/verifier.c +9 -0
@@ -18546,6 +18546,11 @@ static void clean_func_state(struct bpf_verifier_env *env,for (i =0 ; i < st->allocated_stack /BPF_REG_SIZE ; i++) {if (!bpf_stack_slot_alive (env, st->frameno, i)) { +if (st->stack[i].spilled_ptr.live ®_LIVE_READ ) { +verifier_bug (env,"incorrect live marks #1 for insn %d frameno %d spi %d\n" , + env->insn_idx, st->frameno, i); + env->internal_error = true; + }__mark_reg_not_init (env, &st->stack[i].spilled_ptr);for (j =0 ; j <BPF_REG_SIZE ; j++) st->stack[i].slot_type[j] =STACK_INVALID ;
@@ -19516,6 +19521,8 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) loop =incomplete_read_marks (env, &sl->state);if (states_equal (env, &sl->state, cur, loop ?RANGE_WITHIN :NOT_EXACT )) { hit: +if (env->internal_error) +return -EFAULT ; sl->hit_cnt++;/* reached equivalent register/stack state, * prune the search.
@@ -19630,6 +19637,8 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)return 1 ; } miss: +if (env->internal_error) +return -EFAULT ;/* when new state is not going to be added do not increase miss count. * Otherwise several loop iterations will remove the state * recorded earlier. The goal of these heuristics is to have
Signature · 1 line
2.47.3
Eduard Zingerman
maintainerPATCH 9/10 bpf: disable and remove registers chain based liveness
In reply to Eduard ZingermanRemove register chain based liveness tracking:
- struct bpf_reg_state->{parent,live} fields are no longer needed;
- REG_LIVE_WRITTEN marks are superseded by bpf_mark_stack_write() calls;
- mark_reg_read() calls are superseded by bpf_mark_stack_read();
- log.c:print_liveness() is superseded by logging in liveness.c;
- propagate_liveness() is superseded by bpf_update_live_stack();
- no need to establish register chains in is_state_visited() anymore;
- fix a bunch of tests expecting "_w" suffixes in verifier log messages.
--- Documentation/bpf/verifier.rst | 264 --------------- include/linux/bpf_verifier.h | 25 -- kernel/bpf/log.c | 26 +- kernel/bpf/verifier.c | 312 ++---------------- .../testing/selftests/bpf/prog_tests/align.c | 178 +++++----- .../selftests/bpf/prog_tests/spin_lock.c | 12 +- .../selftests/bpf/prog_tests/test_veristat.c | 44 +-- .../selftests/bpf/progs/exceptions_assert.c | 34 +- .../selftests/bpf/progs/iters_state_safety.c | 4 +- .../selftests/bpf/progs/iters_testmod_seq.c | 6 +- .../bpf/progs/mem_rdonly_untrusted.c | 4 +- .../selftests/bpf/progs/verifier_bounds.c | 38 +-- .../bpf/progs/verifier_global_ptr_args.c | 4 +- .../selftests/bpf/progs/verifier_ldsx.c | 2 +- .../selftests/bpf/progs/verifier_precision.c | 16 +- .../selftests/bpf/progs/verifier_scalar_ids.c | 10 +- .../selftests/bpf/progs/verifier_spill_fill.c | 40 +-- .../bpf/progs/verifier_subprog_precision.c | 6 +- .../selftests/bpf/verifier/bpf_st_mem.c | 4 +- 19 files changed, 224 insertions(+), 805 deletions(-)
Documentation/bpf/verifier.rst +0 -264
@@ -347,270 +347,6 @@ However, only the value of register ``r1`` is important to successfully finish verification. The goal of the liveness tracking algorithm is to spot this fact and figure out that both states are actually equivalent. -Data structures -~~~~~~~~~~~~~~~ - -Liveness is tracked using the following data structures:: - - enum bpf_reg_liveness { - REG_LIVE_NONE = 0, - REG_LIVE_READ32 = 0x1, - REG_LIVE_READ64 = 0x2, - REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64, - REG_LIVE_WRITTEN = 0x4, - REG_LIVE_DONE = 0x8, - }; - - struct bpf_reg_state { - ... - struct bpf_reg_state *parent; - ... - enum bpf_reg_liveness live; - ... - }; - - struct bpf_stack_state { - struct bpf_reg_state spilled_ptr; - ... - }; - - struct bpf_func_state { - struct bpf_reg_state regs[MAX_BPF_REG]; - ... - struct bpf_stack_state *stack; - } - - struct bpf_verifier_state { - struct bpf_func_state *frame[MAX_CALL_FRAMES]; - struct bpf_verifier_state *parent; - ... - } - -* ``REG_LIVE_NONE`` is an initial value assigned to ``->live`` fields upon new - verifier state creation; - -* ``REG_LIVE_WRITTEN`` means that the value of the register (or stack slot) is - defined by some instruction verified between this verifier state's parent and - verifier state itself; - -* ``REG_LIVE_READ{32,64}`` means that the value of the register (or stack slot) - is read by a some child state of this verifier state; - -* ``REG_LIVE_DONE`` is a marker used by ``clean_verifier_state()`` to avoid - processing same verifier state multiple times and for some sanity checks; - -* ``->live`` field values are formed by combining ``enum bpf_reg_liveness`` - values using bitwise or. - -Register parentage chains -~~~~~~~~~~~~~~~~~~~~~~~~~ - -In order to propagate information between parent and child states, a *register -parentage chain* is established. Each register or stack slot is linked to a -corresponding register or stack slot in its parent state via a ``->parent`` -pointer. This link is established upon state creation in ``is_state_visited()`` -and might be modified by ``set_callee_state()`` called from -``__check_func_call()``. - -The rules for correspondence between registers / stack slots are as follows: - -* For the current stack frame, registers and stack slots of the new state are - linked to the registers and stack slots of the parent state with the same - indices. - -* For the outer stack frames, only callee saved registers (r6-r9) and stack - slots are linked to the registers and stack slots of the parent state with the - same indices. - -* When function call is processed a new ``struct bpf_func_state`` instance is - allocated, it encapsulates a new set of registers and stack slots. For this - new frame, parent links for r6-r9 and stack slots are set to nil, parent links - for r1-r5 are set to match caller r1-r5 parent links. - -This could be illustrated by the following diagram (arrows stand for -``->parent`` pointers):: - - ... ; Frame #0, some instructions - --- checkpoint #0 --- - 1 : r6 = 42 ; Frame #0 - --- checkpoint #1 --- - 2 : call foo() ; Frame #0 - ... ; Frame #1, instructions from foo() - --- checkpoint #2 --- - ... ; Frame #1, instructions from foo() - --- checkpoint #3 --- - exit ; Frame #1, return from foo() - 3 : r1 = r6 ; Frame #0 <- current state - - +-------------------------------+-------------------------------+ - | Frame #0 | Frame #1 | - Checkpoint +-------------------------------+-------------------------------+ - #0 | r0 | r1-r5 | r6-r9 | fp-8 ... | - +-------------------------------+ - ^ ^ ^ ^ - | | | | - Checkpoint +-------------------------------+ - #1 | r0 | r1-r5 | r6-r9 | fp-8 ... | - +-------------------------------+ - ^ ^ ^ - |_______|_______|_______________ - | | | - nil nil | | | nil nil - | | | | | | | - Checkpoint +-------------------------------+-------------------------------+ - #2 | r0 | r1-r5 | r6-r9 | fp-8 ... | r0 | r1-r5 | r6-r9 | fp-8 ... | - +-------------------------------+-------------------------------+ - ^ ^ ^ ^ ^ - nil nil | | | | | - | | | | | | | - Checkpoint +-------------------------------+-------------------------------+ - #3 | r0 | r1-r5 | r6-r9 | fp-8 ... | r0 | r1-r5 | r6-r9 | fp-8 ... | - +-------------------------------+-------------------------------+ - ^ ^ - nil nil | | - | | | | - Current +-------------------------------+ - state | r0 | r1-r5 | r6-r9 | fp-8 ... | - +-------------------------------+ - \ - r6 read mark is propagated via these links - all the way up to checkpoint #1. - The checkpoint #1 contains a write mark for r6 - because of instruction (1), thus read propagation - does not reach checkpoint #0 (see section below). - -Liveness marks tracking -~~~~~~~~~~~~~~~~~~~~~~~ - -For each processed instruction, the verifier tracks read and written registers -and stack slots. The main idea of the algorithm is that read marks propagate -back along the state parentage chain until they hit a write mark, which 'screens -off' earlier states from the read. The information about reads is propagated by -function ``mark_reg_read()`` which could be summarized as follows:: - - mark_reg_read(struct bpf_reg_state *state, ...): - parent = state->parent - while parent: - if state->live & REG_LIVE_WRITTEN: - break - if parent->live & REG_LIVE_READ64: - break - parent->live |= REG_LIVE_READ64 - state = parent - parent = state->parent - -Notes: - -* The read marks are applied to the **parent** state while write marks are - applied to the **current** state. The write mark on a register or stack slot - means that it is updated by some instruction in the straight-line code leading - from the parent state to the current state. - -* Details about REG_LIVE_READ32 are omitted. - -* Function ``propagate_liveness()`` (see section :ref:`read_marks_for_cache_hits`) - might override the first parent link. Please refer to the comments in the - ``propagate_liveness()`` and ``mark_reg_read()`` source code for further - details. - -Because stack writes could have different sizes ``REG_LIVE_WRITTEN`` marks are -applied conservatively: stack slots are marked as written only if write size -corresponds to the size of the register, e.g. see function ``save_register_state()``. - -Consider the following example:: - - 0: (*u64)(r10 - 8) = 0 ; define 8 bytes of fp-8 - --- checkpoint #0 --- - 1: (*u32)(r10 - 8) = 1 ; redefine lower 4 bytes - 2: r1 = (*u32)(r10 - 8) ; read lower 4 bytes defined at (1) - 3: r2 = (*u32)(r10 - 4) ; read upper 4 bytes defined at (0) - -As stated above, the write at (1) does not count as ``REG_LIVE_WRITTEN``. Should -it be otherwise, the algorithm above wouldn't be able to propagate the read mark -from (3) to checkpoint #0. - -Once the ``BPF_EXIT`` instruction is reached ``update_branch_counts()`` is -called to update the ``->branches`` counter for each verifier state in a chain -of parent verifier states. When the ``->branches`` counter reaches zero the -verifier state becomes a valid entry in a set of cached verifier states. - -Each entry of the verifier states cache is post-processed by a function -``clean_live_states()``. This function marks all registers and stack slots -without ``REG_LIVE_READ{32,64}`` marks as ``NOT_INIT`` or ``STACK_INVALID``. -Registers/stack slots marked in this way are ignored in function ``stacksafe()`` -called from ``states_equal()`` when a state cache entry is considered for -equivalence with a current state. - -Now it is possible to explain how the example from the beginning of the section -works:: - - 0: call bpf_get_prandom_u32() - 1: r1 = 0 - 2: if r0 == 0 goto +1 - 3: r0 = 1 - --- checkpoint[0] --- - 4: r0 = r1 - 5: exit - -* At instruction #2 branching point is reached and state ``{ r0 == 0, r1 == 0, pc == 4 }`` - is pushed to states processing queue (pc stands for program counter). - -* At instruction #4: - - * ``checkpoint[0]`` states cache entry is created: ``{ r0 == 1, r1 == 0, pc == 4 }``; - * ``checkpoint[0].r0`` is marked as written; - * ``checkpoint[0].r1`` is marked as read; - -* At instruction #5 exit is reached and ``checkpoint[0]`` can now be processed - by ``clean_live_states()``. After this processing ``checkpoint[0].r1`` has a - read mark and all other registers and stack slots are marked as ``NOT_INIT`` - or ``STACK_INVALID`` - -* The state ``{ r0 == 0, r1 == 0, pc == 4 }`` is popped from the states queue - and is compared against a cached state ``{ r1 == 0, pc == 4 }``, the states - are considered equivalent. - -.. _read_marks_for_cache_hits: - -Read marks propagation for cache hits -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Another point is the handling of read marks when a previously verified state is -found in the states cache. Upon cache hit verifier must behave in the same way -as if the current state was verified to the program exit. This means that all -read marks, present on registers and stack slots of the cached state, must be -propagated over the parentage chain of the current state. Example below shows -why this is important. Function ``propagate_liveness()`` handles this case. - -Consider the following state parentage chain (S is a starting state, A-E are -derived states, -> arrows show which state is derived from which):: - - r1 read - <------------- A[r1] == 0 - C[r1] == 0 - S ---> A ---> B ---> exit E[r1] == 1 - | - ` ---> C ---> D - | - ` ---> E ^ - |___ suppose all these - ^ states are at insn #Y - | - suppose all these - states are at insn #X - -* Chain of states ``S -> A -> B -> exit`` is verified first. - -* While ``B -> exit`` is verified, register ``r1`` is read and this read mark is - propagated up to state ``A``. - -* When chain of states ``C -> D`` is verified the state ``D`` turns out to be - equivalent to state ``B``. - -* The read mark for ``r1`` has to be propagated to state ``C``, otherwise state - ``C`` might get mistakenly marked as equivalent to state ``E`` even though - values for register ``r1`` differ between ``C`` and ``E``. - Understanding eBPF verifier messages ====================================
include/linux/bpf_verifier.h +0 -25
@@ -26,27 +26,6 @@/* Patch buffer size */ #define INSN_BUF_SIZE 32 -/* Liveness marks, used for registers and spilled-regs (in stack slots). -* Read marks propagate upwards until they find a write mark; they record that -* "one of this state's descendants read this reg" (and therefore the reg is -* relevant for states_equal() checks). -* Write marks collect downwards and do not propagate; they record that "the -* straight-line code that reached this state (from its parent) wrote this reg" -* (and therefore that reads propagated from this state or its descendants -* should not propagate to its parent). -* A state with a write mark can receive read marks; it just won't propagate -* them to its parent, since the write mark is a property, not of the state, -* but of the link between it and its parent. See mark_reg_read() and -* mark_stack_slot_read() in kernel/bpf/verifier.c. -*/ -enum bpf_reg_liveness { -REG_LIVE_NONE =0 ,/* reg hasn't been read or written this branch */ -REG_LIVE_READ32 =0x1 ,/* reg was read, so we're sensitive to initial value */ -REG_LIVE_READ64 =0x2 ,/* likewise, but full 64-bit content matters */ -REG_LIVE_READ =REG_LIVE_READ32 |REG_LIVE_READ64 , -REG_LIVE_WRITTEN =0x4 ,/* reg was written first, screening off later reads */ -}; -#define ITER_PREFIX "bpf_iter_"enum bpf_iter_state {
@@ -211,8 +190,6 @@ struct bpf_reg_state {* allowed and has the same effect as bpf_sk_release(sk). */ u32 ref_obj_id; -/* parentage chain for liveness checking */ -struct bpf_reg_state *parent;/* Inside the callee two registers can be both PTR_TO_STACK like * R1=fp-8 and R2=fp-8, but one of them points to this function stack * while another to the caller's stack. To differentiate them 'frameno'
@@ -225,7 +202,6 @@ struct bpf_reg_state {* patching which only happens after main verification finished. */ s32 subreg_def; -enum bpf_reg_liveness live;/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */ bool precise; };
@@ -852,7 +828,6 @@ struct bpf_verifier_env {/* array of pointers to bpf_scc_info indexed by SCC id */ struct bpf_scc_info **scc_info;u32 scc_cnt; -bool internal_error; };static inline struct bpf_func_info_aux *subprog_aux (struct bpf_verifier_env *env,int subprog)
kernel/bpf/log.c +4 -22
@@ -542,17 +542,6 @@ static char slot_type_char[] = { [STACK_IRQ_FLAG] = 'f' }; -static void print_liveness (struct bpf_verifier_env *env, -enum bpf_reg_liveness live) -{ -if (live & (REG_LIVE_READ |REG_LIVE_WRITTEN )) -verbose (env,"_" ); -if (live ®_LIVE_READ ) -verbose (env,"r" ); -if (live ®_LIVE_WRITTEN ) -verbose (env,"w" ); -} -#define UNUM_MAX_DECIMAL U16_MAX#define SNUM_MAX_DECIMAL S16_MAX#define SNUM_MIN_DECIMAL S16_MIN
@@ -770,7 +759,6 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifieif (!print_all && !reg_scratched (env, i))continue ;verbose (env," R%d" , i); -print_liveness (env, reg->live);verbose (env,"=" );print_reg_state (env, state, reg); }
@@ -803,9 +791,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifiebreak ; types_buf[j] ='\0' ; -verbose (env," fp%d" , (-i -1 ) *BPF_REG_SIZE ); -print_liveness (env, reg->live); -verbose (env,"=%s" , types_buf); +verbose (env," fp%d=%s" , (-i -1 ) *BPF_REG_SIZE , types_buf);print_reg_state (env, state, reg);break ;case STACK_DYNPTR :
@@ -814,7 +800,6 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie reg = &state->stack[i].spilled_ptr;verbose (env," fp%d" , (-i -1 ) *BPF_REG_SIZE ); -print_liveness (env, reg->live);verbose (env,"=dynptr_%s(" ,dynptr_type_str (reg->dynptr.type));if (reg->id)verbose_a ("id=%d" , reg->id);
@@ -829,9 +814,8 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifieif (!reg->ref_obj_id)continue ; -verbose (env," fp%d" , (-i -1 ) *BPF_REG_SIZE ); -print_liveness (env, reg->live); -verbose (env,"=iter_%s(ref_id=%d,state=%s,depth=%u)" , +verbose (env," fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)" , + (-i -1 ) *BPF_REG_SIZE ,iter_type_str (reg->iter.btf, reg->iter.btf_id), reg->ref_obj_id,iter_state_str (reg->iter.state), reg->iter.depth);
@@ -839,9 +823,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifiecase STACK_MISC :case STACK_ZERO :default : -verbose (env," fp%d" , (-i -1 ) *BPF_REG_SIZE ); -print_liveness (env, reg->live); -verbose (env,"=%s" , types_buf); +verbose (env," fp%d=%s" , (-i -1 ) *BPF_REG_SIZE , types_buf);break ; } }
kernel/bpf/verifier.c +19 -293
567 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/align.c +89 -89
309 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/spin_lock.c +6 -6
@@ -13,22 +13,22 @@ static struct {const char *err_msg; } spin_lock_fail_tests[] = { {"lock_id_kptr_preserve" , -"5: (bf) r1 = r0 ; R0_w=ptr_foo(id=2,ref_obj_id=2) " -"R1_w=ptr_foo(id=2,ref_obj_id=2) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n" +"5: (bf) r1 = r0 ; R0=ptr_foo(id=2,ref_obj_id=2) " +"R1=ptr_foo(id=2,ref_obj_id=2) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n" "R1 type=ptr_ expected=percpu_ptr_" }, { "lock_id_global_zero", -"; R1_w=map_value(map=.data.A,ks=4,vs=4)\n2: (85) call bpf_this_cpu_ptr#154\n" +"; R1=map_value(map=.data.A,ks=4,vs=4)\n2: (85) call bpf_this_cpu_ptr#154\n" "R1 type=map_value expected=percpu_ptr_" }, { "lock_id_mapval_preserve","[0-9]\\+: (bf) r1 = r0 ;" -" R0_w=map_value(id=1,map=array_map,ks=4,vs=8)" -" R1_w=map_value(id=1,map=array_map,ks=4,vs=8)\n" +" R0=map_value(id=1,map=array_map,ks=4,vs=8)" +" R1=map_value(id=1,map=array_map,ks=4,vs=8)\n" "[0-9]\\+: (85) call bpf_this_cpu_ptr#154\n" "R1 type=map_value expected=percpu_ptr_" }, { "lock_id_innermapval_preserve","[0-9]\\+: (bf) r1 = r0 ;" " R0=map_value(id=2,ks=4,vs=8)" -" R1_w=map_value(id=2,ks=4,vs=8)\n" +" R1=map_value(id=2,ks=4,vs=8)\n" "[0-9]\\+: (85) call bpf_this_cpu_ptr#154\n" "R1 type=map_value expected=percpu_ptr_" }, { "lock_id_mismatch_kptr_kptr", "bpf_spin_unlock of different lock" },
tools/testing/selftests/bpf/prog_tests/test_veristat.c +22 -22
56 lines not on this page · raw message
tools/testing/selftests/bpf/progs/exceptions_assert.c +17 -17
@@ -18,43 +18,43 @@return *(u64 *)num; \ } -__msg(": R0_w=0xffffffff80000000") +__msg(": R0=0xffffffff80000000") check_assert(s64, ==, eq_int_min, INT_MIN); -__msg(": R0_w=0x7fffffff") +__msg(": R0=0x7fffffff") check_assert(s64, ==, eq_int_max, INT_MAX); -__msg(": R0_w=0") +__msg(": R0=0") check_assert(s64, ==, eq_zero, 0); -__msg(": R0_w=0x8000000000000000 R1_w=0x8000000000000000") +__msg(": R0=0x8000000000000000 R1=0x8000000000000000") check_assert(s64, ==, eq_llong_min, LLONG_MIN); -__msg(": R0_w=0x7fffffffffffffff R1_w=0x7fffffffffffffff") +__msg(": R0=0x7fffffffffffffff R1=0x7fffffffffffffff") check_assert(s64, ==, eq_llong_max, LLONG_MAX); -__msg(": R0_w=scalar(id=1,smax=0x7ffffffe)") +__msg(": R0=scalar(id=1,smax=0x7ffffffe)") check_assert(s64, <, lt_pos, INT_MAX); -__msg(": R0_w=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +__msg(": R0=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))") check_assert(s64, <, lt_zero, 0); -__msg(": R0_w=scalar(id=1,smax=0xffffffff7fffffff") +__msg(": R0=scalar(id=1,smax=0xffffffff7fffffff") check_assert(s64, <, lt_neg, INT_MIN); -__msg(": R0_w=scalar(id=1,smax=0x7fffffff)") +__msg(": R0=scalar(id=1,smax=0x7fffffff)") check_assert(s64, <=, le_pos, INT_MAX); -__msg(": R0_w=scalar(id=1,smax=0)") +__msg(": R0=scalar(id=1,smax=0)") check_assert(s64, <=, le_zero, 0); -__msg(": R0_w=scalar(id=1,smax=0xffffffff80000000") +__msg(": R0=scalar(id=1,smax=0xffffffff80000000") check_assert(s64, <=, le_neg, INT_MIN); -__msg(": R0_w=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") +__msg(": R0=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") check_assert(s64, >, gt_pos, INT_MAX); -__msg(": R0_w=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") +__msg(": R0=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") check_assert(s64, >, gt_zero, 0); -__msg(": R0_w=scalar(id=1,smin=0xffffffff80000001") +__msg(": R0=scalar(id=1,smin=0xffffffff80000001") check_assert(s64, >, gt_neg, INT_MIN); -__msg(": R0_w=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") +__msg(": R0=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") check_assert(s64, >=, ge_pos, INT_MAX); -__msg(": R0_w=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") +__msg(": R0=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))") check_assert(s64, >=, ge_zero, 0); -__msg(": R0_w=scalar(id=1,smin=0xffffffff80000000") +__msg(": R0=scalar(id=1,smin=0xffffffff80000000") check_assert(s64, >=, ge_neg, INT_MIN);SEC ("?tc" )
tools/testing/selftests/bpf/progs/iters_state_safety.c +2 -2
@@ -30,7 +30,7 @@ int force_clang_to_emit_btf_for_externs(void *ctx) SEC("?raw_tp") __success __log_level(2) -__msg("fp-8_w=iter_num(ref_id=1,state=active,depth=0)") +__msg("fp-8=iter_num(ref_id=1,state=active,depth=0)")int create_and_destroy (void *ctx) {struct bpf_iter_num iter;
@@ -196,7 +196,7 @@ int leak_iter_from_subprog_fail(void *ctx) SEC("?raw_tp") __success __log_level(2) -__msg("fp-8_w=iter_num(ref_id=1,state=active,depth=0)") +__msg("fp-8=iter_num(ref_id=1,state=active,depth=0)")int valid_stack_reuse (void *ctx) {struct bpf_iter_num iter;
tools/testing/selftests/bpf/progs/iters_testmod_seq.c +3 -3
@@ -20,7 +20,7 @@ __s64 res_empty; SEC("raw_tp/sys_enter") __success __log_level(2) -__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)") +__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)") __msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)") __msg("call bpf_iter_testmod_seq_destroy")int testmod_seq_empty (const void *ctx)
@@ -38,7 +38,7 @@ __s64 res_full; SEC("raw_tp/sys_enter") __success __log_level(2) -__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)") +__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)") __msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)") __msg("call bpf_iter_testmod_seq_destroy")int testmod_seq_full (const void *ctx)
@@ -58,7 +58,7 @@ static volatile int zero = 0; SEC("raw_tp/sys_enter") __success __log_level(2) -__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)") +__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)") __msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)") __msg("call bpf_iter_testmod_seq_destroy")int testmod_seq_truncated (const void *ctx)
tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c +2 -2
@@ -8,8 +8,8 @@ SEC("tp_btf/sys_enter") __success __log_level(2) -__msg("r8 = *(u64 *)(r7 +0) ; R7_w=ptr_nameidata(off={{[0-9]+}}) R8_w=rdonly_untrusted_mem(sz=0)") -__msg("r9 = *(u8 *)(r8 +0) ; R8_w=rdonly_untrusted_mem(sz=0) R9_w=scalar") +__msg("r8 = *(u64 *)(r7 +0) ; R7=ptr_nameidata(off={{[0-9]+}}) R8=rdonly_untrusted_mem(sz=0)") +__msg("r9 = *(u8 *)(r8 +0) ; R8=rdonly_untrusted_mem(sz=0) R9=scalar")int btf_id_to_ptr_mem (void *ctx) {struct task_struct *task;
tools/testing/selftests/bpf/progs/verifier_bounds.c +19 -19
@@ -926,7 +926,7 @@ l1_%=: r0 = 0; \ SEC("socket") __description("bounds check for non const xor src dst") __success __log_level(2) -__msg("5: (af) r0 ^= r6 ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))") +__msg("5: (af) r0 ^= r6 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")__naked void non_const_xor_src_dst (void ) { asm volatile (" \
@@ -947,7 +947,7 @@ __naked void non_const_xor_src_dst(void) SEC("socket") __description("bounds check for non const or src dst") __success __log_level(2) -__msg("5: (4f) r0 |= r6 ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))") +__msg("5: (4f) r0 |= r6 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")__naked void non_const_or_src_dst (void ) { asm volatile (" \
@@ -968,7 +968,7 @@ __naked void non_const_or_src_dst(void) SEC("socket") __description("bounds check for non const mul regs") __success __log_level(2) -__msg("5: (2f) r0 *= r6 ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=3825,var_off=(0x0; 0xfff))") +__msg("5: (2f) r0 *= r6 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=3825,var_off=(0x0; 0xfff))")__naked void non_const_mul_regs (void ) { asm volatile (" \
@@ -1241,7 +1241,7 @@ l0_%=: r0 = 0; \ SEC("tc") __description("multiply mixed sign bounds. test 1") __success __log_level(2) -__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=umin=0x1bc16d5cd4927ee1,smax=umax=0x1bc16d674ec80000,smax32=0x7ffffeff,umax32=0xfffffeff,var_off=(0x1bc16d4000000000; 0x3ffffffeff))") +__msg("r6 *= r7 {{.*}}; R6=scalar(smin=umin=0x1bc16d5cd4927ee1,smax=umax=0x1bc16d674ec80000,smax32=0x7ffffeff,umax32=0xfffffeff,var_off=(0x1bc16d4000000000; 0x3ffffffeff))")__naked void mult_mixed0_sign (void ) { asm volatile (
@@ -1264,7 +1264,7 @@ __naked void mult_mixed0_sign(void) SEC("tc") __description("multiply mixed sign bounds. test 2") __success __log_level(2) -__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=smin32=-100,smax=smax32=200)") +__msg("r6 *= r7 {{.*}}; R6=scalar(smin=smin32=-100,smax=smax32=200)")__naked void mult_mixed1_sign (void ) { asm volatile (
@@ -1287,7 +1287,7 @@ __naked void mult_mixed1_sign(void) SEC("tc") __description("multiply negative bounds") __success __log_level(2) -__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=umin=smin32=umin32=0x3ff280b0,smax=umax=smax32=umax32=0x3fff0001,var_off=(0x3ff00000; 0xf81ff))") +__msg("r6 *= r7 {{.*}}; R6=scalar(smin=umin=smin32=umin32=0x3ff280b0,smax=umax=smax32=umax32=0x3fff0001,var_off=(0x3ff00000; 0xf81ff))")__naked void mult_sign_bounds (void ) { asm volatile (
@@ -1311,7 +1311,7 @@ __naked void mult_sign_bounds(void) SEC("tc") __description("multiply bounds that don't cross signed boundary") __success __log_level(2) -__msg("r8 *= r6 {{.*}}; R6_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=11,var_off=(0x0; 0xb)) R8_w=scalar(smin=0,smax=umax=0x7b96bb0a94a3a7cd,var_off=(0x0; 0x7fffffffffffffff))") +__msg("r8 *= r6 {{.*}}; R6=scalar(smin=smin32=0,smax=umax=smax32=umax32=11,var_off=(0x0; 0xb)) R8=scalar(smin=0,smax=umax=0x7b96bb0a94a3a7cd,var_off=(0x0; 0x7fffffffffffffff))")__naked void mult_no_sign_crossing (void ) { asm volatile (
@@ -1331,7 +1331,7 @@ __naked void mult_no_sign_crossing(void) SEC("tc") __description("multiplication overflow, result in unbounded reg. test 1") __success __log_level(2) -__msg("r6 *= r7 {{.*}}; R6_w=scalar()") +__msg("r6 *= r7 {{.*}}; R6=scalar()")__naked void mult_unsign_ovf (void ) { asm volatile (
@@ -1353,7 +1353,7 @@ __naked void mult_unsign_ovf(void) SEC("tc") __description("multiplication overflow, result in unbounded reg. test 2") __success __log_level(2) -__msg("r6 *= r7 {{.*}}; R6_w=scalar()") +__msg("r6 *= r7 {{.*}}; R6=scalar()")__naked void mult_sign_ovf (void ) { asm volatile (
@@ -1376,7 +1376,7 @@ __naked void mult_sign_ovf(void) SEC("socket") __description("64-bit addition, all outcomes overflow") __success __log_level(2) -__msg("5: (0f) r3 += r3 {{.*}} R3_w=scalar(umin=0x4000000000000000,umax=0xfffffffffffffffe)") +__msg("5: (0f) r3 += r3 {{.*}} R3=scalar(umin=0x4000000000000000,umax=0xfffffffffffffffe)") __retval(0)__naked void add64_full_overflow (void ) {
@@ -1396,7 +1396,7 @@ __naked void add64_full_overflow(void) SEC("socket") __description("64-bit addition, partial overflow, result in unbounded reg") __success __log_level(2) -__msg("4: (0f) r3 += r3 {{.*}} R3_w=scalar()") +__msg("4: (0f) r3 += r3 {{.*}} R3=scalar()") __retval(0)__naked void add64_partial_overflow (void ) {
@@ -1416,7 +1416,7 @@ __naked void add64_partial_overflow(void) SEC("socket") __description("32-bit addition overflow, all outcomes overflow") __success __log_level(2) -__msg("4: (0c) w3 += w3 {{.*}} R3_w=scalar(smin=umin=umin32=0x40000000,smax=umax=umax32=0xfffffffe,var_off=(0x0; 0xffffffff))") +__msg("4: (0c) w3 += w3 {{.*}} R3=scalar(smin=umin=umin32=0x40000000,smax=umax=umax32=0xfffffffe,var_off=(0x0; 0xffffffff))") __retval(0)__naked void add32_full_overflow (void ) {
@@ -1436,7 +1436,7 @@ __naked void add32_full_overflow(void) SEC("socket") __description("32-bit addition, partial overflow, result in unbounded u32 bounds") __success __log_level(2) -__msg("4: (0c) w3 += w3 {{.*}} R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))") +__msg("4: (0c) w3 += w3 {{.*}} R3=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))") __retval(0)__naked void add32_partial_overflow (void ) {
@@ -1456,7 +1456,7 @@ __naked void add32_partial_overflow(void) SEC("socket") __description("64-bit subtraction, all outcomes underflow") __success __log_level(2) -__msg("6: (1f) r3 -= r1 {{.*}} R3_w=scalar(umin=1,umax=0x8000000000000000)") +__msg("6: (1f) r3 -= r1 {{.*}} R3=scalar(umin=1,umax=0x8000000000000000)") __retval(0)__naked void sub64_full_overflow (void ) {
@@ -1477,7 +1477,7 @@ __naked void sub64_full_overflow(void) SEC("socket") __description("64-bit subtraction, partial overflow, result in unbounded reg") __success __log_level(2) -__msg("3: (1f) r3 -= r2 {{.*}} R3_w=scalar()") +__msg("3: (1f) r3 -= r2 {{.*}} R3=scalar()") __retval(0)__naked void sub64_partial_overflow (void ) {
@@ -1496,7 +1496,7 @@ __naked void sub64_partial_overflow(void) SEC("socket") __description("32-bit subtraction overflow, all outcomes underflow") __success __log_level(2) -__msg("5: (1c) w3 -= w1 {{.*}} R3_w=scalar(smin=umin=umin32=1,smax=umax=umax32=0x80000000,var_off=(0x0; 0xffffffff))") +__msg("5: (1c) w3 -= w1 {{.*}} R3=scalar(smin=umin=umin32=1,smax=umax=umax32=0x80000000,var_off=(0x0; 0xffffffff))") __retval(0)__naked void sub32_full_overflow (void ) {
@@ -1517,7 +1517,7 @@ __naked void sub32_full_overflow(void) SEC("socket") __description("32-bit subtraction, partial overflow, result in unbounded u32 bounds") __success __log_level(2) -__msg("3: (1c) w3 -= w2 {{.*}} R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))") +__msg("3: (1c) w3 -= w2 {{.*}} R3=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))") __retval(0)__naked void sub32_partial_overflow (void ) {
@@ -1617,7 +1617,7 @@ l0_%=: r0 = 0; \ SEC("socket") __description("bounds deduction cross sign boundary, positive overlap") __success __log_level(2) __flag(BPF_F_TEST_REG_INVARIANTS) -__msg("3: (2d) if r0 > r1 {{.*}} R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))") +__msg("3: (2d) if r0 > r1 {{.*}} R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))") __retval(0) __naked void bounds_deduct_positive_overlap(void) {
@@ -1650,7 +1650,7 @@ l0_%=: r0 = 0; \ SEC("socket") __description("bounds deduction cross sign boundary, two overlaps") __failure __flag(BPF_F_TEST_REG_INVARIANTS) -__msg("3: (2d) if r0 > r1 {{.*}} R0_w=scalar(smin=smin32=-128,smax=smax32=127,umax=0xffffffffffffff80)") +__msg("3: (2d) if r0 > r1 {{.*}} R0=scalar(smin=smin32=-128,smax=smax32=127,umax=0xffffffffffffff80)") __msg("frame pointer is read only") __naked void bounds_deduct_two_overlaps(void) {
tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c +2 -2
@@ -215,7 +215,7 @@ __weak int subprog_untrusted(const volatile struct task_struct *restrict task __ SEC("tp_btf/sys_enter") __success __log_level(2) -__msg("r1 = {{.*}}; {{.*}}R1_w=trusted_ptr_task_struct()") +__msg("r1 = {{.*}}; {{.*}}R1=trusted_ptr_task_struct()") __msg("Func#1 ('subprog_untrusted') is global and assumed valid.") __msg("Validating subprog_untrusted() func#1...")__msg (": R1=untrusted_ptr_task_struct" )
@@ -278,7 +278,7 @@ __weak int subprog_enum_untrusted(enum bpf_attach_type *p __arg_untrusted) SEC("tp_btf/sys_enter") __success __log_level(2) -__msg("r1 = {{.*}}; {{.*}}R1_w=trusted_ptr_task_struct()") +__msg("r1 = {{.*}}; {{.*}}R1=trusted_ptr_task_struct()") __msg("Func#1 ('subprog_void_untrusted') is global and assumed valid.") __msg("Validating subprog_void_untrusted() func#1...")__msg (": R1=rdonly_untrusted_mem(sz=0)" )
tools/testing/selftests/bpf/progs/verifier_ldsx.c +1 -1
@@ -65,7 +65,7 @@ __naked void ldsx_s32(void) SEC("socket") __description("LDSX, S8 range checking, privileged") __log_level(2) __success __retval(1) -__msg("R1_w=scalar(smin=smin32=-128,smax=smax32=127)") +__msg("R1=scalar(smin=smin32=-128,smax=smax32=127)")__naked void ldsx_s8_range_priv (void ) { asm volatile (
tools/testing/selftests/bpf/progs/verifier_precision.c +8 -8
@@ -144,21 +144,21 @@ SEC("?raw_tp") __success __log_level(2) /* * Without the bug fix there will be no history between "last_idx 3 first_idx 3" - * and "parent state regs=" lines. "R0_w=6" parts are here to help anchor + * and "parent state regs=" lines. "R0=6" parts are here to help anchor * expected log messages to the one specific mark_chain_precision operation. * * This is quite fragile: if verifier checkpointing heuristic changes, this * might need adjusting. */ -__msg("2: (07) r0 += 1 ; R0_w=6") +__msg("2: (07) r0 += 1 ; R0=6") __msg("3: (35) if r0 >= 0xa goto pc+1") __msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1") __msg("mark_precise: frame0: regs=r0 stack= before 2: (07) r0 += 1") __msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 1") __msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4") __msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1") -__msg("mark_precise: frame0: parent state regs= stack=: R0_rw=P4") -__msg("3: R0_w=6") +__msg("mark_precise: frame0: parent state regs= stack=: R0=P4") +__msg("3: R0=6")__naked int state_loop_first_last_equal (void ) { asm volatile (
@@ -233,8 +233,8 @@ __naked void bpf_cond_op_not_r10(void) SEC("lsm.s/socket_connect") __success __log_level(2) -__msg("0: (b7) r0 = 1 ; R0_w=1") -__msg("1: (84) w0 = -w0 ; R0_w=0xffffffff") +__msg("0: (b7) r0 = 1 ; R0=1") +__msg("1: (84) w0 = -w0 ; R0=0xffffffff") __msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1") __msg("mark_precise: frame0: regs=r0 stack= before 1: (84) w0 = -w0")__msg ("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1" )
@@ -268,8 +268,8 @@ __naked int bpf_neg_3(void) SEC("lsm.s/socket_connect") __success __log_level(2) -__msg("0: (b7) r0 = 1 ; R0_w=1") -__msg("1: (87) r0 = -r0 ; R0_w=-1") +__msg("0: (b7) r0 = 1 ; R0=1") +__msg("1: (87) r0 = -r0 ; R0=-1") __msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1") __msg("mark_precise: frame0: regs=r0 stack= before 1: (87) r0 = -r0")__msg ("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1" )
tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +5 -5
@@ -353,7 +353,7 @@ __flag(BPF_F_TEST_STATE_FREQ)* collect_linked_regs() can't tie more than 6 registers for a single insn. */ __msg("8: (25) if r0 > 0x7 goto pc+0 ; R0=scalar(id=1") -__msg ("9: (bf) r6 = r6 ; R6_w=scalar(id=2" ) +__msg ("9: (bf) r6 = r6 ; R6=scalar(id=2" ) /* check that r{0-5} are marked precise after 'if' */ __msg("frame0: regs=r0 stack= before 8: (25) if r0 > 0x7 goto pc+0")__msg ("frame0: parent state regs=r0,r1,r2,r3,r4,r5 stack=:" )
@@ -779,12 +779,12 @@ __success__retval (0 ) /* Check that verifier believes r1/r0 are zero at exit */ __log_level(2) -__msg("4: (77) r1 >>= 32 ; R1_w=0") -__msg("5: (bf) r0 = r1 ; R0_w=0 R1_w=0") +__msg("4: (77) r1 >>= 32 ; R1=0") +__msg("5: (bf) r0 = r1 ; R0=0 R1=0") __msg("6: (95) exit") __msg("from 3 to 4") -__msg("4: (77) r1 >>= 32 ; R1_w=0") -__msg("5: (bf) r0 = r1 ; R0_w=0 R1_w=0") +__msg("4: (77) r1 >>= 32 ; R1=0") +__msg("5: (bf) r0 = r1 ; R0=0 R1=0")__msg ("6: (95) exit" )/* Verify that statements to randomize upper half of r1 had not been * generated.
tools/testing/selftests/bpf/progs/verifier_spill_fill.c +20 -20
@@ -506,17 +506,17 @@ SEC("raw_tp") __log_level(2) __success /* fp-8 is spilled IMPRECISE value zero (represented by a zero value fake reg) */ -__msg("2: (7a) *(u64 *)(r10 -8) = 0 ; R10=fp0 fp-8_w=0") +__msg("2: (7a) *(u64 *)(r10 -8) = 0 ; R10=fp0 fp-8=0")/* but fp-16 is spilled IMPRECISE zero const reg */ -__msg ("4: (7b) *(u64 *)(r10 -16) = r0 ; R0_w=0 R10=fp0 fp-16_w=0" ) +__msg ("4: (7b) *(u64 *)(r10 -16) = r0 ; R0=0 R10=fp0 fp-16=0" )/* validate that assigning R2 from STACK_SPILL with zero value doesn't mark register * precise immediately; if necessary, it will be marked precise later */ -__msg ("6: (71) r2 = *(u8 *)(r10 -1) ; R2_w=0 R10=fp0 fp-8_w=0" ) +__msg ("6: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0" )/* similarly, when R2 is assigned from spilled register, it is initially * imprecise, but will be marked precise later once it is used in precise context */ -__msg("10: (71) r2 = *(u8 *)(r10 -9) ; R2_w=0 R10=fp0 fp-16_w=0") +__msg("10: (71) r2 = *(u8 *)(r10 -9) ; R2=0 R10=fp0 fp-16=0") __msg("11: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")__msg ("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)" )
@@ -598,7 +598,7 @@ __log_level(2) __success /* fp-4 is STACK_ZERO */ __msg("2: (62) *(u32 *)(r10 -4) = 0 ; R10=fp0 fp-8=0000????") -__msg("4: (71) r2 = *(u8 *)(r10 -1) ; R2_w=0 R10=fp0 fp-8=0000????") +__msg("4: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0000????") __msg("5: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")__msg ("mark_precise: frame0: regs=r2 stack= before 4: (71) r2 = *(u8 *)(r10 -1)" )
@@ -640,25 +640,25 @@ SEC("raw_tp") __log_level(2) __flag(BPF_F_TEST_STATE_FREQ) __success /* make sure fp-8 is IMPRECISE fake register spill */ -__msg ("3: (7a) *(u64 *)(r10 -8) = 1 ; R10=fp0 fp-8_w=1" ) +__msg ("3: (7a) *(u64 *)(r10 -8) = 1 ; R10=fp0 fp-8=1" ) /* and fp-16 is spilled IMPRECISE const reg */ -__msg ("5: (7b) *(u64 *)(r10 -16) = r0 ; R0_w=1 R10=fp0 fp-16_w=1" ) +__msg ("5: (7b) *(u64 *)(r10 -16) = r0 ; R0=1 R10=fp0 fp-16=1" ) /* validate load from fp-8, which was initialized using BPF_ST_MEM */ -__msg("8: (79) r2 = *(u64 *)(r10 -8) ; R2_w=1 R10=fp0 fp-8=1") +__msg("8: (79) r2 = *(u64 *)(r10 -8) ; R2=1 R10=fp0 fp-8=1") __msg("9: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1") __msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")__msg ("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6" ) /* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */ -__msg("mark_precise: frame0: parent state regs= stack=-8: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_rw=P1 fp-16_w=1") +__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0") __msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0") __msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1") __msg("mark_precise: frame0: regs= stack=-8 before 3: (7a) *(u64 *)(r10 -8) = 1") -__msg ("10: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1" ) +__msg ("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1" ) /* validate load from fp-16, which was initialized using BPF_STX_MEM */ -__msg("12: (79) r2 = *(u64 *)(r10 -16) ; R2_w=1 R10=fp0 fp-16=1") +__msg("12: (79) r2 = *(u64 *)(r10 -16) ; R2=1 R10=fp0 fp-16=1") __msg("13: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")__msg ("mark_precise: frame0: regs=r2 stack= before 12: (79) r2 = *(u64 *)(r10 -16)" )
@@ -668,12 +668,12 @@ __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2") __msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")__msg ("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6" ) /* now both fp-8 and fp-16 are precise, very good */ -__msg("mark_precise: frame0: parent state regs= stack=-16: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_rw=P1 fp-16_rw=P1") +__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0") __msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0") __msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1") -__msg("14: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1") +__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")__naked void stack_load_preserves_const_precision (void ) { asm volatile (
@@ -719,22 +719,22 @@ __success/* make sure fp-8 is 32-bit FAKE subregister spill */ __msg ("3: (62) *(u32 *)(r10 -8) = 1 ; R10=fp0 fp-8=????1" ) /* but fp-16 is spilled IMPRECISE zero const reg */ -__msg ("5: (63) *(u32 *)(r10 -16) = r0 ; R0_w=1 R10=fp0 fp-16=????1" ) +__msg ("5: (63) *(u32 *)(r10 -16) = r0 ; R0=1 R10=fp0 fp-16=????1" ) /* validate load from fp-8, which was initialized using BPF_ST_MEM */ -__msg("8: (61) r2 = *(u32 *)(r10 -8) ; R2_w=1 R10=fp0 fp-8=????1") +__msg("8: (61) r2 = *(u32 *)(r10 -8) ; R2=1 R10=fp0 fp-8=????1") __msg("9: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1") __msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)") __msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6") -__msg("mark_precise: frame0: parent state regs= stack=-8: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_r=????P1 fp-16=????1") +__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0") __msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0") __msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1") __msg("mark_precise: frame0: regs= stack=-8 before 3: (62) *(u32 *)(r10 -8) = 1") -__msg ("10: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1" ) +__msg ("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1" ) /* validate load from fp-16, which was initialized using BPF_STX_MEM */ -__msg("12: (61) r2 = *(u32 *)(r10 -16) ; R2_w=1 R10=fp0 fp-16=????1") +__msg("12: (61) r2 = *(u32 *)(r10 -16) ; R2=1 R10=fp0 fp-16=????1") __msg("13: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")__msg ("mark_precise: frame0: regs=r2 stack= before 12: (61) r2 = *(u32 *)(r10 -16)" )
@@ -743,12 +743,12 @@ __msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2 __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2") __msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)") __msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6") -__msg("mark_precise: frame0: parent state regs= stack=-16: R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_r=????P1 fp-16_r=????P1") +__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0") __msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0") __msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1") -__msg("14: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1") +__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")__naked void stack_load_preserves_const_precision_subreg (void ) { asm volatile (
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +3 -3
@@ -105,7 +105,7 @@ __msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4") __msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3") __msg("mark_precise: frame0: regs=r0 stack= before 10: (95) exit") __msg("mark_precise: frame1: regs=r0 stack= before 9: (bf) r0 = (s8)r10") -__msg("7: R0_w=scalar") +__msg("7: R0=scalar")__naked int fp_precise_subprog_result (void ) { asm volatile (
@@ -141,7 +141,7 @@ __msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = (s8)r1")* anyways, at which point we'll break precision chain */ __msg("mark_precise: frame1: regs=r1 stack= before 9: (bf) r1 = r10") -__msg("7: R0_w=scalar") +__msg("7: R0=scalar")__naked int sneaky_fp_precise_subprog_result (void ) { asm volatile (
@@ -681,7 +681,7 @@ __msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1") __msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8") __msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4") __msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)") -__msg("mark_precise: frame0: parent state regs= stack=-8: R0_w=2 R6_w=1 R8_rw=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8_rw=P1") +__msg("mark_precise: frame0: parent state regs= stack=-8: R0=2 R6=1 R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1") __msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")__msg ("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2" )
tools/testing/selftests/bpf/verifier/bpf_st_mem.c +2 -2
@@ -93,7 +93,7 @@ .expected_attach_type = BPF_SK_LOOKUP, .result = VERBOSE_ACCEPT, .runs = -1, - .errstr ="0: (7a) *(u64 *)(r10 -8) = -44 ; R10=fp0 fp-8_w=-44\ + .errstr ="0: (7a) *(u64 *)(r10 -8) = -44 ; R10=fp0 fp-8=-44\ 2: (c5) if r0 s< 0x0 goto pc+2\ - R0_w=-44", + R0=-44", },
Signature · 1 line
2.47.3
kernel test robot
build botRe: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
kernel test robot noticed the following build warnings:
Hi Eduard,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness config: x86_64-buildonly-randconfig-003-20250911 (https://download.01.org/0day-ci/archive/20250911/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250911/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> kernel/bpf/verifier.c:19305:11: warning: variable 'err' is uninitialized when used here [-Wuninitialized]
19305 | err = err ? : push_jmp_history(env, cur, 0, 0);
| ^~~
kernel/bpf/verifier.c:19140:12: note: initialize the variable 'err' to silence this warning
19140 | int n, err, states_cnt = 0;
| ^
| = 01 warning generated.
vim +/err +19305 kernel/bpf/verifier.c
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19133
58e2af8b3a6b58 Jakub Kicinski 2016-09-21 19134 static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19135 {
58e2af8b3a6b58 Jakub Kicinski 2016-09-21 19136 struct bpf_verifier_state_list *new_sl;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19137 struct bpf_verifier_state_list *sl;
c9e31900b54cad Eduard Zingerman 2025-06-11 19138 struct bpf_verifier_state *cur = env->cur_state, *new;
c9e31900b54cad Eduard Zingerman 2025-06-11 19139 bool force_new_state, add_new_state, loop;
d5c95ed86213e4 Eduard Zingerman 2025-09-10 19140 int n, err, states_cnt = 0;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19141 struct list_head *pos, *tmp, *head;
aa30eb3260b2de Eduard Zingerman 2024-10-29 19142
aa30eb3260b2de Eduard Zingerman 2024-10-29 19143 force_new_state = env->test_state_freq || is_force_checkpoint(env, insn_idx) ||
aa30eb3260b2de Eduard Zingerman 2024-10-29 19144 /* Avoid accumulating infinitely long jmp history */
baaebe0928bf32 Eduard Zingerman 2025-06-11 19145 cur->jmp_history_cnt > 40;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19146
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19147 /* bpf progs typically have pruning point every 4 instructions
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19148 * http://vger.kernel.org/bpfconf2019.html#session-1
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19149 * Do not add new state for future pruning if the verifier hasn't seen
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19150 * at least 2 jumps and at least 8 instructions.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19151 * This heuristics helps decrease 'total_states' and 'peak_states' metric.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19152 * In tests that amounts to up to 50% reduction into total verifier
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19153 * memory consumption and 20% verifier time speedup.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19154 */
aa30eb3260b2de Eduard Zingerman 2024-10-29 19155 add_new_state = force_new_state;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19156 if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19157 env->insn_processed - env->prev_insn_processed >= 8)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19158 add_new_state = true;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19159
9242b5f5615c82 Alexei Starovoitov 2018-12-13 19160 clean_live_states(env, insn_idx, cur);
9242b5f5615c82 Alexei Starovoitov 2018-12-13 19161
c9e31900b54cad Eduard Zingerman 2025-06-11 19162 loop = false;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19163 head = explored_state(env, insn_idx);
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19164 list_for_each_safe(pos, tmp, head) {
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19165 sl = container_of(pos, struct bpf_verifier_state_list, node);
dc2a4ebc0b44a2 Alexei Starovoitov 2019-05-21 19166 states_cnt++;
dc2a4ebc0b44a2 Alexei Starovoitov 2019-05-21 19167 if (sl->state.insn_idx != insn_idx)
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19168 continue;
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19169
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19170 if (sl->state.branches) {
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19171 struct bpf_func_state *frame = sl->state.frame[sl->state.curframe];
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19172
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19173 if (frame->in_async_callback_fn &&
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19174 frame->async_entry_cnt != cur->frame[cur->curframe]->async_entry_cnt) {
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19175 /* Different async_entry_cnt means that the verifier is
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19176 * processing another entry into async callback.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19177 * Seeing the same state is not an indication of infinite
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19178 * loop or infinite recursion.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19179 * But finding the same state doesn't mean that it's safe
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19180 * to stop processing the current state. The previous state
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19181 * hasn't yet reached bpf_exit, since state.branches > 0.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19182 * Checking in_async_callback_fn alone is not enough either.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19183 * Since the verifier still needs to catch infinite loops
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19184 * inside async callbacks.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19185 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19186 goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19187 }
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19188 /* BPF open-coded iterators loop detection is special.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19189 * states_maybe_looping() logic is too simplistic in detecting
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19190 * states that *might* be equivalent, because it doesn't know
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19191 * about ID remapping, so don't even perform it.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19192 * See process_iter_next_call() and iter_active_depths_differ()
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19193 * for overview of the logic. When current and one of parent
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19194 * states are detected as equivalent, it's a good thing: we prove
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19195 * convergence and can stop simulating further iterations.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19196 * It's safe to assume that iterator loop will finish, taking into
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19197 * account iter_next() contract of eventually returning
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19198 * sticky NULL result.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19199 *
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19200 * Note, that states have to be compared exactly in this case because
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19201 * read and precision marks might not be finalized inside the loop.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19202 * E.g. as in the program below:
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19203 *
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19204 * 1. r7 = -16
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19205 * 2. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19206 * 3. while (bpf_iter_num_next(&fp[-8])) {
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19207 * 4. if (r6 != 42) {
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19208 * 5. r7 = -32
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19209 * 6. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19210 * 7. continue
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19211 * 8. }
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19212 * 9. r0 = r10
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19213 * 10. r0 += r7
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19214 * 11. r8 = *(u64 *)(r0 + 0)
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19215 * 12. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19216 * 13. }
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19217 *
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19218 * Here verifier would first visit path 1-3, create a checkpoint at 3
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19219 * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19220 * not have read or precision mark for r7 yet, thus inexact states
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19221 * comparison would discard current state with r7=-32
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19222 * => unsafe memory access at 11 would not be caught.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19223 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19224 if (is_iter_next_insn(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov 2024-03-05 19225 if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19226 struct bpf_func_state *cur_frame;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19227 struct bpf_reg_state *iter_state, *iter_reg;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19228 int spi;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19229
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19230 cur_frame = cur->frame[cur->curframe];
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19231 /* btf_check_iter_kfuncs() enforces that
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19232 * iter state pointer is always the first arg
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19233 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19234 iter_reg = &cur_frame->regs[BPF_REG_1];
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19235 /* current state is valid due to states_equal(),
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19236 * so we can assume valid iter and reg state,
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19237 * no need for extra (re-)validations
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19238 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19239 spi = __get_spi(iter_reg->off + iter_reg->var_off.value);
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19240 iter_state = &func(env, iter_reg)->stack[spi].spilled_ptr;
2a0992829ea386 Eduard Zingerman 2023-10-24 19241 if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19242 loop = true;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19243 goto hit;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19244 }
2a0992829ea386 Eduard Zingerman 2023-10-24 19245 }
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19246 goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19247 }
011832b97b311b Alexei Starovoitov 2024-03-05 19248 if (is_may_goto_insn_at(env, insn_idx)) {
2b2efe1937ca9f Alexei Starovoitov 2024-06-19 19249 if (sl->state.may_goto_depth != cur->may_goto_depth &&
2b2efe1937ca9f Alexei Starovoitov 2024-06-19 19250 states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19251 loop = true;
011832b97b311b Alexei Starovoitov 2024-03-05 19252 goto hit;
011832b97b311b Alexei Starovoitov 2024-03-05 19253 }
011832b97b311b Alexei Starovoitov 2024-03-05 19254 }
588af0c506ec8e Eduard Zingerman 2025-09-10 19255 if (bpf_calls_callback(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov 2024-03-05 19256 if (states_equal(env, &sl->state, cur, RANGE_WITHIN))
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19257 goto hit;
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19258 goto skip_inf_loop_check;
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19259 }
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19260 /* attempt to detect infinite loop to avoid unnecessary doomed work */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19261 if (states_maybe_looping(&sl->state, cur) &&
4f81c16f50baf6 Alexei Starovoitov 2024-03-05 19262 states_equal(env, &sl->state, cur, EXACT) &&
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19263 !iter_active_depths_differ(&sl->state, cur) &&
011832b97b311b Alexei Starovoitov 2024-03-05 19264 sl->state.may_goto_depth == cur->may_goto_depth &&
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19265 sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19266 verbose_linfo(env, insn_idx, "; ");
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19267 verbose(env, "infinite loop detected at insn %d\n", insn_idx);
b4d8239534fddc Eduard Zingerman 2023-10-24 19268 verbose(env, "cur state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03 19269 print_verifier_state(env, cur, cur->curframe, true);
b4d8239534fddc Eduard Zingerman 2023-10-24 19270 verbose(env, "old state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03 19271 print_verifier_state(env, &sl->state, cur->curframe, true);
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19272 return -EINVAL;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19273 }
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19274 /* if the verifier is processing a loop, avoid adding new state
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19275 * too often, since different loop iterations have distinct
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19276 * states and may not help future pruning.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19277 * This threshold shouldn't be too low to make sure that
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19278 * a loop with large bound will be rejected quickly.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19279 * The most abusive loop will be:
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19280 * r1 += 1
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19281 * if r1 < 1000000 goto pc-2
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19282 * 1M insn_procssed limit / 100 == 10k peak states.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19283 * This threshold shouldn't be too high either, since states
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19284 * at the end of the loop are likely to be useful in pruning.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19285 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19286 skip_inf_loop_check:
4b5ce570dbef57 Andrii Nakryiko 2023-03-09 19287 if (!force_new_state &&
98ddcf389d1bb7 Andrii Nakryiko 2023-03-02 19288 env->jmps_processed - env->prev_jmps_processed < 20 &&
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19289 env->insn_processed - env->prev_insn_processed < 100)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19290 add_new_state = false;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19291 goto miss;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19292 }
c9e31900b54cad Eduard Zingerman 2025-06-11 19293 /* See comments for mark_all_regs_read_and_precise() */
c9e31900b54cad Eduard Zingerman 2025-06-11 19294 loop = incomplete_read_marks(env, &sl->state);
c9e31900b54cad Eduard Zingerman 2025-06-11 19295 if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19296 hit:
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19297 sl->hit_cnt++;
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19298
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19299 /* if previous state reached the exit with precision and
a7de265cb2d849 Rafael Passos 2024-04-17 19300 * current state is equivalent to it (except precision marks)
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19301 * the precision needs to be propagated back in
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19302 * the current state.
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19303 */
41f6f64e6999a8 Andrii Nakryiko 2023-12-05 19304 if (is_jmp_point(env, env->insn_idx))
baaebe0928bf32 Eduard Zingerman 2025-06-11 @19305 err = err ? : push_jmp_history(env, cur, 0, 0);
23b37d616565c8 Eduard Zingerman 2025-06-11 19306 err = err ? : propagate_precision(env, &sl->state, cur, NULL);
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19307 if (err)
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19308 return err;
c9e31900b54cad Eduard Zingerman 2025-06-11 19309 /* When processing iterator based loops above propagate_liveness and
c9e31900b54cad Eduard Zingerman 2025-06-11 19310 * propagate_precision calls are not sufficient to transfer all relevant
c9e31900b54cad Eduard Zingerman 2025-06-11 19311 * read and precision marks. E.g. consider the following case:
c9e31900b54cad Eduard Zingerman 2025-06-11 19312 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19313 * .-> A --. Assume the states are visited in the order A, B, C.
c9e31900b54cad Eduard Zingerman 2025-06-11 19314 * | | | Assume that state B reaches a state equivalent to state A.
c9e31900b54cad Eduard Zingerman 2025-06-11 19315 * | v v At this point, state C is not processed yet, so state A
c9e31900b54cad Eduard Zingerman 2025-06-11 19316 * '-- B C has not received any read or precision marks from C.
c9e31900b54cad Eduard Zingerman 2025-06-11 19317 * Thus, marks propagated from A to B are incomplete.
c9e31900b54cad Eduard Zingerman 2025-06-11 19318 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19319 * The verifier mitigates this by performing the following steps:
c9e31900b54cad Eduard Zingerman 2025-06-11 19320 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19321 * - Prior to the main verification pass, strongly connected components
c9e31900b54cad Eduard Zingerman 2025-06-11 19322 * (SCCs) are computed over the program's control flow graph,
c9e31900b54cad Eduard Zingerman 2025-06-11 19323 * intraprocedurally.
c9e31900b54cad Eduard Zingerman 2025-06-11 19324 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19325 * - During the main verification pass, `maybe_enter_scc()` checks
c9e31900b54cad Eduard Zingerman 2025-06-11 19326 * whether the current verifier state is entering an SCC. If so, an
c9e31900b54cad Eduard Zingerman 2025-06-11 19327 * instance of a `bpf_scc_visit` object is created, and the state
c9e31900b54cad Eduard Zingerman 2025-06-11 19328 * entering the SCC is recorded as the entry state.
c9e31900b54cad Eduard Zingerman 2025-06-11 19329 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19330 * - This instance is associated not with the SCC itself, but with a
c9e31900b54cad Eduard Zingerman 2025-06-11 19331 * `bpf_scc_callchain`: a tuple consisting of the call sites leading to
c9e31900b54cad Eduard Zingerman 2025-06-11 19332 * the SCC and the SCC id. See `compute_scc_callchain()`.
c9e31900b54cad Eduard Zingerman 2025-06-11 19333 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19334 * - When a verification path encounters a `states_equal(...,
c9e31900b54cad Eduard Zingerman 2025-06-11 19335 * RANGE_WITHIN)` condition, there exists a call chain describing the
c9e31900b54cad Eduard Zingerman 2025-06-11 19336 * current state and a corresponding `bpf_scc_visit` instance. A copy
c9e31900b54cad Eduard Zingerman 2025-06-11 19337 * of the current state is created and added to
c9e31900b54cad Eduard Zingerman 2025-06-11 19338 * `bpf_scc_visit->backedges`.
c9e31900b54cad Eduard Zingerman 2025-06-11 19339 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19340 * - When a verification path terminates, `maybe_exit_scc()` is called
c9e31900b54cad Eduard Zingerman 2025-06-11 19341 * from `update_branch_counts()`. For states with `branches == 0`, it
c9e31900b54cad Eduard Zingerman 2025-06-11 19342 * checks whether the state is the entry state of any `bpf_scc_visit`
c9e31900b54cad Eduard Zingerman 2025-06-11 19343 * instance. If it is, this indicates that all paths originating from
c9e31900b54cad Eduard Zingerman 2025-06-11 19344 * this SCC visit have been explored. `propagate_backedges()` is then
c9e31900b54cad Eduard Zingerman 2025-06-11 19345 * called, which propagates read and precision marks through the
c9e31900b54cad Eduard Zingerman 2025-06-11 19346 * backedges until a fixed point is reached.
c9e31900b54cad Eduard Zingerman 2025-06-11 19347 * (In the earlier example, this would propagate marks from A to B,
c9e31900b54cad Eduard Zingerman 2025-06-11 19348 * from C to A, and then again from A to B.)
c9e31900b54cad Eduard Zingerman 2025-06-11 19349 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19350 * A note on callchains
c9e31900b54cad Eduard Zingerman 2025-06-11 19351 * --------------------
c9e31900b54cad Eduard Zingerman 2025-06-11 19352 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19353 * Consider the following example:
c9e31900b54cad Eduard Zingerman 2025-06-11 19354 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19355 * void foo() { loop { ... SCC#1 ... } }
c9e31900b54cad Eduard Zingerman 2025-06-11 19356 * void main() {
c9e31900b54cad Eduard Zingerman 2025-06-11 19357 * A: foo();
c9e31900b54cad Eduard Zingerman 2025-06-11 19358 * B: ...
c9e31900b54cad Eduard Zingerman 2025-06-11 19359 * C: foo();
c9e31900b54cad Eduard Zingerman 2025-06-11 19360 * }
c9e31900b54cad Eduard Zingerman 2025-06-11 19361 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19362 * Here, there are two distinct callchains leading to SCC#1:
c9e31900b54cad Eduard Zingerman 2025-06-11 19363 * - (A, SCC#1)
c9e31900b54cad Eduard Zingerman 2025-06-11 19364 * - (C, SCC#1)
c9e31900b54cad Eduard Zingerman 2025-06-11 19365 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19366 * Each callchain identifies a separate `bpf_scc_visit` instance that
c9e31900b54cad Eduard Zingerman 2025-06-11 19367 * accumulates backedge states. The `propagate_{liveness,precision}()`
c9e31900b54cad Eduard Zingerman 2025-06-11 19368 * functions traverse the parent state of each backedge state, which
c9e31900b54cad Eduard Zingerman 2025-06-11 19369 * means these parent states must remain valid (i.e., not freed) while
c9e31900b54cad Eduard Zingerman 2025-06-11 19370 * the corresponding `bpf_scc_visit` instance exists.
c9e31900b54cad Eduard Zingerman 2025-06-11 19371 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19372 * Associating `bpf_scc_visit` instances directly with SCCs instead of
c9e31900b54cad Eduard Zingerman 2025-06-11 19373 * callchains would break this invariant:
c9e31900b54cad Eduard Zingerman 2025-06-11 19374 * - States explored during `C: foo()` would contribute backedges to
c9e31900b54cad Eduard Zingerman 2025-06-11 19375 * SCC#1, but SCC#1 would only be exited once the exploration of
c9e31900b54cad Eduard Zingerman 2025-06-11 19376 * `A: foo()` completes.
c9e31900b54cad Eduard Zingerman 2025-06-11 19377 * - By that time, the states explored between `A: foo()` and `C: foo()`
c9e31900b54cad Eduard Zingerman 2025-06-11 19378 * (i.e., `B: ...`) may have already been freed, causing the parent
c9e31900b54cad Eduard Zingerman 2025-06-11 19379 * links for states from `C: foo()` to become invalid.
c9e31900b54cad Eduard Zingerman 2025-06-11 19380 */
c9e31900b54cad Eduard Zingerman 2025-06-11 19381 if (loop) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19382 struct bpf_scc_backedge *backedge;
c9e31900b54cad Eduard Zingerman 2025-06-11 19383
43736ec3e02789 Eduard Zingerman 2025-06-13 19384 backedge = kzalloc(sizeof(*backedge), GFP_KERNEL_ACCOUNT);
c9e31900b54cad Eduard Zingerman 2025-06-11 19385 if (!backedge)
c9e31900b54cad Eduard Zingerman 2025-06-11 19386 return -ENOMEM;
c9e31900b54cad Eduard Zingerman 2025-06-11 19387 err = copy_verifier_state(&backedge->state, cur);
c9e31900b54cad Eduard Zingerman 2025-06-11 19388 backedge->state.equal_state = &sl->state;
c9e31900b54cad Eduard Zingerman 2025-06-11 19389 backedge->state.insn_idx = insn_idx;
c9e31900b54cad Eduard Zingerman 2025-06-11 19390 err = err ?: add_scc_backedge(env, &sl->state, backedge);
c9e31900b54cad Eduard Zingerman 2025-06-11 19391 if (err) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19392 free_verifier_state(&backedge->state, false);
bf0c2a84df9fb0 Qianfeng Rong 2025-08-11 19393 kfree(backedge);
c9e31900b54cad Eduard Zingerman 2025-06-11 19394 return err;
c9e31900b54cad Eduard Zingerman 2025-06-11 19395 }
c9e31900b54cad Eduard Zingerman 2025-06-11 19396 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19397 return 1;
dc503a8ad98474 Edward Cree 2017-08-15 19398 }
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19399 miss:
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19400 /* when new state is not going to be added do not increase miss count.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19401 * Otherwise several loop iterations will remove the state
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19402 * recorded earlier. The goal of these heuristics is to have
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19403 * states from some iterations of the loop (some in the beginning
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19404 * and some at the end) to help pruning.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19405 */
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19406 if (add_new_state)
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19407 sl->miss_cnt++;
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19408 /* heuristic to determine whether this state is beneficial
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19409 * to keep checking from state equivalence point of view.
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19410 * Higher numbers increase max_states_per_insn and verification time,
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19411 * but do not meaningfully decrease insn_processed.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19412 * 'n' controls how many times state could miss before eviction.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19413 * Use bigger 'n' for checkpoints because evicting checkpoint states
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19414 * too early would hinder iterator convergence.
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19415 */
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19416 n = is_force_checkpoint(env, insn_idx) && sl->state.branches > 0 ? 64 : 3;
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19417 if (sl->miss_cnt > sl->hit_cnt * n + n) {
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19418 /* the state is unlikely to be useful. Remove it to
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19419 * speed up verification
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19420 */
408fcf946b2bad Eduard Zingerman 2025-02-15 19421 sl->in_free_list = true;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19422 list_del(&sl->node);
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19423 list_add(&sl->node, &env->free_list);
574078b001cdf6 Eduard Zingerman 2025-02-15 19424 env->free_list_size++;
574078b001cdf6 Eduard Zingerman 2025-02-15 19425 env->explored_states_size--;
408fcf946b2bad Eduard Zingerman 2025-02-15 19426 maybe_free_verifier_state(env, sl);
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19427 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19428 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19429
06ee7115b0d174 Alexei Starovoitov 2019-04-01 19430 if (env->max_states_per_insn < states_cnt)
06ee7115b0d174 Alexei Starovoitov 2019-04-01 19431 env->max_states_per_insn = states_cnt;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19432
2c78ee898d8f10 Alexei Starovoitov 2020-05-13 19433 if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
a095f421057e22 Andrii Nakryiko 2022-12-06 19434 return 0;
ceefbc96fa5c5b Alexei Starovoitov 2018-12-03 19435
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19436 if (!add_new_state)
a095f421057e22 Andrii Nakryiko 2022-12-06 19437 return 0;
ceefbc96fa5c5b Alexei Starovoitov 2018-12-03 19438
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19439 /* There were no equivalent states, remember the current one.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19440 * Technically the current state is not proven to be safe yet,
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19441 * but it will either reach outer most bpf_exit (which means it's safe)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19442 * or it will be rejected. When there are no loops the verifier won't be
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19443 * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19444 * again on the way to bpf_exit.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19445 * When looping the sl->state.branches will be > 0 and this state
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19446 * will not be considered for equivalence until branches == 0.
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19447 */
43736ec3e02789 Eduard Zingerman 2025-06-13 19448 new_sl = kzalloc(sizeof(struct bpf_verifier_state_list), GFP_KERNEL_ACCOUNT);
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19449 if (!new_sl)
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19450 return -ENOMEM;
06ee7115b0d174 Alexei Starovoitov 2019-04-01 19451 env->total_states++;
574078b001cdf6 Eduard Zingerman 2025-02-15 19452 env->explored_states_size++;
574078b001cdf6 Eduard Zingerman 2025-02-15 19453 update_peak_states(env);
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19454 env->prev_jmps_processed = env->jmps_processed;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19455 env->prev_insn_processed = env->insn_processed;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19456
7a830b53c17bba Andrii Nakryiko 2022-11-04 19457 /* forget precise markings we inherited, see __mark_chain_precision */
7a830b53c17bba Andrii Nakryiko 2022-11-04 19458 if (env->bpf_capable)
7a830b53c17bba Andrii Nakryiko 2022-11-04 19459 mark_all_scalars_imprecise(env, cur);
7a830b53c17bba Andrii Nakryiko 2022-11-04 19460
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19461 /* add new state to the head of linked list */
679c782de14bd4 Edward Cree 2018-08-22 19462 new = &new_sl->state;
679c782de14bd4 Edward Cree 2018-08-22 19463 err = copy_verifier_state(new, cur);
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19464 if (err) {
679c782de14bd4 Edward Cree 2018-08-22 19465 free_verifier_state(new, false);
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19466 kfree(new_sl);
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19467 return err;
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19468 }
dc2a4ebc0b44a2 Alexei Starovoitov 2019-05-21 19469 new->insn_idx = insn_idx;
0df1a55afa832f Paul Chaignon 2025-07-01 19470 verifier_bug_if(new->branches != 1, env,
0df1a55afa832f Paul Chaignon 2025-07-01 19471 "%s:branches_to_explore=%d insn %d",
0df1a55afa832f Paul Chaignon 2025-07-01 19472 __func__, new->branches, insn_idx);
c9e31900b54cad Eduard Zingerman 2025-06-11 19473 err = maybe_enter_scc(env, new);
c9e31900b54cad Eduard Zingerman 2025-06-11 19474 if (err) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19475 free_verifier_state(new, false);
e4980fa6463624 Feng Yang 2025-08-27 19476 kfree(new_sl);
c9e31900b54cad Eduard Zingerman 2025-06-11 19477 return err;
c9e31900b54cad Eduard Zingerman 2025-06-11 19478 }
b5dc0163d8fd78 Alexei Starovoitov 2019-06-15 19479
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19480 cur->parent = new;
b5dc0163d8fd78 Alexei Starovoitov 2019-06-15 19481 cur->first_insn_idx = insn_idx;
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19482 cur->dfs_depth = new->dfs_depth + 1;
baaebe0928bf32 Eduard Zingerman 2025-06-11 19483 clear_jmp_history(cur);
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19484 list_add(&new_sl->node, head);
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19485 return 0;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19486 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19487 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Eduard Zingerman
maintainerRe: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
Quoted 28 lines · kernel test robot
Hi Eduard,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness config: x86_64-buildonly-randconfig-003-20250911 (https://download.01.org/0day-ci/archive/20250911/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250911/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
Quoted 2 lines
All warnings (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
kernel/bpf/verifier.c:19305:11: warning: variable 'err' is uninitialized when used here [-Wuninitialized]19305 | err = err ? : push_jmp_history(env, cur, 0, 0); | ^~~ kernel/bpf/verifier.c:19140:12: note: initialize the variable 'err' to silence this warning 19140 | int n, err, states_cnt = 0; | ^ | = 0
Quoted 4 lines
1 warning generated.
vim +/err +19305
kernel/bpf/verifier.c
This was sloppy on my side, should look as follows:
kernel/bpf/verifier.c +6 -3
@@ -19297,9 +19297,12 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)* the precision needs to be propagated back in * the current state. */ -if (is_jmp_point (env, env->insn_idx)) - err = err ? :push_jmp_history (env, cur,0 ,0 ); - err = err ? :propagate_precision (env, &sl->state, cur,NULL ); +if (is_jmp_point (env, env->insn_idx)) { + err =push_jmp_history (env, cur,0 ,0 ); +if (err) +return err; + } + err =propagate_precision (env, &sl->state, cur,NULL );if (err)return err;/* When processing iterator based loops above propagate_liveness and
[...]
Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
Quoted 42 lines · Eduard Zingerman
Quoted 28 lines · kernel test robot
Hi Eduard,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness config: x86_64-buildonly-randconfig-003-20250911 (https://download.01.org/0day-ci/archive/20250911/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250911/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
Quoted 2 lines
All warnings (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
kernel/bpf/verifier.c:19305:11: warning: variable 'err' is uninitialized when used here [-Wuninitialized]19305 | err = err ? : push_jmp_history(env, cur, 0, 0); | ^~~ kernel/bpf/verifier.c:19140:12: note: initialize the variable 'err' to silence this warning 19140 | int n, err, states_cnt = 0; | ^ | = 0Quoted 4 lines
1 warning generated.
vim +/err +19305
kernel/bpf/verifier.cThis was sloppy on my side, should look as follows:
kernel/bpf/verifier.c +0
-0@@ -19297,9 +19297,12 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)* the precision needs to be propagated back in * the current state.
kernel/bpf/verifier.c static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) · line 19305
Quoted 10 lines
kernel/bpf/verifier.c +6
-3*/ -if (is_jmp_point (env, env->insn_idx)) - err = err ? :push_jmp_history (env, cur,0 ,0 ); - err = err ? :propagate_precision (env, &sl->state, cur,NULL ); +if (is_jmp_point (env, env->insn_idx)) { + err =push_jmp_history (env, cur,0 ,0 ); +if (err) +return err; + } + err =propagate_precision (env, &sl->state, cur,NULL );
hmm. init err=0 instead and avoid explicit if (err)return err ?
Eduard Zingerman
maintainerRe: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
On Thu, 2025-09-11 at 15:00 -0700, Alexei Starovoitov wrote:
[...]
Quoted 9 lines
Quoted 9 lines
kernel/bpf/verifier.c +0
-3@@ -19297,9 +19297,12 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)* the precision needs to be propagated back in * the current state. */ -if (is_jmp_point (env, env->insn_idx)) - err = err ? :push_jmp_history (env, cur,0 ,0 ); - err = err ? :propagate_precision (env, &sl->state, cur,NULL );
Quoted 8 lines
kernel/bpf/verifier.c static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) · line 19305
Quoted 6 lines
kernel/bpf/verifier.c +6
-0+if (is_jmp_point (env, env->insn_idx)) { + err =push_jmp_history (env, cur,0 ,0 ); +if (err) +return err; + } + err =propagate_precision (env, &sl->state, cur,NULL );hmm. init err=0 instead and avoid explicit if (err)return err ?
Or like that, yes.
Dan Carpenter
build botRe: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
In reply to Eduard ZingermanBCC: [email protected] CC: [email protected] In-Reply-To: <[email protected]> References: <[email protected]> TO: Eduard Zingerman…
BCC: [email protected]
CC: [email protected]
In-Reply-To: <[email protected]>
References: <[email protected]>
TO: Eduard Zingerman <[email protected]>
TO: [email protected]
TO: [email protected]
TO: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
Hi Eduard,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
:::::: branch date: 18 hours ago
:::::: commit date: 18 hours ago
config: arm-randconfig-r071-20250911 (https://download.01.org/0day-ci/archive/20250912/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 21857ae337e0892a5522b6e7337899caa61de2a6)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Reported-by: Dan Carpenter <[email protected]> | Closes: https://lore.kernel.org/r/[email protected]/
smatch warnings:kernel/bpf/verifier.c:19305 is_state_visited() error: uninitialized symbol 'err'.
vim +/err +19305 kernel/bpf/verifier.c
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19133
58e2af8b3a6b58 Jakub Kicinski 2016-09-21 19134 static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19135 {
58e2af8b3a6b58 Jakub Kicinski 2016-09-21 19136 struct bpf_verifier_state_list *new_sl;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19137 struct bpf_verifier_state_list *sl;
c9e31900b54cad Eduard Zingerman 2025-06-11 19138 struct bpf_verifier_state *cur = env->cur_state, *new;
c9e31900b54cad Eduard Zingerman 2025-06-11 19139 bool force_new_state, add_new_state, loop;
d5c95ed86213e4 Eduard Zingerman 2025-09-10 19140 int n, err, states_cnt = 0;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19141 struct list_head *pos, *tmp, *head;
aa30eb3260b2de Eduard Zingerman 2024-10-29 19142
aa30eb3260b2de Eduard Zingerman 2024-10-29 19143 force_new_state = env->test_state_freq || is_force_checkpoint(env, insn_idx) ||
aa30eb3260b2de Eduard Zingerman 2024-10-29 19144 /* Avoid accumulating infinitely long jmp history */
baaebe0928bf32 Eduard Zingerman 2025-06-11 19145 cur->jmp_history_cnt > 40;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19146
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19147 /* bpf progs typically have pruning point every 4 instructions
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19148 * http://vger.kernel.org/bpfconf2019.html#session-1
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19149 * Do not add new state for future pruning if the verifier hasn't seen
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19150 * at least 2 jumps and at least 8 instructions.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19151 * This heuristics helps decrease 'total_states' and 'peak_states' metric.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19152 * In tests that amounts to up to 50% reduction into total verifier
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19153 * memory consumption and 20% verifier time speedup.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19154 */
aa30eb3260b2de Eduard Zingerman 2024-10-29 19155 add_new_state = force_new_state;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19156 if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19157 env->insn_processed - env->prev_insn_processed >= 8)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19158 add_new_state = true;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19159
9242b5f5615c82 Alexei Starovoitov 2018-12-13 19160 clean_live_states(env, insn_idx, cur);
9242b5f5615c82 Alexei Starovoitov 2018-12-13 19161
c9e31900b54cad Eduard Zingerman 2025-06-11 19162 loop = false;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19163 head = explored_state(env, insn_idx);
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19164 list_for_each_safe(pos, tmp, head) {
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19165 sl = container_of(pos, struct bpf_verifier_state_list, node);
dc2a4ebc0b44a2 Alexei Starovoitov 2019-05-21 19166 states_cnt++;
dc2a4ebc0b44a2 Alexei Starovoitov 2019-05-21 19167 if (sl->state.insn_idx != insn_idx)
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19168 continue;
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19169
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19170 if (sl->state.branches) {
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19171 struct bpf_func_state *frame = sl->state.frame[sl->state.curframe];
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19172
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19173 if (frame->in_async_callback_fn &&
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19174 frame->async_entry_cnt != cur->frame[cur->curframe]->async_entry_cnt) {
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19175 /* Different async_entry_cnt means that the verifier is
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19176 * processing another entry into async callback.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19177 * Seeing the same state is not an indication of infinite
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19178 * loop or infinite recursion.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19179 * But finding the same state doesn't mean that it's safe
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19180 * to stop processing the current state. The previous state
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19181 * hasn't yet reached bpf_exit, since state.branches > 0.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19182 * Checking in_async_callback_fn alone is not enough either.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19183 * Since the verifier still needs to catch infinite loops
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19184 * inside async callbacks.
bfc6bb74e4f16a Alexei Starovoitov 2021-07-14 19185 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19186 goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19187 }
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19188 /* BPF open-coded iterators loop detection is special.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19189 * states_maybe_looping() logic is too simplistic in detecting
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19190 * states that *might* be equivalent, because it doesn't know
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19191 * about ID remapping, so don't even perform it.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19192 * See process_iter_next_call() and iter_active_depths_differ()
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19193 * for overview of the logic. When current and one of parent
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19194 * states are detected as equivalent, it's a good thing: we prove
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19195 * convergence and can stop simulating further iterations.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19196 * It's safe to assume that iterator loop will finish, taking into
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19197 * account iter_next() contract of eventually returning
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19198 * sticky NULL result.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19199 *
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19200 * Note, that states have to be compared exactly in this case because
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19201 * read and precision marks might not be finalized inside the loop.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19202 * E.g. as in the program below:
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19203 *
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19204 * 1. r7 = -16
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19205 * 2. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19206 * 3. while (bpf_iter_num_next(&fp[-8])) {
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19207 * 4. if (r6 != 42) {
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19208 * 5. r7 = -32
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19209 * 6. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19210 * 7. continue
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19211 * 8. }
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19212 * 9. r0 = r10
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19213 * 10. r0 += r7
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19214 * 11. r8 = *(u64 *)(r0 + 0)
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19215 * 12. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19216 * 13. }
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19217 *
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19218 * Here verifier would first visit path 1-3, create a checkpoint at 3
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19219 * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19220 * not have read or precision mark for r7 yet, thus inexact states
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19221 * comparison would discard current state with r7=-32
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19222 * => unsafe memory access at 11 would not be caught.
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19223 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19224 if (is_iter_next_insn(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov 2024-03-05 19225 if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19226 struct bpf_func_state *cur_frame;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19227 struct bpf_reg_state *iter_state, *iter_reg;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19228 int spi;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19229
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19230 cur_frame = cur->frame[cur->curframe];
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19231 /* btf_check_iter_kfuncs() enforces that
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19232 * iter state pointer is always the first arg
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19233 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19234 iter_reg = &cur_frame->regs[BPF_REG_1];
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19235 /* current state is valid due to states_equal(),
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19236 * so we can assume valid iter and reg state,
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19237 * no need for extra (re-)validations
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19238 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19239 spi = __get_spi(iter_reg->off + iter_reg->var_off.value);
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19240 iter_state = &func(env, iter_reg)->stack[spi].spilled_ptr;
2a0992829ea386 Eduard Zingerman 2023-10-24 19241 if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19242 loop = true;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19243 goto hit;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19244 }
2a0992829ea386 Eduard Zingerman 2023-10-24 19245 }
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19246 goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19247 }
011832b97b311b Alexei Starovoitov 2024-03-05 19248 if (is_may_goto_insn_at(env, insn_idx)) {
2b2efe1937ca9f Alexei Starovoitov 2024-06-19 19249 if (sl->state.may_goto_depth != cur->may_goto_depth &&
2b2efe1937ca9f Alexei Starovoitov 2024-06-19 19250 states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19251 loop = true;
011832b97b311b Alexei Starovoitov 2024-03-05 19252 goto hit;
011832b97b311b Alexei Starovoitov 2024-03-05 19253 }
011832b97b311b Alexei Starovoitov 2024-03-05 19254 }
588af0c506ec8e Eduard Zingerman 2025-09-10 19255 if (bpf_calls_callback(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov 2024-03-05 19256 if (states_equal(env, &sl->state, cur, RANGE_WITHIN))
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19257 goto hit;
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19258 goto skip_inf_loop_check;
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19259 }
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19260 /* attempt to detect infinite loop to avoid unnecessary doomed work */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19261 if (states_maybe_looping(&sl->state, cur) &&
4f81c16f50baf6 Alexei Starovoitov 2024-03-05 19262 states_equal(env, &sl->state, cur, EXACT) &&
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19263 !iter_active_depths_differ(&sl->state, cur) &&
011832b97b311b Alexei Starovoitov 2024-03-05 19264 sl->state.may_goto_depth == cur->may_goto_depth &&
ab5cfac139ab85 Eduard Zingerman 2023-11-21 19265 sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19266 verbose_linfo(env, insn_idx, "; ");
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19267 verbose(env, "infinite loop detected at insn %d\n", insn_idx);
b4d8239534fddc Eduard Zingerman 2023-10-24 19268 verbose(env, "cur state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03 19269 print_verifier_state(env, cur, cur->curframe, true);
b4d8239534fddc Eduard Zingerman 2023-10-24 19270 verbose(env, "old state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03 19271 print_verifier_state(env, &sl->state, cur->curframe, true);
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19272 return -EINVAL;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19273 }
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19274 /* if the verifier is processing a loop, avoid adding new state
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19275 * too often, since different loop iterations have distinct
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19276 * states and may not help future pruning.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19277 * This threshold shouldn't be too low to make sure that
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19278 * a loop with large bound will be rejected quickly.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19279 * The most abusive loop will be:
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19280 * r1 += 1
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19281 * if r1 < 1000000 goto pc-2
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19282 * 1M insn_procssed limit / 100 == 10k peak states.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19283 * This threshold shouldn't be too high either, since states
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19284 * at the end of the loop are likely to be useful in pruning.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19285 */
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19286 skip_inf_loop_check:
4b5ce570dbef57 Andrii Nakryiko 2023-03-09 19287 if (!force_new_state &&
98ddcf389d1bb7 Andrii Nakryiko 2023-03-02 19288 env->jmps_processed - env->prev_jmps_processed < 20 &&
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19289 env->insn_processed - env->prev_insn_processed < 100)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19290 add_new_state = false;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19291 goto miss;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19292 }
c9e31900b54cad Eduard Zingerman 2025-06-11 19293 /* See comments for mark_all_regs_read_and_precise() */
c9e31900b54cad Eduard Zingerman 2025-06-11 19294 loop = incomplete_read_marks(env, &sl->state);
c9e31900b54cad Eduard Zingerman 2025-06-11 19295 if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
06accc8779c1d5 Andrii Nakryiko 2023-03-08 19296 hit:
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19297 sl->hit_cnt++;
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19298
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19299 /* if previous state reached the exit with precision and
a7de265cb2d849 Rafael Passos 2024-04-17 19300 * current state is equivalent to it (except precision marks)
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19301 * the precision needs to be propagated back in
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19302 * the current state.
a3ce685dd01a78 Alexei Starovoitov 2019-06-28 19303 */
41f6f64e6999a8 Andrii Nakryiko 2023-12-05 19304 if (is_jmp_point(env, env->insn_idx))
baaebe0928bf32 Eduard Zingerman 2025-06-11 @19305 err = err ? : push_jmp_history(env, cur, 0, 0);
23b37d616565c8 Eduard Zingerman 2025-06-11 19306 err = err ? : propagate_precision(env, &sl->state, cur, NULL);
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19307 if (err)
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19308 return err;
c9e31900b54cad Eduard Zingerman 2025-06-11 19309 /* When processing iterator based loops above propagate_liveness and
c9e31900b54cad Eduard Zingerman 2025-06-11 19310 * propagate_precision calls are not sufficient to transfer all relevant
c9e31900b54cad Eduard Zingerman 2025-06-11 19311 * read and precision marks. E.g. consider the following case:
c9e31900b54cad Eduard Zingerman 2025-06-11 19312 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19313 * .-> A --. Assume the states are visited in the order A, B, C.
c9e31900b54cad Eduard Zingerman 2025-06-11 19314 * | | | Assume that state B reaches a state equivalent to state A.
c9e31900b54cad Eduard Zingerman 2025-06-11 19315 * | v v At this point, state C is not processed yet, so state A
c9e31900b54cad Eduard Zingerman 2025-06-11 19316 * '-- B C has not received any read or precision marks from C.
c9e31900b54cad Eduard Zingerman 2025-06-11 19317 * Thus, marks propagated from A to B are incomplete.
c9e31900b54cad Eduard Zingerman 2025-06-11 19318 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19319 * The verifier mitigates this by performing the following steps:
c9e31900b54cad Eduard Zingerman 2025-06-11 19320 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19321 * - Prior to the main verification pass, strongly connected components
c9e31900b54cad Eduard Zingerman 2025-06-11 19322 * (SCCs) are computed over the program's control flow graph,
c9e31900b54cad Eduard Zingerman 2025-06-11 19323 * intraprocedurally.
c9e31900b54cad Eduard Zingerman 2025-06-11 19324 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19325 * - During the main verification pass, `maybe_enter_scc()` checks
c9e31900b54cad Eduard Zingerman 2025-06-11 19326 * whether the current verifier state is entering an SCC. If so, an
c9e31900b54cad Eduard Zingerman 2025-06-11 19327 * instance of a `bpf_scc_visit` object is created, and the state
c9e31900b54cad Eduard Zingerman 2025-06-11 19328 * entering the SCC is recorded as the entry state.
c9e31900b54cad Eduard Zingerman 2025-06-11 19329 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19330 * - This instance is associated not with the SCC itself, but with a
c9e31900b54cad Eduard Zingerman 2025-06-11 19331 * `bpf_scc_callchain`: a tuple consisting of the call sites leading to
c9e31900b54cad Eduard Zingerman 2025-06-11 19332 * the SCC and the SCC id. See `compute_scc_callchain()`.
c9e31900b54cad Eduard Zingerman 2025-06-11 19333 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19334 * - When a verification path encounters a `states_equal(...,
c9e31900b54cad Eduard Zingerman 2025-06-11 19335 * RANGE_WITHIN)` condition, there exists a call chain describing the
c9e31900b54cad Eduard Zingerman 2025-06-11 19336 * current state and a corresponding `bpf_scc_visit` instance. A copy
c9e31900b54cad Eduard Zingerman 2025-06-11 19337 * of the current state is created and added to
c9e31900b54cad Eduard Zingerman 2025-06-11 19338 * `bpf_scc_visit->backedges`.
c9e31900b54cad Eduard Zingerman 2025-06-11 19339 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19340 * - When a verification path terminates, `maybe_exit_scc()` is called
c9e31900b54cad Eduard Zingerman 2025-06-11 19341 * from `update_branch_counts()`. For states with `branches == 0`, it
c9e31900b54cad Eduard Zingerman 2025-06-11 19342 * checks whether the state is the entry state of any `bpf_scc_visit`
c9e31900b54cad Eduard Zingerman 2025-06-11 19343 * instance. If it is, this indicates that all paths originating from
c9e31900b54cad Eduard Zingerman 2025-06-11 19344 * this SCC visit have been explored. `propagate_backedges()` is then
c9e31900b54cad Eduard Zingerman 2025-06-11 19345 * called, which propagates read and precision marks through the
c9e31900b54cad Eduard Zingerman 2025-06-11 19346 * backedges until a fixed point is reached.
c9e31900b54cad Eduard Zingerman 2025-06-11 19347 * (In the earlier example, this would propagate marks from A to B,
c9e31900b54cad Eduard Zingerman 2025-06-11 19348 * from C to A, and then again from A to B.)
c9e31900b54cad Eduard Zingerman 2025-06-11 19349 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19350 * A note on callchains
c9e31900b54cad Eduard Zingerman 2025-06-11 19351 * --------------------
c9e31900b54cad Eduard Zingerman 2025-06-11 19352 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19353 * Consider the following example:
c9e31900b54cad Eduard Zingerman 2025-06-11 19354 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19355 * void foo() { loop { ... SCC#1 ... } }
c9e31900b54cad Eduard Zingerman 2025-06-11 19356 * void main() {
c9e31900b54cad Eduard Zingerman 2025-06-11 19357 * A: foo();
c9e31900b54cad Eduard Zingerman 2025-06-11 19358 * B: ...
c9e31900b54cad Eduard Zingerman 2025-06-11 19359 * C: foo();
c9e31900b54cad Eduard Zingerman 2025-06-11 19360 * }
c9e31900b54cad Eduard Zingerman 2025-06-11 19361 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19362 * Here, there are two distinct callchains leading to SCC#1:
c9e31900b54cad Eduard Zingerman 2025-06-11 19363 * - (A, SCC#1)
c9e31900b54cad Eduard Zingerman 2025-06-11 19364 * - (C, SCC#1)
c9e31900b54cad Eduard Zingerman 2025-06-11 19365 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19366 * Each callchain identifies a separate `bpf_scc_visit` instance that
c9e31900b54cad Eduard Zingerman 2025-06-11 19367 * accumulates backedge states. The `propagate_{liveness,precision}()`
c9e31900b54cad Eduard Zingerman 2025-06-11 19368 * functions traverse the parent state of each backedge state, which
c9e31900b54cad Eduard Zingerman 2025-06-11 19369 * means these parent states must remain valid (i.e., not freed) while
c9e31900b54cad Eduard Zingerman 2025-06-11 19370 * the corresponding `bpf_scc_visit` instance exists.
c9e31900b54cad Eduard Zingerman 2025-06-11 19371 *
c9e31900b54cad Eduard Zingerman 2025-06-11 19372 * Associating `bpf_scc_visit` instances directly with SCCs instead of
c9e31900b54cad Eduard Zingerman 2025-06-11 19373 * callchains would break this invariant:
c9e31900b54cad Eduard Zingerman 2025-06-11 19374 * - States explored during `C: foo()` would contribute backedges to
c9e31900b54cad Eduard Zingerman 2025-06-11 19375 * SCC#1, but SCC#1 would only be exited once the exploration of
c9e31900b54cad Eduard Zingerman 2025-06-11 19376 * `A: foo()` completes.
c9e31900b54cad Eduard Zingerman 2025-06-11 19377 * - By that time, the states explored between `A: foo()` and `C: foo()`
c9e31900b54cad Eduard Zingerman 2025-06-11 19378 * (i.e., `B: ...`) may have already been freed, causing the parent
c9e31900b54cad Eduard Zingerman 2025-06-11 19379 * links for states from `C: foo()` to become invalid.
c9e31900b54cad Eduard Zingerman 2025-06-11 19380 */
c9e31900b54cad Eduard Zingerman 2025-06-11 19381 if (loop) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19382 struct bpf_scc_backedge *backedge;
c9e31900b54cad Eduard Zingerman 2025-06-11 19383
43736ec3e02789 Eduard Zingerman 2025-06-13 19384 backedge = kzalloc(sizeof(*backedge), GFP_KERNEL_ACCOUNT);
c9e31900b54cad Eduard Zingerman 2025-06-11 19385 if (!backedge)
c9e31900b54cad Eduard Zingerman 2025-06-11 19386 return -ENOMEM;
c9e31900b54cad Eduard Zingerman 2025-06-11 19387 err = copy_verifier_state(&backedge->state, cur);
c9e31900b54cad Eduard Zingerman 2025-06-11 19388 backedge->state.equal_state = &sl->state;
c9e31900b54cad Eduard Zingerman 2025-06-11 19389 backedge->state.insn_idx = insn_idx;
c9e31900b54cad Eduard Zingerman 2025-06-11 19390 err = err ?: add_scc_backedge(env, &sl->state, backedge);
c9e31900b54cad Eduard Zingerman 2025-06-11 19391 if (err) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19392 free_verifier_state(&backedge->state, false);
bf0c2a84df9fb0 Qianfeng Rong 2025-08-11 19393 kfree(backedge);
c9e31900b54cad Eduard Zingerman 2025-06-11 19394 return err;
c9e31900b54cad Eduard Zingerman 2025-06-11 19395 }
c9e31900b54cad Eduard Zingerman 2025-06-11 19396 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19397 return 1;
dc503a8ad98474 Edward Cree 2017-08-15 19398 }
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19399 miss:
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19400 /* when new state is not going to be added do not increase miss count.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19401 * Otherwise several loop iterations will remove the state
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19402 * recorded earlier. The goal of these heuristics is to have
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19403 * states from some iterations of the loop (some in the beginning
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19404 * and some at the end) to help pruning.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19405 */
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19406 if (add_new_state)
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19407 sl->miss_cnt++;
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19408 /* heuristic to determine whether this state is beneficial
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19409 * to keep checking from state equivalence point of view.
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19410 * Higher numbers increase max_states_per_insn and verification time,
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19411 * but do not meaningfully decrease insn_processed.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19412 * 'n' controls how many times state could miss before eviction.
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19413 * Use bigger 'n' for checkpoints because evicting checkpoint states
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19414 * too early would hinder iterator convergence.
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19415 */
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19416 n = is_force_checkpoint(env, insn_idx) && sl->state.branches > 0 ? 64 : 3;
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19417 if (sl->miss_cnt > sl->hit_cnt * n + n) {
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19418 /* the state is unlikely to be useful. Remove it to
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19419 * speed up verification
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19420 */
408fcf946b2bad Eduard Zingerman 2025-02-15 19421 sl->in_free_list = true;
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19422 list_del(&sl->node);
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19423 list_add(&sl->node, &env->free_list);
574078b001cdf6 Eduard Zingerman 2025-02-15 19424 env->free_list_size++;
574078b001cdf6 Eduard Zingerman 2025-02-15 19425 env->explored_states_size--;
408fcf946b2bad Eduard Zingerman 2025-02-15 19426 maybe_free_verifier_state(env, sl);
9f4686c41bdff0 Alexei Starovoitov 2019-04-01 19427 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19428 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19429
06ee7115b0d174 Alexei Starovoitov 2019-04-01 19430 if (env->max_states_per_insn < states_cnt)
06ee7115b0d174 Alexei Starovoitov 2019-04-01 19431 env->max_states_per_insn = states_cnt;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19432
2c78ee898d8f10 Alexei Starovoitov 2020-05-13 19433 if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
a095f421057e22 Andrii Nakryiko 2022-12-06 19434 return 0;
ceefbc96fa5c5b Alexei Starovoitov 2018-12-03 19435
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19436 if (!add_new_state)
a095f421057e22 Andrii Nakryiko 2022-12-06 19437 return 0;
ceefbc96fa5c5b Alexei Starovoitov 2018-12-03 19438
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19439 /* There were no equivalent states, remember the current one.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19440 * Technically the current state is not proven to be safe yet,
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19441 * but it will either reach outer most bpf_exit (which means it's safe)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19442 * or it will be rejected. When there are no loops the verifier won't be
f4d7e40a5b7157 Alexei Starovoitov 2017-12-14 19443 * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19444 * again on the way to bpf_exit.
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19445 * When looping the sl->state.branches will be > 0 and this state
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19446 * will not be considered for equivalence until branches == 0.
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19447 */
43736ec3e02789 Eduard Zingerman 2025-06-13 19448 new_sl = kzalloc(sizeof(struct bpf_verifier_state_list), GFP_KERNEL_ACCOUNT);
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19449 if (!new_sl)
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19450 return -ENOMEM;
06ee7115b0d174 Alexei Starovoitov 2019-04-01 19451 env->total_states++;
574078b001cdf6 Eduard Zingerman 2025-02-15 19452 env->explored_states_size++;
574078b001cdf6 Eduard Zingerman 2025-02-15 19453 update_peak_states(env);
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19454 env->prev_jmps_processed = env->jmps_processed;
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19455 env->prev_insn_processed = env->insn_processed;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19456
7a830b53c17bba Andrii Nakryiko 2022-11-04 19457 /* forget precise markings we inherited, see __mark_chain_precision */
7a830b53c17bba Andrii Nakryiko 2022-11-04 19458 if (env->bpf_capable)
7a830b53c17bba Andrii Nakryiko 2022-11-04 19459 mark_all_scalars_imprecise(env, cur);
7a830b53c17bba Andrii Nakryiko 2022-11-04 19460
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19461 /* add new state to the head of linked list */
679c782de14bd4 Edward Cree 2018-08-22 19462 new = &new_sl->state;
679c782de14bd4 Edward Cree 2018-08-22 19463 err = copy_verifier_state(new, cur);
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19464 if (err) {
679c782de14bd4 Edward Cree 2018-08-22 19465 free_verifier_state(new, false);
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19466 kfree(new_sl);
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19467 return err;
1969db47f8d0e8 Alexei Starovoitov 2017-11-01 19468 }
dc2a4ebc0b44a2 Alexei Starovoitov 2019-05-21 19469 new->insn_idx = insn_idx;
0df1a55afa832f Paul Chaignon 2025-07-01 19470 verifier_bug_if(new->branches != 1, env,
0df1a55afa832f Paul Chaignon 2025-07-01 19471 "%s:branches_to_explore=%d insn %d",
0df1a55afa832f Paul Chaignon 2025-07-01 19472 __func__, new->branches, insn_idx);
c9e31900b54cad Eduard Zingerman 2025-06-11 19473 err = maybe_enter_scc(env, new);
c9e31900b54cad Eduard Zingerman 2025-06-11 19474 if (err) {
c9e31900b54cad Eduard Zingerman 2025-06-11 19475 free_verifier_state(new, false);
e4980fa6463624 Feng Yang 2025-08-27 19476 kfree(new_sl);
c9e31900b54cad Eduard Zingerman 2025-06-11 19477 return err;
c9e31900b54cad Eduard Zingerman 2025-06-11 19478 }
b5dc0163d8fd78 Alexei Starovoitov 2019-06-15 19479
2589726d12a1b1 Alexei Starovoitov 2019-06-15 19480 cur->parent = new;
b5dc0163d8fd78 Alexei Starovoitov 2019-06-15 19481 cur->first_insn_idx = insn_idx;
2793a8b015f7f1 Eduard Zingerman 2023-10-24 19482 cur->dfs_depth = new->dfs_depth + 1;
baaebe0928bf32 Eduard Zingerman 2025-06-11 19483 clear_jmp_history(cur);
5564ee3abb2ebe Eduard Zingerman 2025-02-15 19484 list_add(&new_sl->node, head);
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19485 return 0;
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19486 }
f1bca824dabba4 Alexei Starovoitov 2014-09-29 19487 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Eduard Zingerman
maintainerRe: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
Quoted 11 lines · Dan Carpenter
Hi Eduard,
kernel test robot noticed the following build warnings:
url: https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604 base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master patch link: https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness config: arm-randconfig-r071-20250911 (https://download.01.org/0day-ci/archive/20250912/[email protected]/config) compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 21857ae337e0892a5522b6e7337899caa61de2a6)
Quoted 8 lines
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
Quoted 3 lines
Reported-by kernel test robot <[email protected]>Reported-by Dan Carpenter <[email protected]>smatch warnings:
kernel/bpf/verifier.c:19305is_state_visited()error: uninitialized symbol 'err'.
Hi Dan,
Thank you for the report, kernel test robot already notified me here:
https://lore.kernel.org/bpf/[email protected]/T/#m103a269f4c096b34043bef16fa5f1d629b794968
Thanks,
Eduard
[...]
Eduard Zingerman
maintainerPATCH 10/10 bpf: table based bpf_insn_successors()
In reply to Eduard ZingermanConverting bpf_insn_successors() to use lookup table makes it ~1.5 times faster.
Also remove unnecessary conditionals:
- idx + 1 < prog->len is unnecessary because after check_cfg() all jump targets are guaranteed to be within a program;
- i == 0 || succ[0] != dst is unnecessary because any client of bpf_insn_successors() can handle duplicate edges:
- compute_live_registers()
- compute_scc()
Moving bpf_insn_successors() to liveness.c allows its inlining in liveness.c:__update_stack_liveness().
Such inlining speeds up __update_stack_liveness() by ~40%. bpf_insn_successors() is used in both verifier.c and liveness.c. perf shows such move does not negatively impact users in verifier.c, as these are executed only once before main varification pass. Unlike __update_stack_liveness() which can be triggered multiple times.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/liveness.c | 51 +++++++++++++++++++++++++ kernel/bpf/verifier.c | 72 +----------------------------------- 3 files changed, 53 insertions(+), 71 deletions(-)
include/linux/bpf_verifier.h +1 -0
@@ -1049,6 +1049,7 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st u32 frameno);struct bpf_subprog_info *bpf_find_containing_subprog (struct bpf_verifier_env *env,int off); +int bpf_jmp_offset (struct bpf_insn *insn);int bpf_insn_successors (struct bpf_prog *prog,u32 idx,u32 succ[2 ]);void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask);bool bpf_calls_callback (struct bpf_verifier_env *env,int insn_idx);
kernel/bpf/liveness.c +51 -0
57 lines not on this page · raw message
kernel/bpf/verifier.c +1 -71
90 lines not on this page · raw message
Signature · 1 line
2.47.3
syzbot ci has tested the following series
syzbot ci has tested the following series
[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: use compute_live_registers() info in clean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundant REG_LIVE_READ check in stacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table based bpf_insn_successors()
and found the following issue:
KASAN: slab-out-of-bounds Write in compute_postorder
Full report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17
***
KASAN: slab-out-of-bounds Write in compute_postorder
tree: bpf-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git base: e12873ee856ffa6f104869b8ea10c0f741606f13 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config C repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro
==================================================================
BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840 Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991 CPU: 0 UID: 0 PID: 5991 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x189/0x250 lib/dump_stack.c:120 print_address_description mm/kasan/report.c:378 [inline] print_report+0xca/0x240 mm/kasan/report.c:482 kasan_report+0x118/0x150 mm/kasan/report.c:595 compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840 bpf_check+0x1f90/0x1d440 kernel/bpf/verifier.c:24437 bpf_prog_load+0x1318/0x1930 kernel/bpf/syscall.c:2979 __sys_bpf+0x528/0x870 kernel/bpf/syscall.c:6029 __do_sys_bpf kernel/bpf/syscall.c:6139 [inline] __se_sys_bpf kernel/bpf/syscall.c:6137 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6137 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0x3b0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f366058eba9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffef8486b28 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f36607d5fa0 RCX: 00007f366058eba9 RDX: 0000000000000070 RSI: 0000200000000440 RDI: 0000000000000005 RBP: 00007f3660611e19 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f36607d5fa0 R14: 00007f36607d5fa0 R15: 0000000000000003 </TASK> Allocated by task 5991: kasan_save_stack mm/kasan/common.c:47 [inline] kasan_save_track+0x3e/0x80 mm/kasan/common.c:68 poison_kmalloc_redzone mm/kasan/common.c:388 [inline] __kasan_kmalloc+0x93/0xb0 mm/kasan/common.c:405 kasan_kmalloc include/linux/kasan.h:260 [inline] __do_kmalloc_node mm/slub.c:4365 [inline] __kvmalloc_node_noprof+0x30d/0x5f0 mm/slub.c:5052 kvmalloc_array_node_noprof include/linux/slab.h:1065 [inline] compute_postorder+0xd6/0xcb0 kernel/bpf/verifier.c:17823 bpf_check+0x1f90/0x1d440 kernel/bpf/verifier.c:24437 bpf_prog_load+0x1318/0x1930 kernel/bpf/syscall.c:2979 __sys_bpf+0x528/0x870 kernel/bpf/syscall.c:6029 __do_sys_bpf kernel/bpf/syscall.c:6139 [inline] __se_sys_bpf kernel/bpf/syscall.c:6137 [inline] __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6137 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0x3b0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f The buggy address belongs to the object at ffff88801f1d4b80 which belongs to the cache kmalloc-cg-32 of size 32
The buggy address is located 0 bytes to the right of
allocated 24-byte region [ffff88801f1d4b80, ffff88801f1d4b98)
The buggy address belongs to the physical page: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0xffff88801f1d4e40 pfn:0x1f1d4 memcg:ffff888026bbb801 flags: 0xfff00000000000(node=0|zone=1|lastcpupid=0x7ff) page_type: f5(slab) raw: 00fff00000000000 ffff88801a449b40 dead000000000100 dead000000000122 raw: ffff88801f1d4e40 000000008040003f 00000000f5000000 ffff888026bbb801 page dumped because: kasan: bad access detected page_owner tracks the page as allocated page last allocated via order 0, migratetype Unmovable, gfp_mask 0x52cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP), pid 532, tgid 532 (kworker/u10:0), ts 5351847364, free_ts 0 set_page_owner include/linux/page_owner.h:32 [inline] post_alloc_hook+0x240/0x2a0 mm/page_alloc.c:1851 prep_new_page mm/page_alloc.c:1859 [inline] get_page_from_freelist+0x21e4/0x22c0 mm/page_alloc.c:3858 __alloc_frozen_pages_noprof+0x181/0x370 mm/page_alloc.c:5148 alloc_pages_mpol+0x232/0x4a0 mm/mempolicy.c:2416 alloc_slab_page mm/slub.c:2487 [inline] allocate_slab+0x8a/0x370 mm/slub.c:2655 new_slab mm/slub.c:2709 [inline] ___slab_alloc+0xbeb/0x1410 mm/slub.c:3891 __slab_alloc mm/slub.c:3981 [inline] __slab_alloc_node mm/slub.c:4056 [inline] slab_alloc_node mm/slub.c:4217 [inline] __do_kmalloc_node mm/slub.c:4364 [inline] __kmalloc_noprof+0x305/0x4f0 mm/slub.c:4377 kmalloc_noprof include/linux/slab.h:909 [inline] kzalloc_noprof include/linux/slab.h:1039 [inline] lsm_blob_alloc security/security.c:684 [inline] lsm_cred_alloc security/security.c:701 [inline] security_prepare_creds+0x52/0x390 security/security.c:3271 prepare_kernel_cred+0x2ee/0x500 kernel/cred.c:617 call_usermodehelper_exec_async+0xd0/0x360 kernel/umh.c:88 ret_from_fork+0x3fc/0x770 arch/x86/kernel/process.c:148 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
page_owner free stack trace missing
Memory state around the buggy address:
ffff88801f1d4a80: fa fb fb fb fc fc fc fc 00 00 00 fc fc fc fc fc
ffff88801f1d4b00: 00 00 00 fc fc fc fc fc fa fb fb fb fc fc fc fc
>ffff88801f1d4b80: 00 00 00 fc fc fc fc fc fa fb fb fb fc fc fc fc
^
ffff88801f1d4c00: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
ffff88801f1d4c80: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
==================================================================***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
Eduard Zingerman
maintainerQuoted 34 lines · syzbot ci
syzbot ci has tested the following series
[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf:bpf_verifier_state->cleanedflag instead ofREG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: usecompute_live_registers()info inclean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundantREG_LIVE_READcheck instacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table basedbpf_insn_successors()and found the following issue:
KASAN: slab-out-of-bounds Write in compute_postorderFull report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17***
KASAN: slab-out-of-bounds Write in compute_postordertree: bpf-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git base: e12873ee856ffa6f104869b8ea10c0f741606f13 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config C repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro
Quoted 3 lines
==================================================================
BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840 Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991
The error is caused by the following program:
(e5) if r15 (null) 0xffffffff goto pc-1 <---- absence of DISCOVERED/EXPLORED mark here
(71) r1 = *(u8 *)(r1 +70) leads to instruction being put on stack second time
(85) call pc+2
(85) call bpf_get_numa_node_id#42
(95) exit
(95) exit
And this is the fix:
kernel/bpf/verifier.c +1 -0
@@ -17840,6 +17840,7 @@ static int compute_postorder(struct bpf_verifier_env *env) stack_sz =1 ; do { top = stack[stack_sz - 1]; + state[top] |= DISCOVERED; if (state[top] & EXPLORED) { postorder[cur_postorder++] = top; stack_sz--;
[...]
Quoted 36 lines · Eduard Zingerman
Quoted 34 lines · syzbot ci
syzbot ci has tested the following series
[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf:bpf_verifier_state->cleanedflag instead ofREG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: usecompute_live_registers()info inclean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundantREG_LIVE_READcheck instacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table basedbpf_insn_successors()and found the following issue:
KASAN: slab-out-of-bounds Write in compute_postorderFull report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17***
KASAN: slab-out-of-bounds Write in compute_postordertree: bpf-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git base: e12873ee856ffa6f104869b8ea10c0f741606f13 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config C repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro
Quoted 7 lines
Quoted 3 lines
==================================================================
BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840 Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991The error is caused by the following program:
(e5) if r15 (null)
0xffffffffgoto pc-1 <---- absence of DISCOVERED/EXPLORED mark here
(null) ?
Is it jset again? but insn_successors() handles it already. Or pc-1 infinite loop caused it?
but we have pc-1 selftest...
Quoted 19 lines
(71) r1 = *(u8 *)(r1 +70) leads to instruction being put on stack second time
(85) call pc+2
(85) call bpf_get_numa_node_id#42
(95) exit
(95) exitAnd this is the fix:
kernel/bpf/verifier.c +1
-0@@ -17840,6 +17840,7 @@ static int compute_postorder(struct bpf_verifier_env *env) stack_sz =1 ; do { top = stack[stack_sz - 1]; + state[top] |= DISCOVERED; if (state[top] & EXPLORED) { postorder[cur_postorder++] = top; stack_sz--;
Eduard Zingerman
maintainerQuoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Eduard Zingerman
Quoted 37 lines · syzbot ci
syzbot ci has tested the following series
[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf:bpf_verifier_state->cleanedflag instead ofREG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: usecompute_live_registers()info inclean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundantREG_LIVE_READcheck instacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table basedbpf_insn_successors()and found the following issue:
KASAN: slab-out-of-bounds Write in compute_postorderFull report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17***
KASAN: slab-out-of-bounds Write in compute_postordertree: bpf-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git base: e12873ee856ffa6f104869b8ea10c0f741606f13 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config C repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro==================================================================
BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840 Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991
Quoted 5 lines
Quoted 3 lines
The error is caused by the following program:
(e5) if r15 (null)
0xffffffffgoto pc-1 <---- absence of DISCOVERED/EXPLORED mark here(null) ?
The code byte is 0xe5, BPF_OP(0xe5) == 0xe0, which is an invalid opcode. But opcodes are verified after check_cfg()/compute_postorder().
Quoted 3 lines
Is it jset again? but
insn_successors()handles it already. Or pc-1 infinite loop caused it?
but we have pc-1 selftest...
It's not infinite, but it causes instruction to be put twice on the stack array, and this array is allocated expecting max prog->len instructions. KASAN would only catch this error if program really needs to consume full stack depth during postorder construction, as far as I understand.
[...]
Quoted 43 lines · Eduard Zingerman
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Eduard Zingerman
Quoted 37 lines · syzbot ci
syzbot ci has tested the following series
[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf:bpf_verifier_state->cleanedflag instead ofREG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: usecompute_live_registers()info inclean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundantREG_LIVE_READcheck instacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table basedbpf_insn_successors()and found the following issue:
KASAN: slab-out-of-bounds Write in compute_postorderFull report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17***
KASAN: slab-out-of-bounds Write in compute_postordertree: bpf-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git base: e12873ee856ffa6f104869b8ea10c0f741606f13 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config C repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro==================================================================
BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840 Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
The error is caused by the following program:
(e5) if r15 (null)
0xffffffffgoto pc-1 <---- absence of DISCOVERED/EXPLORED mark here(null) ?
The
codebyte is0xe5,BPF_OP(0xe5)==0xe0, which is an invalid opcode. But opcodes are verified after check_cfg()/compute_postorder().
hmm.resolve_pseudo_ldimm64() -> bpf_opcode_in_insntable() does it before check_cfg.