Skip to contentSkip to conversation

bpf: replace path-sensitive with path-insensitive live stack analysis

PATCH Superseded Eduard Zingerman 5 participants 22 messages Sep 11, 2025 – Sep 12, 2025

Subsystems (6)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Eduard Zingerman

author maintainer
#0 lore raw

cover bpf: replace path-sensitive with path-insensitive live stack analysis

Consider the following program, assuming checkpoint is created for a state at instruction (3):

1: call bpf_get_prandom_u32()
2: *(u64 *)(r10 - 8) = 42
-- checkpoint #1 --
3: if r0 != 0 goto +1
4: exit;
5: r0 = *(u64 *)(r10 - 8)
6: exit

The verifier processes this program by exploring two paths:
- 1 -> 2 -> 3 -> 4
- 1 -> 2 -> 3 -> 5 -> 6

When instruction (5) is processed, the current liveness tracking mechanism moves up the register parent links and records a "read" mark for stack slot -8 at checkpoint #1, stopping because of the "write" mark recorded at instruction (2).

This patch set replaces the existing liveness tracking mechanism with a path-insensitive data flow analysis. The program above is processed as follows:
- a data structure representing live stack slots for instructions 1-6 in frame #0 is allocated;
- when instruction (2) is processed, record that slot -8 is written at instruction (2) in frame #0;
- when instruction (5) is processed, record that slot -8 is read at instruction (5) in frame #0;
- when instruction (6) is processed, propagate read mark for slot -8 up the control flow graph to instructions 3 and 2.

The key difference is that the new mechanism operates on a control flow graph and associates read and write marks with pairs of (call chain, instruction index). In contrast, the old mechanism operates on verifier states and register parent links, associating read and write marks with verifier states.

Motivation
==========

As it stands, this patch set makes liveness tracking slightly less precise, as it no longer distinguishes individual program paths taken by the verifier during symbolic execution.
See the "Impact on verification performance" section for details.

However, this change is intended as a stepping stone toward the following goals:
- Short term, integrate precision tracking into liveness analysis and remove the following code:
- verifier backedge states accumulation in is_state_visited();
- most of the logic for precision tracking;
- jump history tracking.
- Long term, help with more efficient loop verification handling.

Why integrating precision tracking?
-----------------------------------

In a sense, precision tracking is very similar to liveness tracking. The data flow equations for liveness tracking look as follows:

live_after =
U [state[s].live_before for s in insn_successors(i)]

state[i].live_before =
(live_after / state[i].must_write) U state[i].may_read

While data flow equations for precision tracking look as follows:

precise_after =
U [state[s].precise_before for s in insn_successors(i)]

// if some of the instruction outputs are precise, // assume its inputs to be precise induced_precise =
⎧ state[i].may_read if (state[i].may_write ∩ precise_after) ≠ ∅
⎨
⎩ ∅ otherwise

state[i].precise_before =
(precise_after / state[i].must_write) ∩ induced_precise

Where:
- may_read set represents a union of all possibly read slots
(any slot in may_read set might be by the instruction);
- must_write set represents an intersection of all possibly written slots (any slot in must_write set is guaranteed to be written by the instruction).
- may_write set represents a union of all possibly written slots (any slot in may_write set might be written by the instruction).

This means that precision tracking can be implemented as a logical extension of liveness tracking:
- track registers as well as stack slots;
- add bit masks to represent precise_before and may_write;
- add above equations for precise_before computation;
- (linked registers require some additional consideration).

Such extension would allow removal of:
- precision propagation logic in verifier.c:
- backtrack_insn()
- mark_chain_precision()
- propagate_{precision,backedges}()
- push_jmp_history() and related data structures, which are only used by precision tracking;
- add_scc_backedge() and related backedge state accumulation in is_state_visited(), superseded by per-callchain function state accumulated by liveness analysis.

The hope here is that unifying liveness and precision tracking will reduce overall amount of code and make it easier to reason about.

How this helps with loops?
--------------------------

As it stands, this patch set shares the same deficiency as the current liveness tracking mechanism. Liveness marks on stack slots cannot be used to prune states when processing iterator-based loops:
- such states still have branches to be explored;
- meaning that not all stack slot reads have been discovered.

For example:

  1: while(iter_next()) {
  2:   if (...)
  3:     r0 = *(u64 *)(r10 - 8)
  4:   if (...)
  5:     r0 = *(u64 *)(r10 - 16)
  6:   ...
  7: }

For any checkpoint state created at instruction (1), it is only possible to rely on read marks for slots fp[-8] and fp[-16] once all child states of (1) have been explored. Thus, when the verifier transitions from (7) to (1), it cannot rely on read marks.

However, sacrificing path-sensitivity makes it possible to run analysis defined in this patch set before main verification pass, if estimates for value ranges are available.
E.g. for the following program:

  1: while(iter_next()) {
  2:   r0 = r10
  3:   r0 += r2
  4:   r0 = *(u64 *)(r2 + 0)
  5:   ...
  6: }

If an estimate for r2 range is available before the main verification pass, it can be used to populate read marks at instruction (4) and run the liveness analysis. Thus making conservative liveness information available during loops verification.

Such estimates can be provided by some form of value range analysis. Value range analysis is also necessary to address loop verification from another angle: computing boundaries for loop induction variables and iteration counts.

The hope here is that the new liveness tracking mechanism will support the broader goal of making loop verification more efficient.

Validation
==========

The change was tested on three program sets:
- bpf selftests
- sched_ext
- Meta's internal set of programs

Commit [#8] enables a special mode where both the current and new liveness analyses are enabled simultaneously. This mode signals an error if the new algorithm considers a stack slot dead while the current algorithm assumes it is alive. This mode was very useful for debugging. At the time of posting, no such errors have been reported for the above program sets.

[#8] "bpf: signal error if old liveness is more conservative than new"

Impact on memory consumption
============================

Debug patch [1] extends the kernel and veristat to count the amount of memory allocated for storing analysis data. This patch is not included in the submission. The maximal observed impact for the above program sets is 2.6Mb.

Data below is shown in bytes.

For bpf selftests top 5 consumers look as follows:

  File                     Program           liveness mem
  -----------------------  ----------------  ------------
  pyperf180.bpf.o          on_event               2629740
  pyperf600.bpf.o          on_event               2287662
  pyperf100.bpf.o          on_event               1427022
  test_verif_scale3.bpf.o  balancer_ingress       1121283
  pyperf_subprogs.bpf.o    on_event                756900

For sched_ext top 5 consumers loog as follows:

  File       Program                          liveness mem
  ---------  -------------------------------  ------------
  bpf.bpf.o  lavd_enqueue                           164686
  bpf.bpf.o  lavd_select_cpu                        157393
  bpf.bpf.o  layered_enqueue                        154817
  bpf.bpf.o  lavd_init                              127865
  bpf.bpf.o  layered_dispatch                       110129

For Meta's internal set of programs top consumer is 1Mb.

[1] https://github.com/kernel-patches/bpf/commit/085588e787b7998a296eb320666897d80bca7c08

Impact on verification performance
==================================

Veristat results below are reported using -f insns_pct>1 -f !insns<500 filter and -t option (BPF_F_TEST_STATE_FREQ flag).

master vs patch-set, selftests (out of ~4K programs)
----------------------------------------------------

  File                              Program                                 Insns (A)  Insns (B)  Insns    (DIFF)
  --------------------------------  --------------------------------------  ---------  ---------  ---------------
  cpumask_success.bpf.o             test_global_mask_nested_deep_array_rcu       1622       1655     +33 (+2.03%)
  strobemeta_bpf_loop.bpf.o         on_event                                     2163       2684   +521 (+24.09%)
  test_cls_redirect.bpf.o           cls_redirect                                36001      42515  +6514 (+18.09%)
  test_cls_redirect_dynptr.bpf.o    cls_redirect                                 2299       2339     +40 (+1.74%)
  test_cls_redirect_subprogs.bpf.o  cls_redirect                                69545      78497  +8952 (+12.87%)
  test_l4lb_noinline.bpf.o          balancer_ingress                             2993       3084     +91 (+3.04%)
  test_xdp_noinline.bpf.o           balancer_ingress_v4                          3539       3616     +77 (+2.18%)
  test_xdp_noinline.bpf.o           balancer_ingress_v6                          3608       3685     +77 (+2.13%)

master vs patch-set, sched_ext (out of 148 programs)
----------------------------------------------------

  File       Program           Insns (A)  Insns (B)  Insns    (DIFF)
  ---------  ----------------  ---------  ---------  ---------------
  bpf.bpf.o  chaos_dispatch         2257       2287     +30 (+1.33%)
  bpf.bpf.o  lavd_enqueue          20735      22101   +1366 (+6.59%)
  bpf.bpf.o  lavd_select_cpu       22100      24409  +2309 (+10.45%)
  bpf.bpf.o  layered_dispatch      25051      25606    +555 (+2.22%)
  bpf.bpf.o  p2dq_dispatch           961        990     +29 (+3.02%)
  bpf.bpf.o  rusty_quiescent         526        534      +8 (+1.52%)
  bpf.bpf.o  rusty_runnable          541        547      +6 (+1.11%)

Perf report
===========

In relative terms, the analysis does not consume much CPU time. For example, here is a perf report collected for pyperf180 selftest:

 # Children      Self  Command   Shared Object         Symbol
 # ........  ........  ........  ....................  ........................................
        ...
      1.22%     1.22%  veristat  [kernel.kallsyms]     [k] bpf_update_live_stack
        ...

Eduard Zingerman (10):
bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
bpf: use compute_live_registers() info in clean_func_state
bpf: remove redundant REG_LIVE_READ check in stacksafe()
bpf: declare a few utility functions as internal api
bpf: compute instructions postorder per subprogram
bpf: callchain sensitive stack liveness tracking using CFG
bpf: enable callchain sensitive stack liveness tracking
bpf: signal error if old liveness is more conservative than new
bpf: disable and remove registers chain based liveness
bpf: table based bpf_insn_successors()

 Documentation/bpf/verifier.rst                | 264 -------
 include/linux/bpf_verifier.h                  |  52 +-
 kernel/bpf/Makefile                           |   2 +-
 kernel/bpf/liveness.c                         | 723 ++++++++++++++++++
 kernel/bpf/log.c                              |  28 +-
 kernel/bpf/verifier.c                         | 564 ++++----------
 .../testing/selftests/bpf/prog_tests/align.c  | 178 ++---
 .../selftests/bpf/prog_tests/spin_lock.c      |  12 +-
 .../selftests/bpf/prog_tests/test_veristat.c  |  44 +-
 .../selftests/bpf/progs/exceptions_assert.c   |  34 +-
 .../selftests/bpf/progs/iters_state_safety.c  |   4 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |   6 +-
 .../bpf/progs/mem_rdonly_untrusted.c          |   4 +-
 .../selftests/bpf/progs/verifier_bounds.c     |  38 +-
 .../bpf/progs/verifier_global_ptr_args.c      |   4 +-
 .../selftests/bpf/progs/verifier_ldsx.c       |   2 +-
 .../selftests/bpf/progs/verifier_precision.c  |  16 +-
 .../selftests/bpf/progs/verifier_scalar_ids.c |  10 +-
 .../selftests/bpf/progs/verifier_spill_fill.c |  40 +-
 .../bpf/progs/verifier_subprog_precision.c    |   6 +-
 .../selftests/bpf/verifier/bpf_st_mem.c       |   4 +-
 21 files changed, 1107 insertions(+), 928 deletions(-)
 create mode 100644 kernel/bpf/liveness.c
Signature · 1 line
2.47.3
author maintainer
#1 lore raw

PATCH 1/10 bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE

Prepare for bpf_reg_state->live field removal by introducing a separate flag to track if clean_verifier_state() had been applied to the state. No functional changes.

---
 include/linux/bpf_verifier.h |  2 +-
 kernel/bpf/log.c             |  6 ++----
 kernel/bpf/verifier.c        | 13 ++++---------
 3 files changed, 7 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +1 -1
index 020de62bd09c..ac16da8b49dc 100644
@@ -45,7 +45,6 @@ enum bpf_reg_liveness {
 	REG_LIVE_READ64 = 0x2, /* likewise, but full 64-bit content matters */
 	REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64,
 	REG_LIVE_WRITTEN = 0x4, /* reg was written first, screening off later reads */
-	REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */
 };
 
 #define ITER_PREFIX "bpf_iter_"
@@ -445,6 +444,7 @@ struct bpf_verifier_state {
 
 	bool speculative;
 	bool in_sleepable;
+	bool cleaned;
 
 	/* first and last insn idx of this verifier state */
 	u32 first_insn_idx;
kernel/bpf/log.c +2 -4
index e4983c1303e7..0d6d7bfb2fd0 100644
@@ -545,14 +545,12 @@ static char slot_type_char[] = {
 static void print_liveness(struct bpf_verifier_env *env,
 			   enum bpf_reg_liveness live)
 {
-	if (live & (REG_LIVE_READ | REG_LIVE_WRITTEN | REG_LIVE_DONE))
-	    verbose(env, "_");
+	if (live & (REG_LIVE_READ | REG_LIVE_WRITTEN))
+		verbose(env, "_");
 	if (live & REG_LIVE_READ)
 		verbose(env, "r");
 	if (live & REG_LIVE_WRITTEN)
 		verbose(env, "w");
-	if (live & REG_LIVE_DONE)
-		verbose(env, "D");
 }
 
 #define UNUM_MAX_DECIMAL U16_MAX
kernel/bpf/verifier.c +4 -9
index b9394f8fac0e..46a6d69de309 100644
@@ -1758,6 +1758,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state,
 		return err;
 	dst_state->speculative = src->speculative;
 	dst_state->in_sleepable = src->in_sleepable;
+	dst_state->cleaned = src->cleaned;
 	dst_state->curframe = src->curframe;
 	dst_state->branches = src->branches;
 	dst_state->parent = src->parent;
@@ -3574,11 +3575,6 @@ static int mark_reg_read(struct bpf_verifier_env *env,
 		/* if read wasn't screened by an earlier write ... */
 		if (writes && state->live & REG_LIVE_WRITTEN)
 			break;
-		if (verifier_bug_if(parent->live & REG_LIVE_DONE, env,
-				    "type %s var_off %lld off %d",
-				    reg_type_str(env, parent->type),
-				    parent->var_off.value, parent->off))
-			return -EFAULT;
 		/* The first condition is more likely to be true than the
 		 * second, checked it first.
 		 */
@@ -18472,7 +18468,6 @@ static void clean_func_state(struct bpf_verifier_env *env,
 	for (i = 0; i < BPF_REG_FP; i++) {
 		live = st->regs[i].live;
 		/* liveness must not touch this register anymore */
-		st->regs[i].live |= REG_LIVE_DONE;
 		if (!(live & REG_LIVE_READ))
 			/* since the register is unused, clear its state
 			 * to make further comparison simpler
@@ -18483,7 +18478,6 @@ static void clean_func_state(struct bpf_verifier_env *env,
 	for (i = 0; i < st->allocated_stack / BPF_REG_SIZE; i++) {
 		live = st->stack[i].spilled_ptr.live;
 		/* liveness must not touch this stack slot anymore */
-		st->stack[i].spilled_ptr.live |= REG_LIVE_DONE;
 		if (!(live & REG_LIVE_READ)) {
 			__mark_reg_not_init(env, &st->stack[i].spilled_ptr);
 			for (j = 0; j < BPF_REG_SIZE; j++)
@@ -18497,6 +18491,7 @@ static void clean_verifier_state(struct bpf_verifier_env *env,
 {
 	int i;
 
+	st->cleaned = true;
 	for (i = 0; i <= st->curframe; i++)
 		clean_func_state(env, st->frame[i]);
 }
@@ -18524,7 +18519,7 @@ static void clean_verifier_state(struct bpf_verifier_env *env,
  * their final liveness marks are already propagated.
  * Hence when the verifier completes the search of state list in is_state_visited()
  * we can call this clean_live_states() function to mark all liveness states
- * as REG_LIVE_DONE to indicate that 'parent' pointers of 'struct bpf_reg_state'
+ * as st->cleaned to indicate that 'parent' pointers of 'struct bpf_reg_state'
  * will not be used.
  * This function also clears the registers and stack for states that !READ
  * to simplify state merging.
@@ -18547,7 +18542,7 @@ static void clean_live_states(struct bpf_verifier_env *env, int insn,
 		if (sl->state.insn_idx != insn ||
 		    !same_callsites(&sl->state, cur))
 			continue;
-		if (sl->state.frame[0]->regs[0].live & REG_LIVE_DONE)
+		if (sl->state.cleaned)
 			/* all regs in this state in all frames were already marked */
 			continue;
 		if (incomplete_read_marks(env, &sl->state))
Signature · 1 line
2.47.3
author maintainer
#2 lore raw

PATCH 2/10 bpf: use compute_live_registers() info in clean_func_state

In reply to Eduard Zingerman

Prepare for bpf_reg_state->live field removal by leveraging insn_aux_data->live_regs_before instead of bpf_reg_state->live in compute_live_registers(). This is similar to logic in func_states_equal(). No changes in verification performance for selftests or sched_ext.

---
 kernel/bpf/verifier.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +9 -6
index 46a6d69de309..698e6a24d2a2 100644
@@ -18460,15 +18460,16 @@ static bool check_scalar_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap)
 }
 
 static void clean_func_state(struct bpf_verifier_env *env,
-			     struct bpf_func_state *st)
+			     struct bpf_func_state *st,
+			     u32 ip)
 {
+	u16 live_regs = env->insn_aux_data[ip].live_regs_before;
 	enum bpf_reg_liveness live;
 	int i, j;
 
 	for (i = 0; i < BPF_REG_FP; i++) {
-		live = st->regs[i].live;
 		/* liveness must not touch this register anymore */
-		if (!(live & REG_LIVE_READ))
+		if (!(live_regs & BIT(i)))
 			/* since the register is unused, clear its state
 			 * to make further comparison simpler
 			 */
@@ -18489,11 +18490,13 @@ static void clean_func_state(struct bpf_verifier_env *env,
 static void clean_verifier_state(struct bpf_verifier_env *env,
 				 struct bpf_verifier_state *st)
 {
-	int i;
+	int i, ip;
 
 	st->cleaned = true;
-	for (i = 0; i <= st->curframe; i++)
-		clean_func_state(env, st->frame[i]);
+	for (i = 0; i <= st->curframe; i++) {
+		ip = frame_insn_idx(st, i);
+		clean_func_state(env, st->frame[i], ip);
+	}
 }
 
 /* the parentage chains form a tree.
Signature · 1 line
2.47.3
author maintainer
#3 lore raw

PATCH 3/10 bpf: remove redundant REG_LIVE_READ check in stacksafe()

In reply to Eduard Zingerman

stacksafe() is called in exact == NOT_EXACT mode only for states that had been porcessed by clean_verifier_states(). The latter replaces dead stack spills with a series of STACK_INVALID masks. Such masks are already handled by stacksafe().

---
 kernel/bpf/verifier.c | 7 -------
 1 file changed, 7 deletions(-)
kernel/bpf/verifier.c +0 -7
index 698e6a24d2a2..8673b955a6cd 100644
@@ -18756,13 +18756,6 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
 		     cur->stack[spi].slot_type[i % BPF_REG_SIZE]))
 			return false;
 
-		if (!(old->stack[spi].spilled_ptr.live & REG_LIVE_READ)
-		    && exact == NOT_EXACT) {
-			i += BPF_REG_SIZE - 1;
-			/* explored state didn't use this */
-			continue;
-		}
-
 		if (old->stack[spi].slot_type[i % BPF_REG_SIZE] == STACK_INVALID)
 			continue;
 
Signature · 1 line
2.47.3
author maintainer
#4 lore raw

PATCH 4/10 bpf: declare a few utility functions as internal api

In reply to Eduard Zingerman

Namely, rename the following functions and add prototypes to bpf_verifier.h:
- find_containing_subprog -> bpf_find_containing_subprog

- insn_successors         -> bpf_insn_successors
- calls_callback          -> bpf_calls_callback
- fmt_stack_mask          -> bpf_fmt_stack_mask
---
 include/linux/bpf_verifier.h |  5 +++++
 kernel/bpf/verifier.c        | 34 ++++++++++++++++------------------
 2 files changed, 21 insertions(+), 18 deletions(-)
include/linux/bpf_verifier.h +5 -0
index ac16da8b49dc..93563564bde5 100644
@@ -1065,4 +1065,9 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate,
 		      u32 frameno);
 
+struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off);
+int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]);
+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
+bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx);
+
 #endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/verifier.c +16 -18
index 8673b955a6cd..5658e1e1d5c5 100644
@@ -2964,7 +2964,7 @@ static int cmp_subprogs(const void *a, const void *b)
 }
 
 /* Find subprogram that contains instruction at 'off' */
-static struct bpf_subprog_info *find_containing_subprog(struct bpf_verifier_env *env, int off)
+struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off)
 {
 	struct bpf_subprog_info *vals = env->subprog_info;
 	int l, r, m;
@@ -2989,7 +2989,7 @@ static int find_subprog(struct bpf_verifier_env *env, int off)
 {
 	struct bpf_subprog_info *p;
 
-	p = find_containing_subprog(env, off);
+	p = bpf_find_containing_subprog(env, off);
 	if (!p || p->start != off)
 		return -ENOENT;
 	return p - env->subprog_info;
@@ -4196,7 +4196,7 @@ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
 	}
 }
 /* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */
-static void fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
+void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
 {
 	DECLARE_BITMAP(mask, 64);
 	bool first = true;
@@ -4251,8 +4251,6 @@ static void bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_histo
 	}
 }
 
-static bool calls_callback(struct bpf_verifier_env *env, int insn_idx);
-
 /* For given verifier state backtrack_insn() is called from the last insn to
  * the first insn. Its purpose is to compute a bitmask of registers and
  * stack slots that needs precision in the parent verifier state.
@@ -4279,7 +4277,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 		fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_reg_mask(bt));
 		verbose(env, "mark_precise: frame%d: regs=%s ",
 			bt->frame, env->tmp_str_buf);
-		fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
+		bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
 		verbose(env, "stack=%s before ", env->tmp_str_buf);
 		verbose(env, "%d: ", idx);
 		verbose_insn(env, insn);
@@ -4480,7 +4478,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			 * backtracking, as these registers are set by the function
 			 * invoking callback.
 			 */
-			if (subseq_idx >= 0 && calls_callback(env, subseq_idx))
+			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
@@ -4919,7 +4917,7 @@ static int __mark_chain_precision(struct bpf_verifier_env *env,
 					     bt_frame_reg_mask(bt, fr));
 				verbose(env, "mark_precise: frame%d: parent state regs=%s ",
 					fr, env->tmp_str_buf);
-				fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
+				bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
 					       bt_frame_stack_mask(bt, fr));
 				verbose(env, "stack=%s: ", env->tmp_str_buf);
 				print_verifier_state(env, st, fr, true);
@@ -11004,7 +11002,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 					       "At callback return", "R0");
 			return -EINVAL;
 		}
-		if (!calls_callback(env, callee->callsite)) {
+		if (!bpf_calls_callback(env, callee->callsite)) {
 			verifier_bug(env, "in callback at %d, callsite %d !calls_callback",
 				     *insn_idx, callee->callsite);
 			return -EFAULT;
@@ -17269,7 +17267,7 @@ static void mark_subprog_changes_pkt_data(struct bpf_verifier_env *env, int off)
 {
 	struct bpf_subprog_info *subprog;
 
-	subprog = find_containing_subprog(env, off);
+	subprog = bpf_find_containing_subprog(env, off);
 	subprog->changes_pkt_data = true;
 }
 
@@ -17277,7 +17275,7 @@ static void mark_subprog_might_sleep(struct bpf_verifier_env *env, int off)
 {
 	struct bpf_subprog_info *subprog;
 
-	subprog = find_containing_subprog(env, off);
+	subprog = bpf_find_containing_subprog(env, off);
 	subprog->might_sleep = true;
 }
 
@@ -17291,8 +17289,8 @@ static void merge_callee_effects(struct bpf_verifier_env *env, int t, int w)
 {
 	struct bpf_subprog_info *caller, *callee;
 
-	caller = find_containing_subprog(env, t);
-	callee = find_containing_subprog(env, w);
+	caller = bpf_find_containing_subprog(env, t);
+	callee = bpf_find_containing_subprog(env, w);
 	caller->changes_pkt_data |= callee->changes_pkt_data;
 	caller->might_sleep |= callee->might_sleep;
 }
@@ -17362,7 +17360,7 @@ static void mark_calls_callback(struct bpf_verifier_env *env, int idx)
 	env->insn_aux_data[idx].calls_callback = true;
 }
 
-static bool calls_callback(struct bpf_verifier_env *env, int insn_idx)
+bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx)
 {
 	return env->insn_aux_data[insn_idx].calls_callback;
 }
@@ -19410,7 +19408,7 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 					goto hit;
 				}
 			}
-			if (calls_callback(env, insn_idx)) {
+			if (bpf_calls_callback(env, insn_idx)) {
 				if (states_equal(env, &sl->state, cur, RANGE_WITHIN))
 					goto hit;
 				goto skip_inf_loop_check;
@@ -24136,7 +24134,7 @@ static bool can_jump(struct bpf_insn *insn)
 	return false;
 }
 
-static int insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2])
+int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2])
 {
 	struct bpf_insn *insn = &prog->insnsi[idx];
 	int i = 0, insn_sz;
@@ -24352,7 +24350,7 @@ static int compute_live_registers(struct bpf_verifier_env *env)
 			u16 new_out = 0;
 			u16 new_in = 0;
 
-			succ_num = insn_successors(env->prog, insn_idx, succ);
+			succ_num = bpf_insn_successors(env->prog, insn_idx, succ);
 			for (int s = 0; s < succ_num; ++s)
 				new_out |= state[succ[s]].in;
 			new_in = (new_out & ~live->def) | live->use;
@@ -24521,7 +24519,7 @@ static int compute_scc(struct bpf_verifier_env *env)
 				stack[stack_sz++] = w;
 			}
 			/* Visit 'w' successors */
-			succ_cnt = insn_successors(env->prog, w, succ);
+			succ_cnt = bpf_insn_successors(env->prog, w, succ);
 			for (j = 0; j < succ_cnt; ++j) {
 				if (pre[succ[j]]) {
 					low[w] = min(low[w], low[succ[j]]);
Signature · 1 line
2.47.3
author maintainer
#5 lore raw

PATCH 5/10 bpf: compute instructions postorder per subprogram

In reply to Eduard Zingerman

The next patch would require doing postorder traversal of individual subprograms. Facilitate this by moving env->cfg.insn_postorder computation from check_cfg() to a separate pass, as check_cfg() descends into called subprograms (and it needs to, because of merge_callee_effects() logic).

env->cfg.insn_postorder is used only by compute_live_registers(), this function does not track cross subprogram dependencies, thus the change does not affect it's operation.

---
 include/linux/bpf_verifier.h |  6 +++-
 kernel/bpf/verifier.c        | 67 +++++++++++++++++++++++++++++-------
 2 files changed, 59 insertions(+), 14 deletions(-)
include/linux/bpf_verifier.h +5 -1
index 93563564bde5..bd87e80f9423 100644
@@ -665,6 +665,7 @@ struct bpf_subprog_info {
 	/* 'start' has to be the first field otherwise find_subprog() won't work */
 	u32 start; /* insn idx of function entry point */
 	u32 linfo_idx; /* The idx to the main_prog->aux->linfo */
+	u32 postorder_start; /* The idx to the env->cfg.insn_postorder */
 	u16 stack_depth; /* max. stack depth used by this function */
 	u16 stack_extra;
 	/* offsets in range [stack_depth .. fastcall_stack_off)
@@ -794,7 +795,10 @@ struct bpf_verifier_env {
 	struct {
 		int *insn_state;
 		int *insn_stack;
-		/* vector of instruction indexes sorted in post-order */
+		/*
+		 * vector of instruction indexes sorted in post-order, grouped by subprogram,
+		 * see bpf_subprog_info->postorder_start.
+		 */
 		int *insn_postorder;
 		int cur_stack;
 		/* current position in the insn_postorder vector */
kernel/bpf/verifier.c +54 -13

103 lines not on this page · raw message

Signature · 1 line
2.47.3
author maintainer
#6 lore raw

PATCH 6/10 bpf: callchain sensitive stack liveness tracking using CFG

In reply to Eduard Zingerman

This commit adds a flow-sensitive, context-sensitive, path-insensitive data flow analysis for live stack slots:
- flow-sensitive: uses program control flow graph to compute data flow values;
- context-sensitive: collects data flow values for each possible call chain in a program;
- path-insensitive: does not distinguish between separate control flow graph paths reaching the same instruction.

Compared to the current path-sensitive analysis, this approach trades some precision for not having to enumerate every path in the program. This gives a theoretical capability to run the analysis before main verification pass. See cover letter for motivation.

The basic idea is as follows:
- Data flow values indicate stack slots that might be read and stack slots that are definitely written.
- Data flow values are collected for each
(call chain, instruction number) combination in the program.
- Within a subprogram, data flow values are propagated using control flow graph.
- Data flow values are transferred from entry instructions of callee subprograms to call sites in caller subprograms.

In other words, a tree of all possible call chains is constructed. Each node of this tree represents a subprogram. Read and write marks are collected for each instruction of each node. Live stack slots are first computed for lower level nodes. Then, information about outer stack slots that might be read or are definitely written by a subprogram is propagated one level up, to the corresponding call instructions of the upper nodes. Procedure repeats until root node is processed.

In the absence of value range analysis, stack read/write marks are collected during main verification pass, and data flow computation is triggered each time verifier.c:states_equal() needs to query the information.

Implementation details are documented in kernel/bpf/liveness.c. Quantitative data about verification performance changes and memory consumption is in the cover letter.

---
 include/linux/bpf_verifier.h |  14 +
 kernel/bpf/Makefile          |   2 +-
 kernel/bpf/liveness.c        | 672 +++++++++++++++++++++++++++++++++++
 3 files changed, 687 insertions(+), 1 deletion(-)
 create mode 100644 kernel/bpf/liveness.c
include/linux/bpf_verifier.h +14 -0
index bd87e80f9423..2e3bdd50e2ba 100644
@@ -745,6 +745,8 @@ struct bpf_scc_info {
 	struct bpf_scc_visit visits[];
 };
 
+struct bpf_liveness;
+
 /* single container for all structs
  * one verifier_env per bpf_check() call
  */
@@ -846,6 +848,7 @@ struct bpf_verifier_env {
 	struct bpf_insn insn_buf[INSN_BUF_SIZE];
 	struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
 	struct bpf_scc_callchain callchain_buf;
+	struct bpf_liveness *liveness;
 	/* array of pointers to bpf_scc_info indexed by SCC id */
 	struct bpf_scc_info **scc_info;
 	u32 scc_cnt;
@@ -1074,4 +1077,15 @@ int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx);
 
+int bpf_stack_liveness_init(struct bpf_verifier_env *env);
+void bpf_stack_liveness_free(struct bpf_verifier_env *env);
+int bpf_update_live_stack(struct bpf_verifier_env *env);
+int bpf_mark_stack_read(struct bpf_verifier_env *env, u32 frameno, u32 insn_idx, u64 mask);
+void bpf_mark_stack_write(struct bpf_verifier_env *env, u32 frameno, u64 mask);
+int bpf_reset_stack_write_marks(struct bpf_verifier_env *env, u32 insn_idx);
+int bpf_commit_stack_write_marks(struct bpf_verifier_env *env);
+int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
+bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi);
+void bpf_reset_live_stack_callchain(struct bpf_verifier_env *env);
+
 #endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/Makefile +1 -1
index 269c04a24664..5df982b316ec 100644
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
 endif
 CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
 
-obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o
+obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
kernel/bpf/liveness.c new file +672 -0

672 lines not on this page · raw message

Signature · 1 line
2.47.3
author maintainer
#7 lore raw

PATCH 7/10 bpf: enable callchain sensitive stack liveness tracking

In reply to Eduard Zingerman

Allocate analysis instance:
- Add bpf_stack_liveness_{init,free}() calls to bpf_check().

Notify the instance about any stack reads and writes:
- Add bpf_mark_stack_write() call at every location where REG_LIVE_WRITTEN is recorded for a stack slot.
- Add bpf_mark_stack_read() call at every location mark_reg_read() is called.
- Both bpf_mark_stack_{read,write}() rely on env->liveness->cur_instance callchain being in sync with env->cur_state. It is possible to update env->liveness->cur_instance every time a mark read/write is called, but that costs a hash table lookup and is noticeable in the performance profile. Hence, manually reset env->liveness->cur_instance whenever the verifier changes env->cur_state call stack:
- call bpf_reset_live_stack_callchain() when the verifier enters a

    subprogram;

- call bpf_update_live_stack() when the verifier exits a subprogram (it implies the reset).

Make sure bpf_update_live_stack() is called for a callchain before issuing liveness queries. And make sure that bpf_update_live_stack() is called for any callee callchain first:
- Add bpf_update_live_stack() call at every location that processes BPF_EXIT:
- exit from a subprogram;
- before pop_stack() call.
This makes sure that bpf_update_live_stack() is called for callee callchains before caller callchains.

Make sure must_write marks are set to zero for instructions that do not always access the stack:
- Wrap do_check_insn() with bpf_reset_stack_write_marks() / bpf_commit_stack_write_marks() calls.
Any calls to bpf_mark_stack_write() are accumulated between this pair of calls. If no bpf_mark_stack_write() calls were made it means that the instruction does not access stack (at-least on the current verification path) and it is important to record this fact.

Finally, use bpf_live_stack_query_init() / bpf_stack_slot_alive() to query stack liveness info.

The manual tracking of the correct order for callee/caller bpf_update_live_stack() calls is a bit convoluted and may warrant some automation in future revisions.

---
 kernel/bpf/verifier.c | 61 +++++++++++++++++++++++++++++++++++++------
 1 file changed, 53 insertions(+), 8 deletions(-)
kernel/bpf/verifier.c +53 -8
index bdcc20d2fab6..33cb8beb8706 100644
@@ -789,6 +789,7 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_
 
 	state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN;
 	state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
+	bpf_mark_stack_write(env, state->frameno, BIT(spi - 1) | BIT(spi));
 
 	return 0;
 }
@@ -828,6 +829,7 @@ static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_stat
 	 */
 	state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN;
 	state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
+	bpf_mark_stack_write(env, state->frameno, BIT(spi - 1) | BIT(spi));
 }
 
 static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg)
@@ -939,6 +941,7 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env,
 	/* Same reason as unmark_stack_slots_dynptr above */
 	state->stack[spi].spilled_ptr.live |= REG_LIVE_WRITTEN;
 	state->stack[spi - 1].spilled_ptr.live |= REG_LIVE_WRITTEN;
+	bpf_mark_stack_write(env, state->frameno, BIT(spi - 1) | BIT(spi));
 
 	return 0;
 }
@@ -1066,6 +1069,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
 		for (j = 0; j < BPF_REG_SIZE; j++)
 			slot->slot_type[j] = STACK_ITER;
 
+		bpf_mark_stack_write(env, state->frameno, BIT(spi - i));
 		mark_stack_slot_scratched(env, spi - i);
 	}
 
@@ -1097,6 +1101,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env,
 		for (j = 0; j < BPF_REG_SIZE; j++)
 			slot->slot_type[j] = STACK_INVALID;
 
+		bpf_mark_stack_write(env, state->frameno, BIT(spi - i));
 		mark_stack_slot_scratched(env, spi - i);
 	}
 
@@ -1186,6 +1191,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env,
 	slot = &state->stack[spi];
 	st = &slot->spilled_ptr;
 
+	bpf_mark_stack_write(env, reg->frameno, BIT(spi));
 	__mark_reg_known_zero(st);
 	st->type = PTR_TO_STACK; /* we don't have dedicated reg type */
 	st->live |= REG_LIVE_WRITTEN;
@@ -1244,6 +1250,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r
 
 	/* see unmark_stack_slots_dynptr() for why we need to set REG_LIVE_WRITTEN */
 	st->live |= REG_LIVE_WRITTEN;
+	bpf_mark_stack_write(env, reg->frameno, BIT(spi));
 
 	for (i = 0; i < BPF_REG_SIZE; i++)
 		slot->slot_type[i] = STACK_INVALID;
@@ -3619,6 +3626,9 @@ static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_reg
 		if (err)
 			return err;
 
+		err = bpf_mark_stack_read(env, reg->frameno, env->insn_idx, BIT(spi - i));
+		if (err)
+			return err;
 		mark_stack_slot_scratched(env, spi - i);
 	}
 	return 0;
@@ -5151,6 +5161,18 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
 	if (err)
 		return err;
 
+	if (!(off % BPF_REG_SIZE) && size == BPF_REG_SIZE) {
+		/* only mark the slot as written if all 8 bytes were written
+		 * otherwise read propagation may incorrectly stop too soon
+		 * when stack slots are partially written.
+		 * This heuristic means that read propagation will be
+		 * conservative, since it will add reg_live_read marks
+		 * to stack slots all the way to first state when programs
+		 * writes+reads less than 8 bytes
+		 */
+		bpf_mark_stack_write(env, state->frameno, BIT(spi));
+	}
+
 	check_fastcall_stack_contract(env, state, insn_idx, off);
 	mark_stack_slot_scratched(env, spi);
 	if (reg && !(off % BPF_REG_SIZE) && reg->type == SCALAR_VALUE && env->bpf_capable) {
@@ -5420,12 +5442,16 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
 	struct bpf_reg_state *reg;
 	u8 *stype, type;
 	int insn_flags = insn_stack_access_flags(reg_state->frameno, spi);
+	int err;
 
 	stype = reg_state->stack[spi].slot_type;
 	reg = &reg_state->stack[spi].spilled_ptr;
 
 	mark_stack_slot_scratched(env, spi);
 	check_fastcall_stack_contract(env, state, env->insn_idx, off);
+	err = bpf_mark_stack_read(env, reg_state->frameno, env->insn_idx, BIT(spi));
+	if (err)
+		return err;
 
 	if (is_spilled_reg(&reg_state->stack[spi])) {
 		u8 spill_size = 1;
@@ -8159,6 +8185,9 @@ static int check_stack_range_initialized(
 		mark_reg_read(env, &state->stack[spi].spilled_ptr,
 			      state->stack[spi].spilled_ptr.parent,
 			      REG_LIVE_READ64);
+		err = bpf_mark_stack_read(env, reg->frameno, env->insn_idx, BIT(spi));
+		if (err)
+			return err;
 		/* We do not set REG_LIVE_WRITTEN for stack slot, as we can not
 		 * be sure that whether stack slot is written to or not. Hence,
 		 * we must still conservatively propagate reads upwards even if
@@ -10716,6 +10745,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	/* and go analyze first insn of the callee */
 	*insn_idx = env->subprog_info[subprog].start - 1;
 
+	bpf_reset_live_stack_callchain(env);
+
 	if (env->log.level & BPF_LOG_LEVEL) {
 		verbose(env, "caller:\n");
 		print_verifier_state(env, state, caller->frameno, true);
@@ -18502,7 +18533,6 @@ static void clean_func_state(struct bpf_verifier_env *env,
 			     u32 ip)
 {
 	u16 live_regs = env->insn_aux_data[ip].live_regs_before;
-	enum bpf_reg_liveness live;
 	int i, j;
 
 	for (i = 0; i < BPF_REG_FP; i++) {
@@ -18515,9 +18545,7 @@ static void clean_func_state(struct bpf_verifier_env *env,
 	}
 
 	for (i = 0; i < st->allocated_stack / BPF_REG_SIZE; i++) {
-		live = st->stack[i].spilled_ptr.live;
-		/* liveness must not touch this stack slot anymore */
-		if (!(live & REG_LIVE_READ)) {
+		if (!bpf_stack_slot_alive(env, st->frameno, i)) {
 			__mark_reg_not_init(env, &st->stack[i].spilled_ptr);
 			for (j = 0; j < BPF_REG_SIZE; j++)
 				st->stack[i].slot_type[j] = STACK_INVALID;
@@ -18530,6 +18558,7 @@ static void clean_verifier_state(struct bpf_verifier_env *env,
 {
 	int i, ip;
 
+	bpf_live_stack_query_init(env, st);
 	st->cleaned = true;
 	for (i = 0; i <= st->curframe; i++) {
 		ip = frame_insn_idx(st, i);
@@ -18615,9 +18644,6 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
 	if (exact == EXACT)
 		return regs_exact(rold, rcur, idmap);
 
-	if (!(rold->live & REG_LIVE_READ) && exact == NOT_EXACT)
-		/* explored state didn't use this */
-		return true;
 	if (rold->type == NOT_INIT) {
 		if (exact == NOT_EXACT || rcur->type == NOT_INIT)
 			/* explored state can't have used this */
@@ -19856,6 +19882,9 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 		return PROCESS_BPF_EXIT;
 
 	if (env->cur_state->curframe) {
+		err = bpf_update_live_stack(env);
+		if (err)
+			return err;
 		/* exit from nested function */
 		err = prepare_func_exit(env, &env->insn_idx);
 		if (err)
@@ -20041,7 +20070,7 @@ static int do_check(struct bpf_verifier_env *env)
 	for (;;) {
 		struct bpf_insn *insn;
 		struct bpf_insn_aux_data *insn_aux;
-		int err;
+		int err, marks_err;
 
 		/* reset current history entry on each new instruction */
 		env->cur_hist_ent = NULL;
@@ -20134,7 +20163,15 @@ static int do_check(struct bpf_verifier_env *env)
 		if (state->speculative && insn_aux->nospec)
 			goto process_bpf_exit;
 
+		err = bpf_reset_stack_write_marks(env, env->insn_idx);
+		if (err)
+			return err;
 		err = do_check_insn(env, &do_print_state);
+		if (err >= 0 || error_recoverable_with_nospec(err)) {
+			marks_err = bpf_commit_stack_write_marks(env);
+			if (marks_err)
+				return marks_err;
+		}
 		if (error_recoverable_with_nospec(err) && state->speculative) {
 			/* Prevent this speculative path from ever reaching the
 			 * insn that would have been unsafe to execute.
@@ -20173,6 +20210,9 @@ static int do_check(struct bpf_verifier_env *env)
 process_bpf_exit:
 			mark_verifier_state_scratched(env);
 			err = update_branch_counts(env, env->cur_state);
+			if (err)
+				return err;
+			err = bpf_update_live_stack(env);
 			if (err)
 				return err;
 			err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
@@ -24733,6 +24773,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 	if (ret < 0)
 		goto skip_full_check;
 
+	ret = bpf_stack_liveness_init(env);
+	if (ret)
+		goto skip_full_check;
+
 	ret = check_attach_btf_id(env);
 	if (ret)
 		goto skip_full_check;
@@ -24882,6 +24926,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
 		mutex_unlock(&bpf_verifier_lock);
 	vfree(env->insn_aux_data);
 err_free_env:
+	bpf_stack_liveness_free(env);
 	kvfree(env->cfg.insn_postorder);
 	kvfree(env->scc_info);
 	kvfree(env);
Signature · 1 line
2.47.3
author maintainer
#8 lore raw

PATCH 8/10 bpf: signal error if old liveness is more conservative than new

In reply to Eduard Zingerman

Unlike the new algorithm, register chain based liveness tracking is fully path sensitive, and thus should be strictly more accurate. Validate the new algorithm by signaling an error whenever it considers a stack slot dead while the old algorithm considers it alive.

---
 include/linux/bpf_verifier.h | 1 +
 kernel/bpf/verifier.c        | 9 +++++++++
 2 files changed, 10 insertions(+)
include/linux/bpf_verifier.h +1 -0
index 2e3bdd50e2ba..dec5da3a2e59 100644
@@ -852,6 +852,7 @@ struct bpf_verifier_env {
 	/* array of pointers to bpf_scc_info indexed by SCC id */
 	struct bpf_scc_info **scc_info;
 	u32 scc_cnt;
+	bool internal_error;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
kernel/bpf/verifier.c +9 -0
index 33cb8beb8706..07115f8b9e5f 100644
@@ -18546,6 +18546,11 @@ static void clean_func_state(struct bpf_verifier_env *env,
 
 	for (i = 0; i < st->allocated_stack / BPF_REG_SIZE; i++) {
 		if (!bpf_stack_slot_alive(env, st->frameno, i)) {
+			if (st->stack[i].spilled_ptr.live & REG_LIVE_READ) {
+				verifier_bug(env, "incorrect live marks #1 for insn %d frameno %d spi %d\n",
+					     env->insn_idx, st->frameno, i);
+				env->internal_error = true;
+			}
 			__mark_reg_not_init(env, &st->stack[i].spilled_ptr);
 			for (j = 0; j < BPF_REG_SIZE; j++)
 				st->stack[i].slot_type[j] = STACK_INVALID;
@@ -19516,6 +19521,8 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 		loop = incomplete_read_marks(env, &sl->state);
 		if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
 hit:
+			if (env->internal_error)
+				return -EFAULT;
 			sl->hit_cnt++;
 			/* reached equivalent register/stack state,
 			 * prune the search.
@@ -19630,6 +19637,8 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
 			return 1;
 		}
 miss:
+		if (env->internal_error)
+			return -EFAULT;
 		/* when new state is not going to be added do not increase miss count.
 		 * Otherwise several loop iterations will remove the state
 		 * recorded earlier. The goal of these heuristics is to have
Signature · 1 line
2.47.3
author maintainer
#9 lore raw

PATCH 9/10 bpf: disable and remove registers chain based liveness

In reply to Eduard Zingerman

Remove register chain based liveness tracking:
- struct bpf_reg_state->{parent,live} fields are no longer needed;
- REG_LIVE_WRITTEN marks are superseded by bpf_mark_stack_write() calls;
- mark_reg_read() calls are superseded by bpf_mark_stack_read();
- log.c:print_liveness() is superseded by logging in liveness.c;
- propagate_liveness() is superseded by bpf_update_live_stack();
- no need to establish register chains in is_state_visited() anymore;
- fix a bunch of tests expecting "_w" suffixes in verifier log messages.

---
 Documentation/bpf/verifier.rst                | 264 ---------------
 include/linux/bpf_verifier.h                  |  25 --
 kernel/bpf/log.c                              |  26 +-
 kernel/bpf/verifier.c                         | 312 ++----------------
 .../testing/selftests/bpf/prog_tests/align.c  | 178 +++++-----
 .../selftests/bpf/prog_tests/spin_lock.c      |  12 +-
 .../selftests/bpf/prog_tests/test_veristat.c  |  44 +--
 .../selftests/bpf/progs/exceptions_assert.c   |  34 +-
 .../selftests/bpf/progs/iters_state_safety.c  |   4 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |   6 +-
 .../bpf/progs/mem_rdonly_untrusted.c          |   4 +-
 .../selftests/bpf/progs/verifier_bounds.c     |  38 +--
 .../bpf/progs/verifier_global_ptr_args.c      |   4 +-
 .../selftests/bpf/progs/verifier_ldsx.c       |   2 +-
 .../selftests/bpf/progs/verifier_precision.c  |  16 +-
 .../selftests/bpf/progs/verifier_scalar_ids.c |  10 +-
 .../selftests/bpf/progs/verifier_spill_fill.c |  40 +--
 .../bpf/progs/verifier_subprog_precision.c    |   6 +-
 .../selftests/bpf/verifier/bpf_st_mem.c       |   4 +-
 19 files changed, 224 insertions(+), 805 deletions(-)
Documentation/bpf/verifier.rst +0 -264
index 95e6f80a407e..510d15bc697b 100644
@@ -347,270 +347,6 @@ However, only the value of register ``r1`` is important to successfully finish
 verification. The goal of the liveness tracking algorithm is to spot this fact
 and figure out that both states are actually equivalent.
 
-Data structures
-~~~~~~~~~~~~~~~
-
-Liveness is tracked using the following data structures::
-
-  enum bpf_reg_liveness {
-	REG_LIVE_NONE = 0,
-	REG_LIVE_READ32 = 0x1,
-	REG_LIVE_READ64 = 0x2,
-	REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64,
-	REG_LIVE_WRITTEN = 0x4,
-	REG_LIVE_DONE = 0x8,
-  };
-
-  struct bpf_reg_state {
- 	...
-	struct bpf_reg_state *parent;
- 	...
-	enum bpf_reg_liveness live;
- 	...
-  };
-
-  struct bpf_stack_state {
-	struct bpf_reg_state spilled_ptr;
-	...
-  };
-
-  struct bpf_func_state {
-	struct bpf_reg_state regs[MAX_BPF_REG];
-        ...
-	struct bpf_stack_state *stack;
-  }
-
-  struct bpf_verifier_state {
-	struct bpf_func_state *frame[MAX_CALL_FRAMES];
-	struct bpf_verifier_state *parent;
-        ...
-  }
-
-* ``REG_LIVE_NONE`` is an initial value assigned to ``->live`` fields upon new
-  verifier state creation;
-
-* ``REG_LIVE_WRITTEN`` means that the value of the register (or stack slot) is
-  defined by some instruction verified between this verifier state's parent and
-  verifier state itself;
-
-* ``REG_LIVE_READ{32,64}`` means that the value of the register (or stack slot)
-  is read by a some child state of this verifier state;
-
-* ``REG_LIVE_DONE`` is a marker used by ``clean_verifier_state()`` to avoid
-  processing same verifier state multiple times and for some sanity checks;
-
-* ``->live`` field values are formed by combining ``enum bpf_reg_liveness``
-  values using bitwise or.
-
-Register parentage chains
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In order to propagate information between parent and child states, a *register
-parentage chain* is established. Each register or stack slot is linked to a
-corresponding register or stack slot in its parent state via a ``->parent``
-pointer. This link is established upon state creation in ``is_state_visited()``
-and might be modified by ``set_callee_state()`` called from
-``__check_func_call()``.
-
-The rules for correspondence between registers / stack slots are as follows:
-
-* For the current stack frame, registers and stack slots of the new state are
-  linked to the registers and stack slots of the parent state with the same
-  indices.
-
-* For the outer stack frames, only callee saved registers (r6-r9) and stack
-  slots are linked to the registers and stack slots of the parent state with the
-  same indices.
-
-* When function call is processed a new ``struct bpf_func_state`` instance is
-  allocated, it encapsulates a new set of registers and stack slots. For this
-  new frame, parent links for r6-r9 and stack slots are set to nil, parent links
-  for r1-r5 are set to match caller r1-r5 parent links.
-
-This could be illustrated by the following diagram (arrows stand for
-``->parent`` pointers)::
-
-      ...                    ; Frame #0, some instructions
-  --- checkpoint #0 ---
-  1 : r6 = 42                ; Frame #0
-  --- checkpoint #1 ---
-  2 : call foo()             ; Frame #0
-      ...                    ; Frame #1, instructions from foo()
-  --- checkpoint #2 ---
-      ...                    ; Frame #1, instructions from foo()
-  --- checkpoint #3 ---
-      exit                   ; Frame #1, return from foo()
-  3 : r1 = r6                ; Frame #0  <- current state
-
-             +-------------------------------+-------------------------------+
-             |           Frame #0            |           Frame #1            |
-  Checkpoint +-------------------------------+-------------------------------+
-  #0         | r0 | r1-r5 | r6-r9 | fp-8 ... |
-             +-------------------------------+
-                ^    ^       ^       ^
-                |    |       |       |
-  Checkpoint +-------------------------------+
-  #1         | r0 | r1-r5 | r6-r9 | fp-8 ... |
-             +-------------------------------+
-                     ^       ^       ^
-                     |_______|_______|_______________
-                             |       |               |
-               nil  nil      |       |               |      nil     nil
-                |    |       |       |               |       |       |
-  Checkpoint +-------------------------------+-------------------------------+
-  #2         | r0 | r1-r5 | r6-r9 | fp-8 ... | r0 | r1-r5 | r6-r9 | fp-8 ... |
-             +-------------------------------+-------------------------------+
-                             ^       ^               ^       ^       ^
-               nil  nil      |       |               |       |       |
-                |    |       |       |               |       |       |
-  Checkpoint +-------------------------------+-------------------------------+
-  #3         | r0 | r1-r5 | r6-r9 | fp-8 ... | r0 | r1-r5 | r6-r9 | fp-8 ... |
-             +-------------------------------+-------------------------------+
-                             ^       ^
-               nil  nil      |       |
-                |    |       |       |
-  Current    +-------------------------------+
-  state      | r0 | r1-r5 | r6-r9 | fp-8 ... |
-             +-------------------------------+
-                             \
-                               r6 read mark is propagated via these links
-                               all the way up to checkpoint #1.
-                               The checkpoint #1 contains a write mark for r6
-                               because of instruction (1), thus read propagation
-                               does not reach checkpoint #0 (see section below).
-
-Liveness marks tracking
-~~~~~~~~~~~~~~~~~~~~~~~
-
-For each processed instruction, the verifier tracks read and written registers
-and stack slots. The main idea of the algorithm is that read marks propagate
-back along the state parentage chain until they hit a write mark, which 'screens
-off' earlier states from the read. The information about reads is propagated by
-function ``mark_reg_read()`` which could be summarized as follows::
-
-  mark_reg_read(struct bpf_reg_state *state, ...):
-      parent = state->parent
-      while parent:
-          if state->live & REG_LIVE_WRITTEN:
-              break
-          if parent->live & REG_LIVE_READ64:
-              break
-          parent->live |= REG_LIVE_READ64
-          state = parent
-          parent = state->parent
-
-Notes:
-
-* The read marks are applied to the **parent** state while write marks are
-  applied to the **current** state. The write mark on a register or stack slot
-  means that it is updated by some instruction in the straight-line code leading
-  from the parent state to the current state.
-
-* Details about REG_LIVE_READ32 are omitted.
-
-* Function ``propagate_liveness()`` (see section :ref:`read_marks_for_cache_hits`)
-  might override the first parent link. Please refer to the comments in the
-  ``propagate_liveness()`` and ``mark_reg_read()`` source code for further
-  details.
-
-Because stack writes could have different sizes ``REG_LIVE_WRITTEN`` marks are
-applied conservatively: stack slots are marked as written only if write size
-corresponds to the size of the register, e.g. see function ``save_register_state()``.
-
-Consider the following example::
-
-  0: (*u64)(r10 - 8) = 0   ; define 8 bytes of fp-8
-  --- checkpoint #0 ---
-  1: (*u32)(r10 - 8) = 1   ; redefine lower 4 bytes
-  2: r1 = (*u32)(r10 - 8)  ; read lower 4 bytes defined at (1)
-  3: r2 = (*u32)(r10 - 4)  ; read upper 4 bytes defined at (0)
-
-As stated above, the write at (1) does not count as ``REG_LIVE_WRITTEN``. Should
-it be otherwise, the algorithm above wouldn't be able to propagate the read mark
-from (3) to checkpoint #0.
-
-Once the ``BPF_EXIT`` instruction is reached ``update_branch_counts()`` is
-called to update the ``->branches`` counter for each verifier state in a chain
-of parent verifier states. When the ``->branches`` counter reaches zero the
-verifier state becomes a valid entry in a set of cached verifier states.
-
-Each entry of the verifier states cache is post-processed by a function
-``clean_live_states()``. This function marks all registers and stack slots
-without ``REG_LIVE_READ{32,64}`` marks as ``NOT_INIT`` or ``STACK_INVALID``.
-Registers/stack slots marked in this way are ignored in function ``stacksafe()``
-called from ``states_equal()`` when a state cache entry is considered for
-equivalence with a current state.
-
-Now it is possible to explain how the example from the beginning of the section
-works::
-
-  0: call bpf_get_prandom_u32()
-  1: r1 = 0
-  2: if r0 == 0 goto +1
-  3: r0 = 1
-  --- checkpoint[0] ---
-  4: r0 = r1
-  5: exit
-
-* At instruction #2 branching point is reached and state ``{ r0 == 0, r1 == 0, pc == 4 }``
-  is pushed to states processing queue (pc stands for program counter).
-
-* At instruction #4:
-
-  * ``checkpoint[0]`` states cache entry is created: ``{ r0 == 1, r1 == 0, pc == 4 }``;
-  * ``checkpoint[0].r0`` is marked as written;
-  * ``checkpoint[0].r1`` is marked as read;
-
-* At instruction #5 exit is reached and ``checkpoint[0]`` can now be processed
-  by ``clean_live_states()``. After this processing ``checkpoint[0].r1`` has a
-  read mark and all other registers and stack slots are marked as ``NOT_INIT``
-  or ``STACK_INVALID``
-
-* The state ``{ r0 == 0, r1 == 0, pc == 4 }`` is popped from the states queue
-  and is compared against a cached state ``{ r1 == 0, pc == 4 }``, the states
-  are considered equivalent.
-
-.. _read_marks_for_cache_hits:
-
-Read marks propagation for cache hits
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Another point is the handling of read marks when a previously verified state is
-found in the states cache. Upon cache hit verifier must behave in the same way
-as if the current state was verified to the program exit. This means that all
-read marks, present on registers and stack slots of the cached state, must be
-propagated over the parentage chain of the current state. Example below shows
-why this is important. Function ``propagate_liveness()`` handles this case.
-
-Consider the following state parentage chain (S is a starting state, A-E are
-derived states, -> arrows show which state is derived from which)::
-
-                   r1 read
-            <-------------                A[r1] == 0
-                                          C[r1] == 0
-      S ---> A ---> B ---> exit           E[r1] == 1
-      |
-      ` ---> C ---> D
-      |
-      ` ---> E      ^
-                    |___   suppose all these
-             ^           states are at insn #Y
-             |
-      suppose all these
-    states are at insn #X
-
-* Chain of states ``S -> A -> B -> exit`` is verified first.
-
-* While ``B -> exit`` is verified, register ``r1`` is read and this read mark is
-  propagated up to state ``A``.
-
-* When chain of states ``C -> D`` is verified the state ``D`` turns out to be
-  equivalent to state ``B``.
-
-* The read mark for ``r1`` has to be propagated to state ``C``, otherwise state
-  ``C`` might get mistakenly marked as equivalent to state ``E`` even though
-  values for register ``r1`` differ between ``C`` and ``E``.
-
 Understanding eBPF verifier messages
 ====================================
 
include/linux/bpf_verifier.h +0 -25
index dec5da3a2e59..c7515da8500c 100644
@@ -26,27 +26,6 @@
 /* Patch buffer size */
 #define INSN_BUF_SIZE 32
 
-/* Liveness marks, used for registers and spilled-regs (in stack slots).
- * Read marks propagate upwards until they find a write mark; they record that
- * "one of this state's descendants read this reg" (and therefore the reg is
- * relevant for states_equal() checks).
- * Write marks collect downwards and do not propagate; they record that "the
- * straight-line code that reached this state (from its parent) wrote this reg"
- * (and therefore that reads propagated from this state or its descendants
- * should not propagate to its parent).
- * A state with a write mark can receive read marks; it just won't propagate
- * them to its parent, since the write mark is a property, not of the state,
- * but of the link between it and its parent.  See mark_reg_read() and
- * mark_stack_slot_read() in kernel/bpf/verifier.c.
- */
-enum bpf_reg_liveness {
-	REG_LIVE_NONE = 0, /* reg hasn't been read or written this branch */
-	REG_LIVE_READ32 = 0x1, /* reg was read, so we're sensitive to initial value */
-	REG_LIVE_READ64 = 0x2, /* likewise, but full 64-bit content matters */
-	REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64,
-	REG_LIVE_WRITTEN = 0x4, /* reg was written first, screening off later reads */
-};
-
 #define ITER_PREFIX "bpf_iter_"
 
 enum bpf_iter_state {
@@ -211,8 +190,6 @@ struct bpf_reg_state {
 	 * allowed and has the same effect as bpf_sk_release(sk).
 	 */
 	u32 ref_obj_id;
-	/* parentage chain for liveness checking */
-	struct bpf_reg_state *parent;
 	/* Inside the callee two registers can be both PTR_TO_STACK like
 	 * R1=fp-8 and R2=fp-8, but one of them points to this function stack
 	 * while another to the caller's stack. To differentiate them 'frameno'
@@ -225,7 +202,6 @@ struct bpf_reg_state {
 	 * patching which only happens after main verification finished.
 	 */
 	s32 subreg_def;
-	enum bpf_reg_liveness live;
 	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
 	bool precise;
 };
@@ -852,7 +828,6 @@ struct bpf_verifier_env {
 	/* array of pointers to bpf_scc_info indexed by SCC id */
 	struct bpf_scc_info **scc_info;
 	u32 scc_cnt;
-	bool internal_error;
 };
 
 static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
kernel/bpf/log.c +4 -22
index 0d6d7bfb2fd0..f50533169cc3 100644
@@ -542,17 +542,6 @@ static char slot_type_char[] = {
 	[STACK_IRQ_FLAG] = 'f'
 };
 
-static void print_liveness(struct bpf_verifier_env *env,
-			   enum bpf_reg_liveness live)
-{
-	if (live & (REG_LIVE_READ | REG_LIVE_WRITTEN))
-		verbose(env, "_");
-	if (live & REG_LIVE_READ)
-		verbose(env, "r");
-	if (live & REG_LIVE_WRITTEN)
-		verbose(env, "w");
-}
-
 #define UNUM_MAX_DECIMAL U16_MAX
 #define SNUM_MAX_DECIMAL S16_MAX
 #define SNUM_MIN_DECIMAL S16_MIN
@@ -770,7 +759,6 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 		if (!print_all && !reg_scratched(env, i))
 			continue;
 		verbose(env, " R%d", i);
-		print_liveness(env, reg->live);
 		verbose(env, "=");
 		print_reg_state(env, state, reg);
 	}
@@ -803,9 +791,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 					break;
 			types_buf[j] = '\0';
 
-			verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
-			print_liveness(env, reg->live);
-			verbose(env, "=%s", types_buf);
+			verbose(env, " fp%d=%s", (-i - 1) * BPF_REG_SIZE, types_buf);
 			print_reg_state(env, state, reg);
 			break;
 		case STACK_DYNPTR:
@@ -814,7 +800,6 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 			reg = &state->stack[i].spilled_ptr;
 
 			verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
-			print_liveness(env, reg->live);
 			verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type));
 			if (reg->id)
 				verbose_a("id=%d", reg->id);
@@ -829,9 +814,8 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 			if (!reg->ref_obj_id)
 				continue;
 
-			verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
-			print_liveness(env, reg->live);
-			verbose(env, "=iter_%s(ref_id=%d,state=%s,depth=%u)",
+			verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)",
+				(-i - 1) * BPF_REG_SIZE,
 				iter_type_str(reg->iter.btf, reg->iter.btf_id),
 				reg->ref_obj_id, iter_state_str(reg->iter.state),
 				reg->iter.depth);
@@ -839,9 +823,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
 		case STACK_MISC:
 		case STACK_ZERO:
 		default:
-			verbose(env, " fp%d", (-i - 1) * BPF_REG_SIZE);
-			print_liveness(env, reg->live);
-			verbose(env, "=%s", types_buf);
+			verbose(env, " fp%d=%s", (-i - 1) * BPF_REG_SIZE, types_buf);
 			break;
 		}
 	}
kernel/bpf/verifier.c +19 -293

567 lines not on this page · raw message

tools/testing/selftests/bpf/prog_tests/align.c +89 -89

309 lines not on this page · raw message

tools/testing/selftests/bpf/prog_tests/spin_lock.c +6 -6
index e3ea5dc2f697..254fbfeab06a 100644
@@ -13,22 +13,22 @@ static struct {
 	const char *err_msg;
 } spin_lock_fail_tests[] = {
 	{ "lock_id_kptr_preserve",
-	  "5: (bf) r1 = r0                       ; R0_w=ptr_foo(id=2,ref_obj_id=2) "
-	  "R1_w=ptr_foo(id=2,ref_obj_id=2) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n"
+	  "5: (bf) r1 = r0                       ; R0=ptr_foo(id=2,ref_obj_id=2) "
+	  "R1=ptr_foo(id=2,ref_obj_id=2) refs=2\n6: (85) call bpf_this_cpu_ptr#154\n"
 	  "R1 type=ptr_ expected=percpu_ptr_" },
 	{ "lock_id_global_zero",
-	  "; R1_w=map_value(map=.data.A,ks=4,vs=4)\n2: (85) call bpf_this_cpu_ptr#154\n"
+	  "; R1=map_value(map=.data.A,ks=4,vs=4)\n2: (85) call bpf_this_cpu_ptr#154\n"
 	  "R1 type=map_value expected=percpu_ptr_" },
 	{ "lock_id_mapval_preserve",
 	  "[0-9]\\+: (bf) r1 = r0                       ;"
-	  " R0_w=map_value(id=1,map=array_map,ks=4,vs=8)"
-	  " R1_w=map_value(id=1,map=array_map,ks=4,vs=8)\n"
+	  " R0=map_value(id=1,map=array_map,ks=4,vs=8)"
+	  " R1=map_value(id=1,map=array_map,ks=4,vs=8)\n"
 	  "[0-9]\\+: (85) call bpf_this_cpu_ptr#154\n"
 	  "R1 type=map_value expected=percpu_ptr_" },
 	{ "lock_id_innermapval_preserve",
 	  "[0-9]\\+: (bf) r1 = r0                      ;"
 	  " R0=map_value(id=2,ks=4,vs=8)"
-	  " R1_w=map_value(id=2,ks=4,vs=8)\n"
+	  " R1=map_value(id=2,ks=4,vs=8)\n"
 	  "[0-9]\\+: (85) call bpf_this_cpu_ptr#154\n"
 	  "R1 type=map_value expected=percpu_ptr_" },
 	{ "lock_id_mismatch_kptr_kptr", "bpf_spin_unlock of different lock" },
tools/testing/selftests/bpf/prog_tests/test_veristat.c +22 -22

56 lines not on this page · raw message

tools/testing/selftests/bpf/progs/exceptions_assert.c +17 -17
index 5e0a1ca96d4e..a01c2736890f 100644
@@ -18,43 +18,43 @@
 		return *(u64 *)num;					\
 	}
 
-__msg(": R0_w=0xffffffff80000000")
+__msg(": R0=0xffffffff80000000")
 check_assert(s64, ==, eq_int_min, INT_MIN);
-__msg(": R0_w=0x7fffffff")
+__msg(": R0=0x7fffffff")
 check_assert(s64, ==, eq_int_max, INT_MAX);
-__msg(": R0_w=0")
+__msg(": R0=0")
 check_assert(s64, ==, eq_zero, 0);
-__msg(": R0_w=0x8000000000000000 R1_w=0x8000000000000000")
+__msg(": R0=0x8000000000000000 R1=0x8000000000000000")
 check_assert(s64, ==, eq_llong_min, LLONG_MIN);
-__msg(": R0_w=0x7fffffffffffffff R1_w=0x7fffffffffffffff")
+__msg(": R0=0x7fffffffffffffff R1=0x7fffffffffffffff")
 check_assert(s64, ==, eq_llong_max, LLONG_MAX);
 
-__msg(": R0_w=scalar(id=1,smax=0x7ffffffe)")
+__msg(": R0=scalar(id=1,smax=0x7ffffffe)")
 check_assert(s64, <, lt_pos, INT_MAX);
-__msg(": R0_w=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
+__msg(": R0=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
 check_assert(s64, <, lt_zero, 0);
-__msg(": R0_w=scalar(id=1,smax=0xffffffff7fffffff")
+__msg(": R0=scalar(id=1,smax=0xffffffff7fffffff")
 check_assert(s64, <, lt_neg, INT_MIN);
 
-__msg(": R0_w=scalar(id=1,smax=0x7fffffff)")
+__msg(": R0=scalar(id=1,smax=0x7fffffff)")
 check_assert(s64, <=, le_pos, INT_MAX);
-__msg(": R0_w=scalar(id=1,smax=0)")
+__msg(": R0=scalar(id=1,smax=0)")
 check_assert(s64, <=, le_zero, 0);
-__msg(": R0_w=scalar(id=1,smax=0xffffffff80000000")
+__msg(": R0=scalar(id=1,smax=0xffffffff80000000")
 check_assert(s64, <=, le_neg, INT_MIN);
 
-__msg(": R0_w=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg(": R0=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >, gt_pos, INT_MAX);
-__msg(": R0_w=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg(": R0=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >, gt_zero, 0);
-__msg(": R0_w=scalar(id=1,smin=0xffffffff80000001")
+__msg(": R0=scalar(id=1,smin=0xffffffff80000001")
 check_assert(s64, >, gt_neg, INT_MIN);
 
-__msg(": R0_w=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg(": R0=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >=, ge_pos, INT_MAX);
-__msg(": R0_w=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg(": R0=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
 check_assert(s64, >=, ge_zero, 0);
-__msg(": R0_w=scalar(id=1,smin=0xffffffff80000000")
+__msg(": R0=scalar(id=1,smin=0xffffffff80000000")
 check_assert(s64, >=, ge_neg, INT_MIN);
 
 SEC("?tc")
tools/testing/selftests/bpf/progs/iters_state_safety.c +2 -2
index b381ac0c736c..d273b46dfc7c 100644
@@ -30,7 +30,7 @@ int force_clang_to_emit_btf_for_externs(void *ctx)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("fp-8_w=iter_num(ref_id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(ref_id=1,state=active,depth=0)")
 int create_and_destroy(void *ctx)
 {
 	struct bpf_iter_num iter;
@@ -196,7 +196,7 @@ int leak_iter_from_subprog_fail(void *ctx)
 
 SEC("?raw_tp")
 __success __log_level(2)
-__msg("fp-8_w=iter_num(ref_id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(ref_id=1,state=active,depth=0)")
 int valid_stack_reuse(void *ctx)
 {
 	struct bpf_iter_num iter;
tools/testing/selftests/bpf/progs/iters_testmod_seq.c +3 -3
index 6543d5b6e0a9..83791348bed5 100644
@@ -20,7 +20,7 @@ __s64 res_empty;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)")
+__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)")
 __msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_empty(const void *ctx)
@@ -38,7 +38,7 @@ __s64 res_full;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)")
+__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)")
 __msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_full(const void *ctx)
@@ -58,7 +58,7 @@ static volatile int zero = 0;
 
 SEC("raw_tp/sys_enter")
 __success __log_level(2)
-__msg("fp-16_w=iter_testmod_seq(ref_id=1,state=active,depth=0)")
+__msg("fp-16=iter_testmod_seq(ref_id=1,state=active,depth=0)")
 __msg("fp-16=iter_testmod_seq(ref_id=1,state=drained,depth=0)")
 __msg("call bpf_iter_testmod_seq_destroy")
 int testmod_seq_truncated(const void *ctx)
tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c +2 -2
index 4f94c971ae86..3b984b6ae7c0 100644
@@ -8,8 +8,8 @@
 SEC("tp_btf/sys_enter")
 __success
 __log_level(2)
-__msg("r8 = *(u64 *)(r7 +0)          ; R7_w=ptr_nameidata(off={{[0-9]+}}) R8_w=rdonly_untrusted_mem(sz=0)")
-__msg("r9 = *(u8 *)(r8 +0)           ; R8_w=rdonly_untrusted_mem(sz=0) R9_w=scalar")
+__msg("r8 = *(u64 *)(r7 +0)          ; R7=ptr_nameidata(off={{[0-9]+}}) R8=rdonly_untrusted_mem(sz=0)")
+__msg("r9 = *(u8 *)(r8 +0)           ; R8=rdonly_untrusted_mem(sz=0) R9=scalar")
 int btf_id_to_ptr_mem(void *ctx)
 {
 	struct task_struct *task;
tools/testing/selftests/bpf/progs/verifier_bounds.c +19 -19
index fbccc20555f4..0a72e0228ea9 100644
@@ -926,7 +926,7 @@ l1_%=:	r0 = 0;						\
 SEC("socket")
 __description("bounds check for non const xor src dst")
 __success __log_level(2)
-__msg("5: (af) r0 ^= r6                      ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
+__msg("5: (af) r0 ^= r6                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
 __naked void non_const_xor_src_dst(void)
 {
 	asm volatile ("					\
@@ -947,7 +947,7 @@ __naked void non_const_xor_src_dst(void)
 SEC("socket")
 __description("bounds check for non const or src dst")
 __success __log_level(2)
-__msg("5: (4f) r0 |= r6                      ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
+__msg("5: (4f) r0 |= r6                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=431,var_off=(0x0; 0x1af))")
 __naked void non_const_or_src_dst(void)
 {
 	asm volatile ("					\
@@ -968,7 +968,7 @@ __naked void non_const_or_src_dst(void)
 SEC("socket")
 __description("bounds check for non const mul regs")
 __success __log_level(2)
-__msg("5: (2f) r0 *= r6                      ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=3825,var_off=(0x0; 0xfff))")
+__msg("5: (2f) r0 *= r6                      ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=3825,var_off=(0x0; 0xfff))")
 __naked void non_const_mul_regs(void)
 {
 	asm volatile ("					\
@@ -1241,7 +1241,7 @@ l0_%=:	r0 = 0;						\
 SEC("tc")
 __description("multiply mixed sign bounds. test 1")
 __success __log_level(2)
-__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=umin=0x1bc16d5cd4927ee1,smax=umax=0x1bc16d674ec80000,smax32=0x7ffffeff,umax32=0xfffffeff,var_off=(0x1bc16d4000000000; 0x3ffffffeff))")
+__msg("r6 *= r7 {{.*}}; R6=scalar(smin=umin=0x1bc16d5cd4927ee1,smax=umax=0x1bc16d674ec80000,smax32=0x7ffffeff,umax32=0xfffffeff,var_off=(0x1bc16d4000000000; 0x3ffffffeff))")
 __naked void mult_mixed0_sign(void)
 {
 	asm volatile (
@@ -1264,7 +1264,7 @@ __naked void mult_mixed0_sign(void)
 SEC("tc")
 __description("multiply mixed sign bounds. test 2")
 __success __log_level(2)
-__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=smin32=-100,smax=smax32=200)")
+__msg("r6 *= r7 {{.*}}; R6=scalar(smin=smin32=-100,smax=smax32=200)")
 __naked void mult_mixed1_sign(void)
 {
 	asm volatile (
@@ -1287,7 +1287,7 @@ __naked void mult_mixed1_sign(void)
 SEC("tc")
 __description("multiply negative bounds")
 __success __log_level(2)
-__msg("r6 *= r7 {{.*}}; R6_w=scalar(smin=umin=smin32=umin32=0x3ff280b0,smax=umax=smax32=umax32=0x3fff0001,var_off=(0x3ff00000; 0xf81ff))")
+__msg("r6 *= r7 {{.*}}; R6=scalar(smin=umin=smin32=umin32=0x3ff280b0,smax=umax=smax32=umax32=0x3fff0001,var_off=(0x3ff00000; 0xf81ff))")
 __naked void mult_sign_bounds(void)
 {
 	asm volatile (
@@ -1311,7 +1311,7 @@ __naked void mult_sign_bounds(void)
 SEC("tc")
 __description("multiply bounds that don't cross signed boundary")
 __success __log_level(2)
-__msg("r8 *= r6 {{.*}}; R6_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=11,var_off=(0x0; 0xb)) R8_w=scalar(smin=0,smax=umax=0x7b96bb0a94a3a7cd,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("r8 *= r6 {{.*}}; R6=scalar(smin=smin32=0,smax=umax=smax32=umax32=11,var_off=(0x0; 0xb)) R8=scalar(smin=0,smax=umax=0x7b96bb0a94a3a7cd,var_off=(0x0; 0x7fffffffffffffff))")
 __naked void mult_no_sign_crossing(void)
 {
 	asm volatile (
@@ -1331,7 +1331,7 @@ __naked void mult_no_sign_crossing(void)
 SEC("tc")
 __description("multiplication overflow, result in unbounded reg. test 1")
 __success __log_level(2)
-__msg("r6 *= r7 {{.*}}; R6_w=scalar()")
+__msg("r6 *= r7 {{.*}}; R6=scalar()")
 __naked void mult_unsign_ovf(void)
 {
 	asm volatile (
@@ -1353,7 +1353,7 @@ __naked void mult_unsign_ovf(void)
 SEC("tc")
 __description("multiplication overflow, result in unbounded reg. test 2")
 __success __log_level(2)
-__msg("r6 *= r7 {{.*}}; R6_w=scalar()")
+__msg("r6 *= r7 {{.*}}; R6=scalar()")
 __naked void mult_sign_ovf(void)
 {
 	asm volatile (
@@ -1376,7 +1376,7 @@ __naked void mult_sign_ovf(void)
 SEC("socket")
 __description("64-bit addition, all outcomes overflow")
 __success __log_level(2)
-__msg("5: (0f) r3 += r3 {{.*}} R3_w=scalar(umin=0x4000000000000000,umax=0xfffffffffffffffe)")
+__msg("5: (0f) r3 += r3 {{.*}} R3=scalar(umin=0x4000000000000000,umax=0xfffffffffffffffe)")
 __retval(0)
 __naked void add64_full_overflow(void)
 {
@@ -1396,7 +1396,7 @@ __naked void add64_full_overflow(void)
 SEC("socket")
 __description("64-bit addition, partial overflow, result in unbounded reg")
 __success __log_level(2)
-__msg("4: (0f) r3 += r3 {{.*}} R3_w=scalar()")
+__msg("4: (0f) r3 += r3 {{.*}} R3=scalar()")
 __retval(0)
 __naked void add64_partial_overflow(void)
 {
@@ -1416,7 +1416,7 @@ __naked void add64_partial_overflow(void)
 SEC("socket")
 __description("32-bit addition overflow, all outcomes overflow")
 __success __log_level(2)
-__msg("4: (0c) w3 += w3 {{.*}} R3_w=scalar(smin=umin=umin32=0x40000000,smax=umax=umax32=0xfffffffe,var_off=(0x0; 0xffffffff))")
+__msg("4: (0c) w3 += w3 {{.*}} R3=scalar(smin=umin=umin32=0x40000000,smax=umax=umax32=0xfffffffe,var_off=(0x0; 0xffffffff))")
 __retval(0)
 __naked void add32_full_overflow(void)
 {
@@ -1436,7 +1436,7 @@ __naked void add32_full_overflow(void)
 SEC("socket")
 __description("32-bit addition, partial overflow, result in unbounded u32 bounds")
 __success __log_level(2)
-__msg("4: (0c) w3 += w3 {{.*}} R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
+__msg("4: (0c) w3 += w3 {{.*}} R3=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
 __retval(0)
 __naked void add32_partial_overflow(void)
 {
@@ -1456,7 +1456,7 @@ __naked void add32_partial_overflow(void)
 SEC("socket")
 __description("64-bit subtraction, all outcomes underflow")
 __success __log_level(2)
-__msg("6: (1f) r3 -= r1 {{.*}} R3_w=scalar(umin=1,umax=0x8000000000000000)")
+__msg("6: (1f) r3 -= r1 {{.*}} R3=scalar(umin=1,umax=0x8000000000000000)")
 __retval(0)
 __naked void sub64_full_overflow(void)
 {
@@ -1477,7 +1477,7 @@ __naked void sub64_full_overflow(void)
 SEC("socket")
 __description("64-bit subtraction, partial overflow, result in unbounded reg")
 __success __log_level(2)
-__msg("3: (1f) r3 -= r2 {{.*}} R3_w=scalar()")
+__msg("3: (1f) r3 -= r2 {{.*}} R3=scalar()")
 __retval(0)
 __naked void sub64_partial_overflow(void)
 {
@@ -1496,7 +1496,7 @@ __naked void sub64_partial_overflow(void)
 SEC("socket")
 __description("32-bit subtraction overflow, all outcomes underflow")
 __success __log_level(2)
-__msg("5: (1c) w3 -= w1 {{.*}} R3_w=scalar(smin=umin=umin32=1,smax=umax=umax32=0x80000000,var_off=(0x0; 0xffffffff))")
+__msg("5: (1c) w3 -= w1 {{.*}} R3=scalar(smin=umin=umin32=1,smax=umax=umax32=0x80000000,var_off=(0x0; 0xffffffff))")
 __retval(0)
 __naked void sub32_full_overflow(void)
 {
@@ -1517,7 +1517,7 @@ __naked void sub32_full_overflow(void)
 SEC("socket")
 __description("32-bit subtraction, partial overflow, result in unbounded u32 bounds")
 __success __log_level(2)
-__msg("3: (1c) w3 -= w2 {{.*}} R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
+__msg("3: (1c) w3 -= w2 {{.*}} R3=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
 __retval(0)
 __naked void sub32_partial_overflow(void)
 {
@@ -1617,7 +1617,7 @@ l0_%=:	r0 = 0;				\
 SEC("socket")
 __description("bounds deduction cross sign boundary, positive overlap")
 __success __log_level(2) __flag(BPF_F_TEST_REG_INVARIANTS)
-__msg("3: (2d) if r0 > r1 {{.*}} R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))")
+__msg("3: (2d) if r0 > r1 {{.*}} R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=127,var_off=(0x0; 0x7f))")
 __retval(0)
 __naked void bounds_deduct_positive_overlap(void)
 {
@@ -1650,7 +1650,7 @@ l0_%=:	r0 = 0;				\
 SEC("socket")
 __description("bounds deduction cross sign boundary, two overlaps")
 __failure __flag(BPF_F_TEST_REG_INVARIANTS)
-__msg("3: (2d) if r0 > r1 {{.*}} R0_w=scalar(smin=smin32=-128,smax=smax32=127,umax=0xffffffffffffff80)")
+__msg("3: (2d) if r0 > r1 {{.*}} R0=scalar(smin=smin32=-128,smax=smax32=127,umax=0xffffffffffffff80)")
 __msg("frame pointer is read only")
 __naked void bounds_deduct_two_overlaps(void)
 {
tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c +2 -2
index 181da86ba5f0..6630a92b1b47 100644
@@ -215,7 +215,7 @@ __weak int subprog_untrusted(const volatile struct task_struct *restrict task __
 SEC("tp_btf/sys_enter")
 __success
 __log_level(2)
-__msg("r1 = {{.*}}; {{.*}}R1_w=trusted_ptr_task_struct()")
+__msg("r1 = {{.*}}; {{.*}}R1=trusted_ptr_task_struct()")
 __msg("Func#1 ('subprog_untrusted') is global and assumed valid.")
 __msg("Validating subprog_untrusted() func#1...")
 __msg(": R1=untrusted_ptr_task_struct")
@@ -278,7 +278,7 @@ __weak int subprog_enum_untrusted(enum bpf_attach_type *p __arg_untrusted)
 SEC("tp_btf/sys_enter")
 __success
 __log_level(2)
-__msg("r1 = {{.*}}; {{.*}}R1_w=trusted_ptr_task_struct()")
+__msg("r1 = {{.*}}; {{.*}}R1=trusted_ptr_task_struct()")
 __msg("Func#1 ('subprog_void_untrusted') is global and assumed valid.")
 __msg("Validating subprog_void_untrusted() func#1...")
 __msg(": R1=rdonly_untrusted_mem(sz=0)")
tools/testing/selftests/bpf/progs/verifier_ldsx.c +1 -1
index 52edee41caf6..f087ffb79f20 100644
@@ -65,7 +65,7 @@ __naked void ldsx_s32(void)
 SEC("socket")
 __description("LDSX, S8 range checking, privileged")
 __log_level(2) __success __retval(1)
-__msg("R1_w=scalar(smin=smin32=-128,smax=smax32=127)")
+__msg("R1=scalar(smin=smin32=-128,smax=smax32=127)")
 __naked void ldsx_s8_range_priv(void)
 {
 	asm volatile (
tools/testing/selftests/bpf/progs/verifier_precision.c +8 -8
index 73fee2aec698..1fe090cd6744 100644
@@ -144,21 +144,21 @@ SEC("?raw_tp")
 __success __log_level(2)
 /*
  * Without the bug fix there will be no history between "last_idx 3 first_idx 3"
- * and "parent state regs=" lines. "R0_w=6" parts are here to help anchor
+ * and "parent state regs=" lines. "R0=6" parts are here to help anchor
  * expected log messages to the one specific mark_chain_precision operation.
  *
  * This is quite fragile: if verifier checkpointing heuristic changes, this
  * might need adjusting.
  */
-__msg("2: (07) r0 += 1                       ; R0_w=6")
+__msg("2: (07) r0 += 1                       ; R0=6")
 __msg("3: (35) if r0 >= 0xa goto pc+1")
 __msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r0 stack= before 2: (07) r0 += 1")
 __msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 1")
 __msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4")
 __msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=:  R0_rw=P4")
-__msg("3: R0_w=6")
+__msg("mark_precise: frame0: parent state regs= stack=:  R0=P4")
+__msg("3: R0=6")
 __naked int state_loop_first_last_equal(void)
 {
 	asm volatile (
@@ -233,8 +233,8 @@ __naked void bpf_cond_op_not_r10(void)
 
 SEC("lsm.s/socket_connect")
 __success __log_level(2)
-__msg("0: (b7) r0 = 1                        ; R0_w=1")
-__msg("1: (84) w0 = -w0                      ; R0_w=0xffffffff")
+__msg("0: (b7) r0 = 1                        ; R0=1")
+__msg("1: (84) w0 = -w0                      ; R0=0xffffffff")
 __msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r0 stack= before 1: (84) w0 = -w0")
 __msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
@@ -268,8 +268,8 @@ __naked int bpf_neg_3(void)
 
 SEC("lsm.s/socket_connect")
 __success __log_level(2)
-__msg("0: (b7) r0 = 1                        ; R0_w=1")
-__msg("1: (87) r0 = -r0                      ; R0_w=-1")
+__msg("0: (b7) r0 = 1                        ; R0=1")
+__msg("1: (87) r0 = -r0                      ; R0=-1")
 __msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r0 stack= before 1: (87) r0 = -r0")
 __msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +5 -5
index dba3ca728f6e..c0ce690ddb68 100644
@@ -353,7 +353,7 @@ __flag(BPF_F_TEST_STATE_FREQ)
  * collect_linked_regs() can't tie more than 6 registers for a single insn.
  */
 __msg("8: (25) if r0 > 0x7 goto pc+0         ; R0=scalar(id=1")
-__msg("9: (bf) r6 = r6                       ; R6_w=scalar(id=2")
+__msg("9: (bf) r6 = r6                       ; R6=scalar(id=2")
 /* check that r{0-5} are marked precise after 'if' */
 __msg("frame0: regs=r0 stack= before 8: (25) if r0 > 0x7 goto pc+0")
 __msg("frame0: parent state regs=r0,r1,r2,r3,r4,r5 stack=:")
@@ -779,12 +779,12 @@ __success
 __retval(0)
 /* Check that verifier believes r1/r0 are zero at exit */
 __log_level(2)
-__msg("4: (77) r1 >>= 32                     ; R1_w=0")
-__msg("5: (bf) r0 = r1                       ; R0_w=0 R1_w=0")
+__msg("4: (77) r1 >>= 32                     ; R1=0")
+__msg("5: (bf) r0 = r1                       ; R0=0 R1=0")
 __msg("6: (95) exit")
 __msg("from 3 to 4")
-__msg("4: (77) r1 >>= 32                     ; R1_w=0")
-__msg("5: (bf) r0 = r1                       ; R0_w=0 R1_w=0")
+__msg("4: (77) r1 >>= 32                     ; R1=0")
+__msg("5: (bf) r0 = r1                       ; R0=0 R1=0")
 __msg("6: (95) exit")
 /* Verify that statements to randomize upper half of r1 had not been
  * generated.
tools/testing/selftests/bpf/progs/verifier_spill_fill.c +20 -20
index 1e5a511e8494..7a13dbd794b2 100644
@@ -506,17 +506,17 @@ SEC("raw_tp")
 __log_level(2)
 __success
 /* fp-8 is spilled IMPRECISE value zero (represented by a zero value fake reg) */
-__msg("2: (7a) *(u64 *)(r10 -8) = 0          ; R10=fp0 fp-8_w=0")
+__msg("2: (7a) *(u64 *)(r10 -8) = 0          ; R10=fp0 fp-8=0")
 /* but fp-16 is spilled IMPRECISE zero const reg */
-__msg("4: (7b) *(u64 *)(r10 -16) = r0        ; R0_w=0 R10=fp0 fp-16_w=0")
+__msg("4: (7b) *(u64 *)(r10 -16) = r0        ; R0=0 R10=fp0 fp-16=0")
 /* validate that assigning R2 from STACK_SPILL with zero value  doesn't mark register
  * precise immediately; if necessary, it will be marked precise later
  */
-__msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2_w=0 R10=fp0 fp-8_w=0")
+__msg("6: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0")
 /* similarly, when R2 is assigned from spilled register, it is initially
  * imprecise, but will be marked precise later once it is used in precise context
  */
-__msg("10: (71) r2 = *(u8 *)(r10 -9)         ; R2_w=0 R10=fp0 fp-16_w=0")
+__msg("10: (71) r2 = *(u8 *)(r10 -9)         ; R2=0 R10=fp0 fp-16=0")
 __msg("11: (0f) r1 += r2")
 __msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)")
@@ -598,7 +598,7 @@ __log_level(2)
 __success
 /* fp-4 is STACK_ZERO */
 __msg("2: (62) *(u32 *)(r10 -4) = 0          ; R10=fp0 fp-8=0000????")
-__msg("4: (71) r2 = *(u8 *)(r10 -1)          ; R2_w=0 R10=fp0 fp-8=0000????")
+__msg("4: (71) r2 = *(u8 *)(r10 -1)          ; R2=0 R10=fp0 fp-8=0000????")
 __msg("5: (0f) r1 += r2")
 __msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r2 stack= before 4: (71) r2 = *(u8 *)(r10 -1)")
@@ -640,25 +640,25 @@ SEC("raw_tp")
 __log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
 __success
 /* make sure fp-8 is IMPRECISE fake register spill */
-__msg("3: (7a) *(u64 *)(r10 -8) = 1          ; R10=fp0 fp-8_w=1")
+__msg("3: (7a) *(u64 *)(r10 -8) = 1          ; R10=fp0 fp-8=1")
 /* and fp-16 is spilled IMPRECISE const reg */
-__msg("5: (7b) *(u64 *)(r10 -16) = r0        ; R0_w=1 R10=fp0 fp-16_w=1")
+__msg("5: (7b) *(u64 *)(r10 -16) = r0        ; R0=1 R10=fp0 fp-16=1")
 /* validate load from fp-8, which was initialized using BPF_ST_MEM */
-__msg("8: (79) r2 = *(u64 *)(r10 -8)         ; R2_w=1 R10=fp0 fp-8=1")
+__msg("8: (79) r2 = *(u64 *)(r10 -8)         ; R2=1 R10=fp0 fp-8=1")
 __msg("9: (0f) r1 += r2")
 __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")
 __msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
 /* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_rw=P1 fp-16_w=1")
+__msg("mark_precise: frame0: parent state regs= stack=-8:  R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
 __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
 __msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
 __msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0")
 __msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
 __msg("mark_precise: frame0: regs= stack=-8 before 3: (7a) *(u64 *)(r10 -8) = 1")
-__msg("10: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
+__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
 /* validate load from fp-16, which was initialized using BPF_STX_MEM */
-__msg("12: (79) r2 = *(u64 *)(r10 -16)       ; R2_w=1 R10=fp0 fp-16=1")
+__msg("12: (79) r2 = *(u64 *)(r10 -16)       ; R2=1 R10=fp0 fp-16=1")
 __msg("13: (0f) r1 += r2")
 __msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r2 stack= before 12: (79) r2 = *(u64 *)(r10 -16)")
@@ -668,12 +668,12 @@ __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
 __msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")
 __msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
 /* now both fp-8 and fp-16 are precise, very good */
-__msg("mark_precise: frame0: parent state regs= stack=-16:  R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_rw=P1 fp-16_rw=P1")
+__msg("mark_precise: frame0: parent state regs= stack=-16:  R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
 __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
 __msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
 __msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0")
 __msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
+__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
 __naked void stack_load_preserves_const_precision(void)
 {
 	asm volatile (
@@ -719,22 +719,22 @@ __success
 /* make sure fp-8 is 32-bit FAKE subregister spill */
 __msg("3: (62) *(u32 *)(r10 -8) = 1          ; R10=fp0 fp-8=????1")
 /* but fp-16 is spilled IMPRECISE zero const reg */
-__msg("5: (63) *(u32 *)(r10 -16) = r0        ; R0_w=1 R10=fp0 fp-16=????1")
+__msg("5: (63) *(u32 *)(r10 -16) = r0        ; R0=1 R10=fp0 fp-16=????1")
 /* validate load from fp-8, which was initialized using BPF_ST_MEM */
-__msg("8: (61) r2 = *(u32 *)(r10 -8)         ; R2_w=1 R10=fp0 fp-8=????1")
+__msg("8: (61) r2 = *(u32 *)(r10 -8)         ; R2=1 R10=fp0 fp-8=????1")
 __msg("9: (0f) r1 += r2")
 __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)")
 __msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_r=????P1 fp-16=????1")
+__msg("mark_precise: frame0: parent state regs= stack=-8:  R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
 __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
 __msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
 __msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0")
 __msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
 __msg("mark_precise: frame0: regs= stack=-8 before 3: (62) *(u32 *)(r10 -8) = 1")
-__msg("10: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
+__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
 /* validate load from fp-16, which was initialized using BPF_STX_MEM */
-__msg("12: (61) r2 = *(u32 *)(r10 -16)       ; R2_w=1 R10=fp0 fp-16=????1")
+__msg("12: (61) r2 = *(u32 *)(r10 -16)       ; R2=1 R10=fp0 fp-16=????1")
 __msg("13: (0f) r1 += r2")
 __msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r2 stack= before 12: (61) r2 = *(u32 *)(r10 -16)")
@@ -743,12 +743,12 @@ __msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2
 __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
 __msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)")
 __msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-16:  R0_w=1 R1=ctx() R6_r=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8_r=????P1 fp-16_r=????P1")
+__msg("mark_precise: frame0: parent state regs= stack=-16:  R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
 __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
 __msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
 __msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0")
 __msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1_w=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2_w=1")
+__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,off=1) R2=1")
 __naked void stack_load_preserves_const_precision_subreg(void)
 {
 	asm volatile (
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +3 -3
index 9d415f7ce599..ac3e418c2a96 100644
@@ -105,7 +105,7 @@ __msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
 __msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
 __msg("mark_precise: frame0: regs=r0 stack= before 10: (95) exit")
 __msg("mark_precise: frame1: regs=r0 stack= before 9: (bf) r0 = (s8)r10")
-__msg("7: R0_w=scalar")
+__msg("7: R0=scalar")
 __naked int fp_precise_subprog_result(void)
 {
 	asm volatile (
@@ -141,7 +141,7 @@ __msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = (s8)r1")
  * anyways, at which point we'll break precision chain
  */
 __msg("mark_precise: frame1: regs=r1 stack= before 9: (bf) r1 = r10")
-__msg("7: R0_w=scalar")
+__msg("7: R0=scalar")
 __naked int sneaky_fp_precise_subprog_result(void)
 {
 	asm volatile (
@@ -681,7 +681,7 @@ __msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1")
 __msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8")
 __msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4")
 __msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8:  R0_w=2 R6_w=1 R8_rw=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8_rw=P1")
+__msg("mark_precise: frame0: parent state regs= stack=-8:  R0=2 R6=1 R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
 __msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7")
 __msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")
 __msg("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2")
tools/testing/selftests/bpf/verifier/bpf_st_mem.c +2 -2
index b616575c3b00..ce13002c7a19 100644
@@ -93,7 +93,7 @@
 	.expected_attach_type = BPF_SK_LOOKUP,
 	.result = VERBOSE_ACCEPT,
 	.runs = -1,
-	.errstr = "0: (7a) *(u64 *)(r10 -8) = -44        ; R10=fp0 fp-8_w=-44\
+	.errstr = "0: (7a) *(u64 *)(r10 -8) = -44        ; R10=fp0 fp-8=-44\
 	2: (c5) if r0 s< 0x0 goto pc+2\
-	R0_w=-44",
+	R0=-44",
 },
Signature · 1 line
2.47.3
#10 lore raw

Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

kernel test robot noticed the following build warnings:

Hi Eduard,

kernel test robot noticed the following build warnings:

[auto build test WARNING on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com
patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
config: x86_64-buildonly-randconfig-003-20250911 (https://download.01.org/0day-ci/archive/20250911/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250911/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> kernel/bpf/verifier.c:19305:11: warning: variable 'err' is uninitialized when used here [-Wuninitialized]
    19305 |                                 err = err ? : push_jmp_history(env, cur, 0, 0);
          |                                       ^~~
   kernel/bpf/verifier.c:19140:12: note: initialize the variable 'err' to silence this warning
    19140 |         int n, err, states_cnt = 0;
          |                   ^
          |                    = 0

1 warning generated.

vim +/err +19305 kernel/bpf/verifier.c

2589726d12a1b1 Alexei Starovoitov      2019-06-15  19133  
58e2af8b3a6b58 Jakub Kicinski          2016-09-21  19134  static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19135  {
58e2af8b3a6b58 Jakub Kicinski          2016-09-21  19136  	struct bpf_verifier_state_list *new_sl;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19137  	struct bpf_verifier_state_list *sl;
c9e31900b54cad Eduard Zingerman        2025-06-11  19138  	struct bpf_verifier_state *cur = env->cur_state, *new;
c9e31900b54cad Eduard Zingerman        2025-06-11  19139  	bool force_new_state, add_new_state, loop;
d5c95ed86213e4 Eduard Zingerman        2025-09-10  19140  	int n, err, states_cnt = 0;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19141  	struct list_head *pos, *tmp, *head;
aa30eb3260b2de Eduard Zingerman        2024-10-29  19142  
aa30eb3260b2de Eduard Zingerman        2024-10-29  19143  	force_new_state = env->test_state_freq || is_force_checkpoint(env, insn_idx) ||
aa30eb3260b2de Eduard Zingerman        2024-10-29  19144  			  /* Avoid accumulating infinitely long jmp history */
baaebe0928bf32 Eduard Zingerman        2025-06-11  19145  			  cur->jmp_history_cnt > 40;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19146  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19147  	/* bpf progs typically have pruning point every 4 instructions
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19148  	 * http://vger.kernel.org/bpfconf2019.html#session-1
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19149  	 * Do not add new state for future pruning if the verifier hasn't seen
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19150  	 * at least 2 jumps and at least 8 instructions.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19151  	 * This heuristics helps decrease 'total_states' and 'peak_states' metric.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19152  	 * In tests that amounts to up to 50% reduction into total verifier
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19153  	 * memory consumption and 20% verifier time speedup.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19154  	 */
aa30eb3260b2de Eduard Zingerman        2024-10-29  19155  	add_new_state = force_new_state;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19156  	if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19157  	    env->insn_processed - env->prev_insn_processed >= 8)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19158  		add_new_state = true;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19159  
9242b5f5615c82 Alexei Starovoitov      2018-12-13  19160  	clean_live_states(env, insn_idx, cur);
9242b5f5615c82 Alexei Starovoitov      2018-12-13  19161  
c9e31900b54cad Eduard Zingerman        2025-06-11  19162  	loop = false;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19163  	head = explored_state(env, insn_idx);
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19164  	list_for_each_safe(pos, tmp, head) {
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19165  		sl = container_of(pos, struct bpf_verifier_state_list, node);
dc2a4ebc0b44a2 Alexei Starovoitov      2019-05-21  19166  		states_cnt++;
dc2a4ebc0b44a2 Alexei Starovoitov      2019-05-21  19167  		if (sl->state.insn_idx != insn_idx)
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19168  			continue;
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19169  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19170  		if (sl->state.branches) {
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19171  			struct bpf_func_state *frame = sl->state.frame[sl->state.curframe];
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19172  
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19173  			if (frame->in_async_callback_fn &&
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19174  			    frame->async_entry_cnt != cur->frame[cur->curframe]->async_entry_cnt) {
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19175  				/* Different async_entry_cnt means that the verifier is
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19176  				 * processing another entry into async callback.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19177  				 * Seeing the same state is not an indication of infinite
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19178  				 * loop or infinite recursion.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19179  				 * But finding the same state doesn't mean that it's safe
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19180  				 * to stop processing the current state. The previous state
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19181  				 * hasn't yet reached bpf_exit, since state.branches > 0.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19182  				 * Checking in_async_callback_fn alone is not enough either.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19183  				 * Since the verifier still needs to catch infinite loops
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19184  				 * inside async callbacks.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19185  				 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19186  				goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19187  			}
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19188  			/* BPF open-coded iterators loop detection is special.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19189  			 * states_maybe_looping() logic is too simplistic in detecting
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19190  			 * states that *might* be equivalent, because it doesn't know
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19191  			 * about ID remapping, so don't even perform it.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19192  			 * See process_iter_next_call() and iter_active_depths_differ()
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19193  			 * for overview of the logic. When current and one of parent
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19194  			 * states are detected as equivalent, it's a good thing: we prove
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19195  			 * convergence and can stop simulating further iterations.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19196  			 * It's safe to assume that iterator loop will finish, taking into
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19197  			 * account iter_next() contract of eventually returning
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19198  			 * sticky NULL result.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19199  			 *
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19200  			 * Note, that states have to be compared exactly in this case because
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19201  			 * read and precision marks might not be finalized inside the loop.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19202  			 * E.g. as in the program below:
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19203  			 *
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19204  			 *     1. r7 = -16
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19205  			 *     2. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19206  			 *     3. while (bpf_iter_num_next(&fp[-8])) {
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19207  			 *     4.   if (r6 != 42) {
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19208  			 *     5.     r7 = -32
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19209  			 *     6.     r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19210  			 *     7.     continue
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19211  			 *     8.   }
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19212  			 *     9.   r0 = r10
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19213  			 *    10.   r0 += r7
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19214  			 *    11.   r8 = *(u64 *)(r0 + 0)
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19215  			 *    12.   r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19216  			 *    13. }
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19217  			 *
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19218  			 * Here verifier would first visit path 1-3, create a checkpoint at 3
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19219  			 * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19220  			 * not have read or precision mark for r7 yet, thus inexact states
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19221  			 * comparison would discard current state with r7=-32
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19222  			 * => unsafe memory access at 11 would not be caught.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19223  			 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19224  			if (is_iter_next_insn(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov      2024-03-05  19225  				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19226  					struct bpf_func_state *cur_frame;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19227  					struct bpf_reg_state *iter_state, *iter_reg;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19228  					int spi;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19229  
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19230  					cur_frame = cur->frame[cur->curframe];
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19231  					/* btf_check_iter_kfuncs() enforces that
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19232  					 * iter state pointer is always the first arg
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19233  					 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19234  					iter_reg = &cur_frame->regs[BPF_REG_1];
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19235  					/* current state is valid due to states_equal(),
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19236  					 * so we can assume valid iter and reg state,
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19237  					 * no need for extra (re-)validations
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19238  					 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19239  					spi = __get_spi(iter_reg->off + iter_reg->var_off.value);
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19240  					iter_state = &func(env, iter_reg)->stack[spi].spilled_ptr;
2a0992829ea386 Eduard Zingerman        2023-10-24  19241  					if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19242  						loop = true;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19243  						goto hit;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19244  					}
2a0992829ea386 Eduard Zingerman        2023-10-24  19245  				}
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19246  				goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19247  			}
011832b97b311b Alexei Starovoitov      2024-03-05  19248  			if (is_may_goto_insn_at(env, insn_idx)) {
2b2efe1937ca9f Alexei Starovoitov      2024-06-19  19249  				if (sl->state.may_goto_depth != cur->may_goto_depth &&
2b2efe1937ca9f Alexei Starovoitov      2024-06-19  19250  				    states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19251  					loop = true;
011832b97b311b Alexei Starovoitov      2024-03-05  19252  					goto hit;
011832b97b311b Alexei Starovoitov      2024-03-05  19253  				}
011832b97b311b Alexei Starovoitov      2024-03-05  19254  			}
588af0c506ec8e Eduard Zingerman        2025-09-10  19255  			if (bpf_calls_callback(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov      2024-03-05  19256  				if (states_equal(env, &sl->state, cur, RANGE_WITHIN))
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19257  					goto hit;
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19258  				goto skip_inf_loop_check;
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19259  			}
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19260  			/* attempt to detect infinite loop to avoid unnecessary doomed work */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19261  			if (states_maybe_looping(&sl->state, cur) &&
4f81c16f50baf6 Alexei Starovoitov      2024-03-05  19262  			    states_equal(env, &sl->state, cur, EXACT) &&
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19263  			    !iter_active_depths_differ(&sl->state, cur) &&
011832b97b311b Alexei Starovoitov      2024-03-05  19264  			    sl->state.may_goto_depth == cur->may_goto_depth &&
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19265  			    sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19266  				verbose_linfo(env, insn_idx, "; ");
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19267  				verbose(env, "infinite loop detected at insn %d\n", insn_idx);
b4d8239534fddc Eduard Zingerman        2023-10-24  19268  				verbose(env, "cur state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03  19269  				print_verifier_state(env, cur, cur->curframe, true);
b4d8239534fddc Eduard Zingerman        2023-10-24  19270  				verbose(env, "old state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03  19271  				print_verifier_state(env, &sl->state, cur->curframe, true);
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19272  				return -EINVAL;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19273  			}
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19274  			/* if the verifier is processing a loop, avoid adding new state
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19275  			 * too often, since different loop iterations have distinct
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19276  			 * states and may not help future pruning.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19277  			 * This threshold shouldn't be too low to make sure that
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19278  			 * a loop with large bound will be rejected quickly.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19279  			 * The most abusive loop will be:
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19280  			 * r1 += 1
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19281  			 * if r1 < 1000000 goto pc-2
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19282  			 * 1M insn_procssed limit / 100 == 10k peak states.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19283  			 * This threshold shouldn't be too high either, since states
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19284  			 * at the end of the loop are likely to be useful in pruning.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19285  			 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19286  skip_inf_loop_check:
4b5ce570dbef57 Andrii Nakryiko         2023-03-09  19287  			if (!force_new_state &&
98ddcf389d1bb7 Andrii Nakryiko         2023-03-02  19288  			    env->jmps_processed - env->prev_jmps_processed < 20 &&
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19289  			    env->insn_processed - env->prev_insn_processed < 100)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19290  				add_new_state = false;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19291  			goto miss;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19292  		}
c9e31900b54cad Eduard Zingerman        2025-06-11  19293  		/* See comments for mark_all_regs_read_and_precise() */
c9e31900b54cad Eduard Zingerman        2025-06-11  19294  		loop = incomplete_read_marks(env, &sl->state);
c9e31900b54cad Eduard Zingerman        2025-06-11  19295  		if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19296  hit:
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19297  			sl->hit_cnt++;
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19298  
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19299  			/* if previous state reached the exit with precision and
a7de265cb2d849 Rafael Passos           2024-04-17  19300  			 * current state is equivalent to it (except precision marks)
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19301  			 * the precision needs to be propagated back in
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19302  			 * the current state.
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19303  			 */
41f6f64e6999a8 Andrii Nakryiko         2023-12-05  19304  			if (is_jmp_point(env, env->insn_idx))
baaebe0928bf32 Eduard Zingerman        2025-06-11 @19305  				err = err ? : push_jmp_history(env, cur, 0, 0);
23b37d616565c8 Eduard Zingerman        2025-06-11  19306  			err = err ? : propagate_precision(env, &sl->state, cur, NULL);
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19307  			if (err)
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19308  				return err;
c9e31900b54cad Eduard Zingerman        2025-06-11  19309  			/* When processing iterator based loops above propagate_liveness and
c9e31900b54cad Eduard Zingerman        2025-06-11  19310  			 * propagate_precision calls are not sufficient to transfer all relevant
c9e31900b54cad Eduard Zingerman        2025-06-11  19311  			 * read and precision marks. E.g. consider the following case:
c9e31900b54cad Eduard Zingerman        2025-06-11  19312  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19313  			 *  .-> A --.  Assume the states are visited in the order A, B, C.
c9e31900b54cad Eduard Zingerman        2025-06-11  19314  			 *  |   |   |  Assume that state B reaches a state equivalent to state A.
c9e31900b54cad Eduard Zingerman        2025-06-11  19315  			 *  |   v   v  At this point, state C is not processed yet, so state A
c9e31900b54cad Eduard Zingerman        2025-06-11  19316  			 *  '-- B   C  has not received any read or precision marks from C.
c9e31900b54cad Eduard Zingerman        2025-06-11  19317  			 *             Thus, marks propagated from A to B are incomplete.
c9e31900b54cad Eduard Zingerman        2025-06-11  19318  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19319  			 * The verifier mitigates this by performing the following steps:
c9e31900b54cad Eduard Zingerman        2025-06-11  19320  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19321  			 * - Prior to the main verification pass, strongly connected components
c9e31900b54cad Eduard Zingerman        2025-06-11  19322  			 *   (SCCs) are computed over the program's control flow graph,
c9e31900b54cad Eduard Zingerman        2025-06-11  19323  			 *   intraprocedurally.
c9e31900b54cad Eduard Zingerman        2025-06-11  19324  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19325  			 * - During the main verification pass, `maybe_enter_scc()` checks
c9e31900b54cad Eduard Zingerman        2025-06-11  19326  			 *   whether the current verifier state is entering an SCC. If so, an
c9e31900b54cad Eduard Zingerman        2025-06-11  19327  			 *   instance of a `bpf_scc_visit` object is created, and the state
c9e31900b54cad Eduard Zingerman        2025-06-11  19328  			 *   entering the SCC is recorded as the entry state.
c9e31900b54cad Eduard Zingerman        2025-06-11  19329  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19330  			 * - This instance is associated not with the SCC itself, but with a
c9e31900b54cad Eduard Zingerman        2025-06-11  19331  			 *   `bpf_scc_callchain`: a tuple consisting of the call sites leading to
c9e31900b54cad Eduard Zingerman        2025-06-11  19332  			 *   the SCC and the SCC id. See `compute_scc_callchain()`.
c9e31900b54cad Eduard Zingerman        2025-06-11  19333  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19334  			 * - When a verification path encounters a `states_equal(...,
c9e31900b54cad Eduard Zingerman        2025-06-11  19335  			 *   RANGE_WITHIN)` condition, there exists a call chain describing the
c9e31900b54cad Eduard Zingerman        2025-06-11  19336  			 *   current state and a corresponding `bpf_scc_visit` instance. A copy
c9e31900b54cad Eduard Zingerman        2025-06-11  19337  			 *   of the current state is created and added to
c9e31900b54cad Eduard Zingerman        2025-06-11  19338  			 *   `bpf_scc_visit->backedges`.
c9e31900b54cad Eduard Zingerman        2025-06-11  19339  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19340  			 * - When a verification path terminates, `maybe_exit_scc()` is called
c9e31900b54cad Eduard Zingerman        2025-06-11  19341  			 *   from `update_branch_counts()`. For states with `branches == 0`, it
c9e31900b54cad Eduard Zingerman        2025-06-11  19342  			 *   checks whether the state is the entry state of any `bpf_scc_visit`
c9e31900b54cad Eduard Zingerman        2025-06-11  19343  			 *   instance. If it is, this indicates that all paths originating from
c9e31900b54cad Eduard Zingerman        2025-06-11  19344  			 *   this SCC visit have been explored. `propagate_backedges()` is then
c9e31900b54cad Eduard Zingerman        2025-06-11  19345  			 *   called, which propagates read and precision marks through the
c9e31900b54cad Eduard Zingerman        2025-06-11  19346  			 *   backedges until a fixed point is reached.
c9e31900b54cad Eduard Zingerman        2025-06-11  19347  			 *   (In the earlier example, this would propagate marks from A to B,
c9e31900b54cad Eduard Zingerman        2025-06-11  19348  			 *    from C to A, and then again from A to B.)
c9e31900b54cad Eduard Zingerman        2025-06-11  19349  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19350  			 * A note on callchains
c9e31900b54cad Eduard Zingerman        2025-06-11  19351  			 * --------------------
c9e31900b54cad Eduard Zingerman        2025-06-11  19352  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19353  			 * Consider the following example:
c9e31900b54cad Eduard Zingerman        2025-06-11  19354  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19355  			 *     void foo() { loop { ... SCC#1 ... } }
c9e31900b54cad Eduard Zingerman        2025-06-11  19356  			 *     void main() {
c9e31900b54cad Eduard Zingerman        2025-06-11  19357  			 *       A: foo();
c9e31900b54cad Eduard Zingerman        2025-06-11  19358  			 *       B: ...
c9e31900b54cad Eduard Zingerman        2025-06-11  19359  			 *       C: foo();
c9e31900b54cad Eduard Zingerman        2025-06-11  19360  			 *     }
c9e31900b54cad Eduard Zingerman        2025-06-11  19361  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19362  			 * Here, there are two distinct callchains leading to SCC#1:
c9e31900b54cad Eduard Zingerman        2025-06-11  19363  			 * - (A, SCC#1)
c9e31900b54cad Eduard Zingerman        2025-06-11  19364  			 * - (C, SCC#1)
c9e31900b54cad Eduard Zingerman        2025-06-11  19365  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19366  			 * Each callchain identifies a separate `bpf_scc_visit` instance that
c9e31900b54cad Eduard Zingerman        2025-06-11  19367  			 * accumulates backedge states. The `propagate_{liveness,precision}()`
c9e31900b54cad Eduard Zingerman        2025-06-11  19368  			 * functions traverse the parent state of each backedge state, which
c9e31900b54cad Eduard Zingerman        2025-06-11  19369  			 * means these parent states must remain valid (i.e., not freed) while
c9e31900b54cad Eduard Zingerman        2025-06-11  19370  			 * the corresponding `bpf_scc_visit` instance exists.
c9e31900b54cad Eduard Zingerman        2025-06-11  19371  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19372  			 * Associating `bpf_scc_visit` instances directly with SCCs instead of
c9e31900b54cad Eduard Zingerman        2025-06-11  19373  			 * callchains would break this invariant:
c9e31900b54cad Eduard Zingerman        2025-06-11  19374  			 * - States explored during `C: foo()` would contribute backedges to
c9e31900b54cad Eduard Zingerman        2025-06-11  19375  			 *   SCC#1, but SCC#1 would only be exited once the exploration of
c9e31900b54cad Eduard Zingerman        2025-06-11  19376  			 *   `A: foo()` completes.
c9e31900b54cad Eduard Zingerman        2025-06-11  19377  			 * - By that time, the states explored between `A: foo()` and `C: foo()`
c9e31900b54cad Eduard Zingerman        2025-06-11  19378  			 *   (i.e., `B: ...`) may have already been freed, causing the parent
c9e31900b54cad Eduard Zingerman        2025-06-11  19379  			 *   links for states from `C: foo()` to become invalid.
c9e31900b54cad Eduard Zingerman        2025-06-11  19380  			 */
c9e31900b54cad Eduard Zingerman        2025-06-11  19381  			if (loop) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19382  				struct bpf_scc_backedge *backedge;
c9e31900b54cad Eduard Zingerman        2025-06-11  19383  
43736ec3e02789 Eduard Zingerman        2025-06-13  19384  				backedge = kzalloc(sizeof(*backedge), GFP_KERNEL_ACCOUNT);
c9e31900b54cad Eduard Zingerman        2025-06-11  19385  				if (!backedge)
c9e31900b54cad Eduard Zingerman        2025-06-11  19386  					return -ENOMEM;
c9e31900b54cad Eduard Zingerman        2025-06-11  19387  				err = copy_verifier_state(&backedge->state, cur);
c9e31900b54cad Eduard Zingerman        2025-06-11  19388  				backedge->state.equal_state = &sl->state;
c9e31900b54cad Eduard Zingerman        2025-06-11  19389  				backedge->state.insn_idx = insn_idx;
c9e31900b54cad Eduard Zingerman        2025-06-11  19390  				err = err ?: add_scc_backedge(env, &sl->state, backedge);
c9e31900b54cad Eduard Zingerman        2025-06-11  19391  				if (err) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19392  					free_verifier_state(&backedge->state, false);
bf0c2a84df9fb0 Qianfeng Rong           2025-08-11  19393  					kfree(backedge);
c9e31900b54cad Eduard Zingerman        2025-06-11  19394  					return err;
c9e31900b54cad Eduard Zingerman        2025-06-11  19395  				}
c9e31900b54cad Eduard Zingerman        2025-06-11  19396  			}
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19397  			return 1;
dc503a8ad98474 Edward Cree             2017-08-15  19398  		}
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19399  miss:
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19400  		/* when new state is not going to be added do not increase miss count.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19401  		 * Otherwise several loop iterations will remove the state
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19402  		 * recorded earlier. The goal of these heuristics is to have
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19403  		 * states from some iterations of the loop (some in the beginning
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19404  		 * and some at the end) to help pruning.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19405  		 */
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19406  		if (add_new_state)
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19407  			sl->miss_cnt++;
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19408  		/* heuristic to determine whether this state is beneficial
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19409  		 * to keep checking from state equivalence point of view.
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19410  		 * Higher numbers increase max_states_per_insn and verification time,
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19411  		 * but do not meaningfully decrease insn_processed.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19412  		 * 'n' controls how many times state could miss before eviction.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19413  		 * Use bigger 'n' for checkpoints because evicting checkpoint states
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19414  		 * too early would hinder iterator convergence.
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19415  		 */
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19416  		n = is_force_checkpoint(env, insn_idx) && sl->state.branches > 0 ? 64 : 3;
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19417  		if (sl->miss_cnt > sl->hit_cnt * n + n) {
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19418  			/* the state is unlikely to be useful. Remove it to
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19419  			 * speed up verification
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19420  			 */
408fcf946b2bad Eduard Zingerman        2025-02-15  19421  			sl->in_free_list = true;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19422  			list_del(&sl->node);
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19423  			list_add(&sl->node, &env->free_list);
574078b001cdf6 Eduard Zingerman        2025-02-15  19424  			env->free_list_size++;
574078b001cdf6 Eduard Zingerman        2025-02-15  19425  			env->explored_states_size--;
408fcf946b2bad Eduard Zingerman        2025-02-15  19426  			maybe_free_verifier_state(env, sl);
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19427  		}
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19428  	}
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19429  
06ee7115b0d174 Alexei Starovoitov      2019-04-01  19430  	if (env->max_states_per_insn < states_cnt)
06ee7115b0d174 Alexei Starovoitov      2019-04-01  19431  		env->max_states_per_insn = states_cnt;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19432  
2c78ee898d8f10 Alexei Starovoitov      2020-05-13  19433  	if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
a095f421057e22 Andrii Nakryiko         2022-12-06  19434  		return 0;
ceefbc96fa5c5b Alexei Starovoitov      2018-12-03  19435  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19436  	if (!add_new_state)
a095f421057e22 Andrii Nakryiko         2022-12-06  19437  		return 0;
ceefbc96fa5c5b Alexei Starovoitov      2018-12-03  19438  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19439  	/* There were no equivalent states, remember the current one.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19440  	 * Technically the current state is not proven to be safe yet,
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19441  	 * but it will either reach outer most bpf_exit (which means it's safe)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19442  	 * or it will be rejected. When there are no loops the verifier won't be
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19443  	 * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19444  	 * again on the way to bpf_exit.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19445  	 * When looping the sl->state.branches will be > 0 and this state
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19446  	 * will not be considered for equivalence until branches == 0.
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19447  	 */
43736ec3e02789 Eduard Zingerman        2025-06-13  19448  	new_sl = kzalloc(sizeof(struct bpf_verifier_state_list), GFP_KERNEL_ACCOUNT);
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19449  	if (!new_sl)
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19450  		return -ENOMEM;
06ee7115b0d174 Alexei Starovoitov      2019-04-01  19451  	env->total_states++;
574078b001cdf6 Eduard Zingerman        2025-02-15  19452  	env->explored_states_size++;
574078b001cdf6 Eduard Zingerman        2025-02-15  19453  	update_peak_states(env);
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19454  	env->prev_jmps_processed = env->jmps_processed;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19455  	env->prev_insn_processed = env->insn_processed;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19456  
7a830b53c17bba Andrii Nakryiko         2022-11-04  19457  	/* forget precise markings we inherited, see __mark_chain_precision */
7a830b53c17bba Andrii Nakryiko         2022-11-04  19458  	if (env->bpf_capable)
7a830b53c17bba Andrii Nakryiko         2022-11-04  19459  		mark_all_scalars_imprecise(env, cur);
7a830b53c17bba Andrii Nakryiko         2022-11-04  19460  
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19461  	/* add new state to the head of linked list */
679c782de14bd4 Edward Cree             2018-08-22  19462  	new = &new_sl->state;
679c782de14bd4 Edward Cree             2018-08-22  19463  	err = copy_verifier_state(new, cur);
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19464  	if (err) {
679c782de14bd4 Edward Cree             2018-08-22  19465  		free_verifier_state(new, false);
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19466  		kfree(new_sl);
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19467  		return err;
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19468  	}
dc2a4ebc0b44a2 Alexei Starovoitov      2019-05-21  19469  	new->insn_idx = insn_idx;
0df1a55afa832f Paul Chaignon           2025-07-01  19470  	verifier_bug_if(new->branches != 1, env,
0df1a55afa832f Paul Chaignon           2025-07-01  19471  			"%s:branches_to_explore=%d insn %d",
0df1a55afa832f Paul Chaignon           2025-07-01  19472  			__func__, new->branches, insn_idx);
c9e31900b54cad Eduard Zingerman        2025-06-11  19473  	err = maybe_enter_scc(env, new);
c9e31900b54cad Eduard Zingerman        2025-06-11  19474  	if (err) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19475  		free_verifier_state(new, false);
e4980fa6463624 Feng Yang               2025-08-27  19476  		kfree(new_sl);
c9e31900b54cad Eduard Zingerman        2025-06-11  19477  		return err;
c9e31900b54cad Eduard Zingerman        2025-06-11  19478  	}
b5dc0163d8fd78 Alexei Starovoitov      2019-06-15  19479  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19480  	cur->parent = new;
b5dc0163d8fd78 Alexei Starovoitov      2019-06-15  19481  	cur->first_insn_idx = insn_idx;
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19482  	cur->dfs_depth = new->dfs_depth + 1;
baaebe0928bf32 Eduard Zingerman        2025-06-11  19483  	clear_jmp_history(cur);
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19484  	list_add(&new_sl->node, head);
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19485  	return 0;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19486  }
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19487  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author maintainer
#11 lore raw

Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

Quoted 28 lines · kernel test robot

Hi Eduard,

kernel test robot noticed the following build warnings:

[auto build test WARNING on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com
patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
config: x86_64-buildonly-randconfig-003-20250911 (https://download.01.org/0day-ci/archive/20250911/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250911/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

Quoted 2 lines

All warnings (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

kernel/bpf/verifier.c:19305:11: warning: variable 'err' is uninitialized when used here [-Wuninitialized]

    19305 |                                 err = err ? : push_jmp_history(env, cur, 0, 0);
          |                                       ^~~
   kernel/bpf/verifier.c:19140:12: note: initialize the variable 'err' to silence this warning
    19140 |         int n, err, states_cnt = 0;
          |                   ^
          |                    = 0
Quoted 4 lines

1 warning generated.

vim +/err +19305 kernel/bpf/verifier.c

This was sloppy on my side, should look as follows:

kernel/bpf/verifier.c +6 -3
@@ -19297,9 +19297,12 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
                         * the precision needs to be propagated back in
                         * the current state.
                         */
-                       if (is_jmp_point(env, env->insn_idx))
-                               err = err ? : push_jmp_history(env, cur, 0, 0);
-                       err = err ? : propagate_precision(env, &sl->state, cur, NULL);
+                       if (is_jmp_point(env, env->insn_idx)) {
+                               err = push_jmp_history(env, cur, 0, 0);
+                               if (err)
+                                       return err;
+                       }
+                       err = propagate_precision(env, &sl->state, cur, NULL);
                        if (err)
                                return err;
                        /* When processing iterator based loops above propagate_liveness and

[...]

#12 lore raw

Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

Quoted 42 lines · Eduard Zingerman
Quoted 28 lines · kernel test robot

Hi Eduard,

kernel test robot noticed the following build warnings:

[auto build test WARNING on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com
patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
config: x86_64-buildonly-randconfig-003-20250911 (https://download.01.org/0day-ci/archive/20250911/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250911/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

Quoted 2 lines

All warnings (new ones prefixed by >>):

Quoted 1 line
Quoted 1 line

kernel/bpf/verifier.c:19305:11: warning: variable 'err' is uninitialized when used here [-Wuninitialized]

    19305 |                                 err = err ? : push_jmp_history(env, cur, 0, 0);
          |                                       ^~~
   kernel/bpf/verifier.c:19140:12: note: initialize the variable 'err' to silence this warning
    19140 |         int n, err, states_cnt = 0;
          |                   ^
          |                    = 0
Quoted 4 lines

1 warning generated.

vim +/err +19305 kernel/bpf/verifier.c

This was sloppy on my side, should look as follows:

kernel/bpf/verifier.c +0 -0
@@ -19297,9 +19297,12 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
                         * the precision needs to be propagated back in
                         * the current state.

kernel/bpf/verifier.c static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) · line 19305

Quoted 10 lines
kernel/bpf/verifier.c +6 -3
                         */
-                       if (is_jmp_point(env, env->insn_idx))
-                               err = err ? : push_jmp_history(env, cur, 0, 0);
-                       err = err ? : propagate_precision(env, &sl->state, cur, NULL);
+                       if (is_jmp_point(env, env->insn_idx)) {
+                               err = push_jmp_history(env, cur, 0, 0);
+                               if (err)
+                                       return err;
+                       }
+                       err = propagate_precision(env, &sl->state, cur, NULL);

hmm. init err=0 instead and avoid explicit if (err)return err ?

author maintainer
#13 lore raw

Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

On Thu, 2025-09-11 at 15:00 -0700, Alexei Starovoitov wrote:

[...]

Quoted 9 lines
Quoted 9 lines
kernel/bpf/verifier.c +0 -3
@@ -19297,9 +19297,12 @@ static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
                         * the precision needs to be propagated back in
                         * the current state.
                         */
-                       if (is_jmp_point(env, env->insn_idx))
-                               err = err ? : push_jmp_history(env, cur, 0, 0);
-                       err = err ? : propagate_precision(env, &sl->state, cur, NULL);
Quoted 8 lines

kernel/bpf/verifier.c static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) · line 19305

Quoted 6 lines
kernel/bpf/verifier.c +6 -0
+                       if (is_jmp_point(env, env->insn_idx)) {
+                               err = push_jmp_history(env, cur, 0, 0);
+                               if (err)
+                                       return err;
+                       }
+                       err = propagate_precision(env, &sl->state, cur, NULL);

hmm. init err=0 instead and avoid explicit if (err)return err ?

Or like that, yes.

build bot
#14 lore raw

Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

In reply to Eduard Zingerman
BCC: [email protected] CC: [email protected] In-Reply-To: <[email protected]> References: <[email protected]> TO: Eduard Zingerman…

BCC: [email protected]
CC: [email protected]
In-Reply-To: <[email protected]>
References: <[email protected]>
TO: Eduard Zingerman <[email protected]>
TO: [email protected]
TO: [email protected]
TO: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]

Hi Eduard,

kernel test robot noticed the following build warnings:

[auto build test WARNING on bpf-next/master]

url:    https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com
patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

:::::: branch date: 18 hours ago
:::::: commit date: 18 hours ago
config: arm-randconfig-r071-20250911 (https://download.01.org/0day-ci/archive/20250912/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 21857ae337e0892a5522b6e7337899caa61de2a6)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

smatch warnings:
kernel/bpf/verifier.c:19305 is_state_visited() error: uninitialized symbol 'err'.

vim +/err +19305 kernel/bpf/verifier.c

2589726d12a1b1 Alexei Starovoitov      2019-06-15  19133  
58e2af8b3a6b58 Jakub Kicinski          2016-09-21  19134  static int is_state_visited(struct bpf_verifier_env *env, int insn_idx)
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19135  {
58e2af8b3a6b58 Jakub Kicinski          2016-09-21  19136  	struct bpf_verifier_state_list *new_sl;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19137  	struct bpf_verifier_state_list *sl;
c9e31900b54cad Eduard Zingerman        2025-06-11  19138  	struct bpf_verifier_state *cur = env->cur_state, *new;
c9e31900b54cad Eduard Zingerman        2025-06-11  19139  	bool force_new_state, add_new_state, loop;
d5c95ed86213e4 Eduard Zingerman        2025-09-10  19140  	int n, err, states_cnt = 0;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19141  	struct list_head *pos, *tmp, *head;
aa30eb3260b2de Eduard Zingerman        2024-10-29  19142  
aa30eb3260b2de Eduard Zingerman        2024-10-29  19143  	force_new_state = env->test_state_freq || is_force_checkpoint(env, insn_idx) ||
aa30eb3260b2de Eduard Zingerman        2024-10-29  19144  			  /* Avoid accumulating infinitely long jmp history */
baaebe0928bf32 Eduard Zingerman        2025-06-11  19145  			  cur->jmp_history_cnt > 40;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19146  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19147  	/* bpf progs typically have pruning point every 4 instructions
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19148  	 * http://vger.kernel.org/bpfconf2019.html#session-1
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19149  	 * Do not add new state for future pruning if the verifier hasn't seen
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19150  	 * at least 2 jumps and at least 8 instructions.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19151  	 * This heuristics helps decrease 'total_states' and 'peak_states' metric.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19152  	 * In tests that amounts to up to 50% reduction into total verifier
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19153  	 * memory consumption and 20% verifier time speedup.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19154  	 */
aa30eb3260b2de Eduard Zingerman        2024-10-29  19155  	add_new_state = force_new_state;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19156  	if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19157  	    env->insn_processed - env->prev_insn_processed >= 8)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19158  		add_new_state = true;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19159  
9242b5f5615c82 Alexei Starovoitov      2018-12-13  19160  	clean_live_states(env, insn_idx, cur);
9242b5f5615c82 Alexei Starovoitov      2018-12-13  19161  
c9e31900b54cad Eduard Zingerman        2025-06-11  19162  	loop = false;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19163  	head = explored_state(env, insn_idx);
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19164  	list_for_each_safe(pos, tmp, head) {
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19165  		sl = container_of(pos, struct bpf_verifier_state_list, node);
dc2a4ebc0b44a2 Alexei Starovoitov      2019-05-21  19166  		states_cnt++;
dc2a4ebc0b44a2 Alexei Starovoitov      2019-05-21  19167  		if (sl->state.insn_idx != insn_idx)
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19168  			continue;
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19169  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19170  		if (sl->state.branches) {
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19171  			struct bpf_func_state *frame = sl->state.frame[sl->state.curframe];
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19172  
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19173  			if (frame->in_async_callback_fn &&
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19174  			    frame->async_entry_cnt != cur->frame[cur->curframe]->async_entry_cnt) {
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19175  				/* Different async_entry_cnt means that the verifier is
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19176  				 * processing another entry into async callback.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19177  				 * Seeing the same state is not an indication of infinite
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19178  				 * loop or infinite recursion.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19179  				 * But finding the same state doesn't mean that it's safe
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19180  				 * to stop processing the current state. The previous state
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19181  				 * hasn't yet reached bpf_exit, since state.branches > 0.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19182  				 * Checking in_async_callback_fn alone is not enough either.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19183  				 * Since the verifier still needs to catch infinite loops
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19184  				 * inside async callbacks.
bfc6bb74e4f16a Alexei Starovoitov      2021-07-14  19185  				 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19186  				goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19187  			}
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19188  			/* BPF open-coded iterators loop detection is special.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19189  			 * states_maybe_looping() logic is too simplistic in detecting
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19190  			 * states that *might* be equivalent, because it doesn't know
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19191  			 * about ID remapping, so don't even perform it.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19192  			 * See process_iter_next_call() and iter_active_depths_differ()
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19193  			 * for overview of the logic. When current and one of parent
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19194  			 * states are detected as equivalent, it's a good thing: we prove
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19195  			 * convergence and can stop simulating further iterations.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19196  			 * It's safe to assume that iterator loop will finish, taking into
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19197  			 * account iter_next() contract of eventually returning
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19198  			 * sticky NULL result.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19199  			 *
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19200  			 * Note, that states have to be compared exactly in this case because
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19201  			 * read and precision marks might not be finalized inside the loop.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19202  			 * E.g. as in the program below:
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19203  			 *
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19204  			 *     1. r7 = -16
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19205  			 *     2. r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19206  			 *     3. while (bpf_iter_num_next(&fp[-8])) {
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19207  			 *     4.   if (r6 != 42) {
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19208  			 *     5.     r7 = -32
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19209  			 *     6.     r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19210  			 *     7.     continue
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19211  			 *     8.   }
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19212  			 *     9.   r0 = r10
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19213  			 *    10.   r0 += r7
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19214  			 *    11.   r8 = *(u64 *)(r0 + 0)
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19215  			 *    12.   r6 = bpf_get_prandom_u32()
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19216  			 *    13. }
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19217  			 *
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19218  			 * Here verifier would first visit path 1-3, create a checkpoint at 3
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19219  			 * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19220  			 * not have read or precision mark for r7 yet, thus inexact states
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19221  			 * comparison would discard current state with r7=-32
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19222  			 * => unsafe memory access at 11 would not be caught.
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19223  			 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19224  			if (is_iter_next_insn(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov      2024-03-05  19225  				if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19226  					struct bpf_func_state *cur_frame;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19227  					struct bpf_reg_state *iter_state, *iter_reg;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19228  					int spi;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19229  
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19230  					cur_frame = cur->frame[cur->curframe];
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19231  					/* btf_check_iter_kfuncs() enforces that
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19232  					 * iter state pointer is always the first arg
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19233  					 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19234  					iter_reg = &cur_frame->regs[BPF_REG_1];
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19235  					/* current state is valid due to states_equal(),
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19236  					 * so we can assume valid iter and reg state,
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19237  					 * no need for extra (re-)validations
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19238  					 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19239  					spi = __get_spi(iter_reg->off + iter_reg->var_off.value);
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19240  					iter_state = &func(env, iter_reg)->stack[spi].spilled_ptr;
2a0992829ea386 Eduard Zingerman        2023-10-24  19241  					if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19242  						loop = true;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19243  						goto hit;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19244  					}
2a0992829ea386 Eduard Zingerman        2023-10-24  19245  				}
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19246  				goto skip_inf_loop_check;
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19247  			}
011832b97b311b Alexei Starovoitov      2024-03-05  19248  			if (is_may_goto_insn_at(env, insn_idx)) {
2b2efe1937ca9f Alexei Starovoitov      2024-06-19  19249  				if (sl->state.may_goto_depth != cur->may_goto_depth &&
2b2efe1937ca9f Alexei Starovoitov      2024-06-19  19250  				    states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19251  					loop = true;
011832b97b311b Alexei Starovoitov      2024-03-05  19252  					goto hit;
011832b97b311b Alexei Starovoitov      2024-03-05  19253  				}
011832b97b311b Alexei Starovoitov      2024-03-05  19254  			}
588af0c506ec8e Eduard Zingerman        2025-09-10  19255  			if (bpf_calls_callback(env, insn_idx)) {
4f81c16f50baf6 Alexei Starovoitov      2024-03-05  19256  				if (states_equal(env, &sl->state, cur, RANGE_WITHIN))
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19257  					goto hit;
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19258  				goto skip_inf_loop_check;
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19259  			}
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19260  			/* attempt to detect infinite loop to avoid unnecessary doomed work */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19261  			if (states_maybe_looping(&sl->state, cur) &&
4f81c16f50baf6 Alexei Starovoitov      2024-03-05  19262  			    states_equal(env, &sl->state, cur, EXACT) &&
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19263  			    !iter_active_depths_differ(&sl->state, cur) &&
011832b97b311b Alexei Starovoitov      2024-03-05  19264  			    sl->state.may_goto_depth == cur->may_goto_depth &&
ab5cfac139ab85 Eduard Zingerman        2023-11-21  19265  			    sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19266  				verbose_linfo(env, insn_idx, "; ");
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19267  				verbose(env, "infinite loop detected at insn %d\n", insn_idx);
b4d8239534fddc Eduard Zingerman        2023-10-24  19268  				verbose(env, "cur state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03  19269  				print_verifier_state(env, cur, cur->curframe, true);
b4d8239534fddc Eduard Zingerman        2023-10-24  19270  				verbose(env, "old state:");
1995edc5f9089e Kumar Kartikeya Dwivedi 2024-12-03  19271  				print_verifier_state(env, &sl->state, cur->curframe, true);
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19272  				return -EINVAL;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19273  			}
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19274  			/* if the verifier is processing a loop, avoid adding new state
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19275  			 * too often, since different loop iterations have distinct
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19276  			 * states and may not help future pruning.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19277  			 * This threshold shouldn't be too low to make sure that
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19278  			 * a loop with large bound will be rejected quickly.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19279  			 * The most abusive loop will be:
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19280  			 * r1 += 1
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19281  			 * if r1 < 1000000 goto pc-2
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19282  			 * 1M insn_procssed limit / 100 == 10k peak states.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19283  			 * This threshold shouldn't be too high either, since states
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19284  			 * at the end of the loop are likely to be useful in pruning.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19285  			 */
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19286  skip_inf_loop_check:
4b5ce570dbef57 Andrii Nakryiko         2023-03-09  19287  			if (!force_new_state &&
98ddcf389d1bb7 Andrii Nakryiko         2023-03-02  19288  			    env->jmps_processed - env->prev_jmps_processed < 20 &&
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19289  			    env->insn_processed - env->prev_insn_processed < 100)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19290  				add_new_state = false;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19291  			goto miss;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19292  		}
c9e31900b54cad Eduard Zingerman        2025-06-11  19293  		/* See comments for mark_all_regs_read_and_precise() */
c9e31900b54cad Eduard Zingerman        2025-06-11  19294  		loop = incomplete_read_marks(env, &sl->state);
c9e31900b54cad Eduard Zingerman        2025-06-11  19295  		if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
06accc8779c1d5 Andrii Nakryiko         2023-03-08  19296  hit:
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19297  			sl->hit_cnt++;
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19298  
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19299  			/* if previous state reached the exit with precision and
a7de265cb2d849 Rafael Passos           2024-04-17  19300  			 * current state is equivalent to it (except precision marks)
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19301  			 * the precision needs to be propagated back in
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19302  			 * the current state.
a3ce685dd01a78 Alexei Starovoitov      2019-06-28  19303  			 */
41f6f64e6999a8 Andrii Nakryiko         2023-12-05  19304  			if (is_jmp_point(env, env->insn_idx))
baaebe0928bf32 Eduard Zingerman        2025-06-11 @19305  				err = err ? : push_jmp_history(env, cur, 0, 0);
23b37d616565c8 Eduard Zingerman        2025-06-11  19306  			err = err ? : propagate_precision(env, &sl->state, cur, NULL);
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19307  			if (err)
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19308  				return err;
c9e31900b54cad Eduard Zingerman        2025-06-11  19309  			/* When processing iterator based loops above propagate_liveness and
c9e31900b54cad Eduard Zingerman        2025-06-11  19310  			 * propagate_precision calls are not sufficient to transfer all relevant
c9e31900b54cad Eduard Zingerman        2025-06-11  19311  			 * read and precision marks. E.g. consider the following case:
c9e31900b54cad Eduard Zingerman        2025-06-11  19312  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19313  			 *  .-> A --.  Assume the states are visited in the order A, B, C.
c9e31900b54cad Eduard Zingerman        2025-06-11  19314  			 *  |   |   |  Assume that state B reaches a state equivalent to state A.
c9e31900b54cad Eduard Zingerman        2025-06-11  19315  			 *  |   v   v  At this point, state C is not processed yet, so state A
c9e31900b54cad Eduard Zingerman        2025-06-11  19316  			 *  '-- B   C  has not received any read or precision marks from C.
c9e31900b54cad Eduard Zingerman        2025-06-11  19317  			 *             Thus, marks propagated from A to B are incomplete.
c9e31900b54cad Eduard Zingerman        2025-06-11  19318  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19319  			 * The verifier mitigates this by performing the following steps:
c9e31900b54cad Eduard Zingerman        2025-06-11  19320  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19321  			 * - Prior to the main verification pass, strongly connected components
c9e31900b54cad Eduard Zingerman        2025-06-11  19322  			 *   (SCCs) are computed over the program's control flow graph,
c9e31900b54cad Eduard Zingerman        2025-06-11  19323  			 *   intraprocedurally.
c9e31900b54cad Eduard Zingerman        2025-06-11  19324  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19325  			 * - During the main verification pass, `maybe_enter_scc()` checks
c9e31900b54cad Eduard Zingerman        2025-06-11  19326  			 *   whether the current verifier state is entering an SCC. If so, an
c9e31900b54cad Eduard Zingerman        2025-06-11  19327  			 *   instance of a `bpf_scc_visit` object is created, and the state
c9e31900b54cad Eduard Zingerman        2025-06-11  19328  			 *   entering the SCC is recorded as the entry state.
c9e31900b54cad Eduard Zingerman        2025-06-11  19329  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19330  			 * - This instance is associated not with the SCC itself, but with a
c9e31900b54cad Eduard Zingerman        2025-06-11  19331  			 *   `bpf_scc_callchain`: a tuple consisting of the call sites leading to
c9e31900b54cad Eduard Zingerman        2025-06-11  19332  			 *   the SCC and the SCC id. See `compute_scc_callchain()`.
c9e31900b54cad Eduard Zingerman        2025-06-11  19333  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19334  			 * - When a verification path encounters a `states_equal(...,
c9e31900b54cad Eduard Zingerman        2025-06-11  19335  			 *   RANGE_WITHIN)` condition, there exists a call chain describing the
c9e31900b54cad Eduard Zingerman        2025-06-11  19336  			 *   current state and a corresponding `bpf_scc_visit` instance. A copy
c9e31900b54cad Eduard Zingerman        2025-06-11  19337  			 *   of the current state is created and added to
c9e31900b54cad Eduard Zingerman        2025-06-11  19338  			 *   `bpf_scc_visit->backedges`.
c9e31900b54cad Eduard Zingerman        2025-06-11  19339  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19340  			 * - When a verification path terminates, `maybe_exit_scc()` is called
c9e31900b54cad Eduard Zingerman        2025-06-11  19341  			 *   from `update_branch_counts()`. For states with `branches == 0`, it
c9e31900b54cad Eduard Zingerman        2025-06-11  19342  			 *   checks whether the state is the entry state of any `bpf_scc_visit`
c9e31900b54cad Eduard Zingerman        2025-06-11  19343  			 *   instance. If it is, this indicates that all paths originating from
c9e31900b54cad Eduard Zingerman        2025-06-11  19344  			 *   this SCC visit have been explored. `propagate_backedges()` is then
c9e31900b54cad Eduard Zingerman        2025-06-11  19345  			 *   called, which propagates read and precision marks through the
c9e31900b54cad Eduard Zingerman        2025-06-11  19346  			 *   backedges until a fixed point is reached.
c9e31900b54cad Eduard Zingerman        2025-06-11  19347  			 *   (In the earlier example, this would propagate marks from A to B,
c9e31900b54cad Eduard Zingerman        2025-06-11  19348  			 *    from C to A, and then again from A to B.)
c9e31900b54cad Eduard Zingerman        2025-06-11  19349  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19350  			 * A note on callchains
c9e31900b54cad Eduard Zingerman        2025-06-11  19351  			 * --------------------
c9e31900b54cad Eduard Zingerman        2025-06-11  19352  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19353  			 * Consider the following example:
c9e31900b54cad Eduard Zingerman        2025-06-11  19354  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19355  			 *     void foo() { loop { ... SCC#1 ... } }
c9e31900b54cad Eduard Zingerman        2025-06-11  19356  			 *     void main() {
c9e31900b54cad Eduard Zingerman        2025-06-11  19357  			 *       A: foo();
c9e31900b54cad Eduard Zingerman        2025-06-11  19358  			 *       B: ...
c9e31900b54cad Eduard Zingerman        2025-06-11  19359  			 *       C: foo();
c9e31900b54cad Eduard Zingerman        2025-06-11  19360  			 *     }
c9e31900b54cad Eduard Zingerman        2025-06-11  19361  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19362  			 * Here, there are two distinct callchains leading to SCC#1:
c9e31900b54cad Eduard Zingerman        2025-06-11  19363  			 * - (A, SCC#1)
c9e31900b54cad Eduard Zingerman        2025-06-11  19364  			 * - (C, SCC#1)
c9e31900b54cad Eduard Zingerman        2025-06-11  19365  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19366  			 * Each callchain identifies a separate `bpf_scc_visit` instance that
c9e31900b54cad Eduard Zingerman        2025-06-11  19367  			 * accumulates backedge states. The `propagate_{liveness,precision}()`
c9e31900b54cad Eduard Zingerman        2025-06-11  19368  			 * functions traverse the parent state of each backedge state, which
c9e31900b54cad Eduard Zingerman        2025-06-11  19369  			 * means these parent states must remain valid (i.e., not freed) while
c9e31900b54cad Eduard Zingerman        2025-06-11  19370  			 * the corresponding `bpf_scc_visit` instance exists.
c9e31900b54cad Eduard Zingerman        2025-06-11  19371  			 *
c9e31900b54cad Eduard Zingerman        2025-06-11  19372  			 * Associating `bpf_scc_visit` instances directly with SCCs instead of
c9e31900b54cad Eduard Zingerman        2025-06-11  19373  			 * callchains would break this invariant:
c9e31900b54cad Eduard Zingerman        2025-06-11  19374  			 * - States explored during `C: foo()` would contribute backedges to
c9e31900b54cad Eduard Zingerman        2025-06-11  19375  			 *   SCC#1, but SCC#1 would only be exited once the exploration of
c9e31900b54cad Eduard Zingerman        2025-06-11  19376  			 *   `A: foo()` completes.
c9e31900b54cad Eduard Zingerman        2025-06-11  19377  			 * - By that time, the states explored between `A: foo()` and `C: foo()`
c9e31900b54cad Eduard Zingerman        2025-06-11  19378  			 *   (i.e., `B: ...`) may have already been freed, causing the parent
c9e31900b54cad Eduard Zingerman        2025-06-11  19379  			 *   links for states from `C: foo()` to become invalid.
c9e31900b54cad Eduard Zingerman        2025-06-11  19380  			 */
c9e31900b54cad Eduard Zingerman        2025-06-11  19381  			if (loop) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19382  				struct bpf_scc_backedge *backedge;
c9e31900b54cad Eduard Zingerman        2025-06-11  19383  
43736ec3e02789 Eduard Zingerman        2025-06-13  19384  				backedge = kzalloc(sizeof(*backedge), GFP_KERNEL_ACCOUNT);
c9e31900b54cad Eduard Zingerman        2025-06-11  19385  				if (!backedge)
c9e31900b54cad Eduard Zingerman        2025-06-11  19386  					return -ENOMEM;
c9e31900b54cad Eduard Zingerman        2025-06-11  19387  				err = copy_verifier_state(&backedge->state, cur);
c9e31900b54cad Eduard Zingerman        2025-06-11  19388  				backedge->state.equal_state = &sl->state;
c9e31900b54cad Eduard Zingerman        2025-06-11  19389  				backedge->state.insn_idx = insn_idx;
c9e31900b54cad Eduard Zingerman        2025-06-11  19390  				err = err ?: add_scc_backedge(env, &sl->state, backedge);
c9e31900b54cad Eduard Zingerman        2025-06-11  19391  				if (err) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19392  					free_verifier_state(&backedge->state, false);
bf0c2a84df9fb0 Qianfeng Rong           2025-08-11  19393  					kfree(backedge);
c9e31900b54cad Eduard Zingerman        2025-06-11  19394  					return err;
c9e31900b54cad Eduard Zingerman        2025-06-11  19395  				}
c9e31900b54cad Eduard Zingerman        2025-06-11  19396  			}
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19397  			return 1;
dc503a8ad98474 Edward Cree             2017-08-15  19398  		}
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19399  miss:
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19400  		/* when new state is not going to be added do not increase miss count.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19401  		 * Otherwise several loop iterations will remove the state
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19402  		 * recorded earlier. The goal of these heuristics is to have
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19403  		 * states from some iterations of the loop (some in the beginning
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19404  		 * and some at the end) to help pruning.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19405  		 */
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19406  		if (add_new_state)
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19407  			sl->miss_cnt++;
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19408  		/* heuristic to determine whether this state is beneficial
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19409  		 * to keep checking from state equivalence point of view.
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19410  		 * Higher numbers increase max_states_per_insn and verification time,
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19411  		 * but do not meaningfully decrease insn_processed.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19412  		 * 'n' controls how many times state could miss before eviction.
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19413  		 * Use bigger 'n' for checkpoints because evicting checkpoint states
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19414  		 * too early would hinder iterator convergence.
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19415  		 */
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19416  		n = is_force_checkpoint(env, insn_idx) && sl->state.branches > 0 ? 64 : 3;
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19417  		if (sl->miss_cnt > sl->hit_cnt * n + n) {
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19418  			/* the state is unlikely to be useful. Remove it to
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19419  			 * speed up verification
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19420  			 */
408fcf946b2bad Eduard Zingerman        2025-02-15  19421  			sl->in_free_list = true;
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19422  			list_del(&sl->node);
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19423  			list_add(&sl->node, &env->free_list);
574078b001cdf6 Eduard Zingerman        2025-02-15  19424  			env->free_list_size++;
574078b001cdf6 Eduard Zingerman        2025-02-15  19425  			env->explored_states_size--;
408fcf946b2bad Eduard Zingerman        2025-02-15  19426  			maybe_free_verifier_state(env, sl);
9f4686c41bdff0 Alexei Starovoitov      2019-04-01  19427  		}
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19428  	}
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19429  
06ee7115b0d174 Alexei Starovoitov      2019-04-01  19430  	if (env->max_states_per_insn < states_cnt)
06ee7115b0d174 Alexei Starovoitov      2019-04-01  19431  		env->max_states_per_insn = states_cnt;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19432  
2c78ee898d8f10 Alexei Starovoitov      2020-05-13  19433  	if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
a095f421057e22 Andrii Nakryiko         2022-12-06  19434  		return 0;
ceefbc96fa5c5b Alexei Starovoitov      2018-12-03  19435  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19436  	if (!add_new_state)
a095f421057e22 Andrii Nakryiko         2022-12-06  19437  		return 0;
ceefbc96fa5c5b Alexei Starovoitov      2018-12-03  19438  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19439  	/* There were no equivalent states, remember the current one.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19440  	 * Technically the current state is not proven to be safe yet,
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19441  	 * but it will either reach outer most bpf_exit (which means it's safe)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19442  	 * or it will be rejected. When there are no loops the verifier won't be
f4d7e40a5b7157 Alexei Starovoitov      2017-12-14  19443  	 * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19444  	 * again on the way to bpf_exit.
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19445  	 * When looping the sl->state.branches will be > 0 and this state
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19446  	 * will not be considered for equivalence until branches == 0.
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19447  	 */
43736ec3e02789 Eduard Zingerman        2025-06-13  19448  	new_sl = kzalloc(sizeof(struct bpf_verifier_state_list), GFP_KERNEL_ACCOUNT);
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19449  	if (!new_sl)
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19450  		return -ENOMEM;
06ee7115b0d174 Alexei Starovoitov      2019-04-01  19451  	env->total_states++;
574078b001cdf6 Eduard Zingerman        2025-02-15  19452  	env->explored_states_size++;
574078b001cdf6 Eduard Zingerman        2025-02-15  19453  	update_peak_states(env);
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19454  	env->prev_jmps_processed = env->jmps_processed;
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19455  	env->prev_insn_processed = env->insn_processed;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19456  
7a830b53c17bba Andrii Nakryiko         2022-11-04  19457  	/* forget precise markings we inherited, see __mark_chain_precision */
7a830b53c17bba Andrii Nakryiko         2022-11-04  19458  	if (env->bpf_capable)
7a830b53c17bba Andrii Nakryiko         2022-11-04  19459  		mark_all_scalars_imprecise(env, cur);
7a830b53c17bba Andrii Nakryiko         2022-11-04  19460  
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19461  	/* add new state to the head of linked list */
679c782de14bd4 Edward Cree             2018-08-22  19462  	new = &new_sl->state;
679c782de14bd4 Edward Cree             2018-08-22  19463  	err = copy_verifier_state(new, cur);
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19464  	if (err) {
679c782de14bd4 Edward Cree             2018-08-22  19465  		free_verifier_state(new, false);
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19466  		kfree(new_sl);
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19467  		return err;
1969db47f8d0e8 Alexei Starovoitov      2017-11-01  19468  	}
dc2a4ebc0b44a2 Alexei Starovoitov      2019-05-21  19469  	new->insn_idx = insn_idx;
0df1a55afa832f Paul Chaignon           2025-07-01  19470  	verifier_bug_if(new->branches != 1, env,
0df1a55afa832f Paul Chaignon           2025-07-01  19471  			"%s:branches_to_explore=%d insn %d",
0df1a55afa832f Paul Chaignon           2025-07-01  19472  			__func__, new->branches, insn_idx);
c9e31900b54cad Eduard Zingerman        2025-06-11  19473  	err = maybe_enter_scc(env, new);
c9e31900b54cad Eduard Zingerman        2025-06-11  19474  	if (err) {
c9e31900b54cad Eduard Zingerman        2025-06-11  19475  		free_verifier_state(new, false);
e4980fa6463624 Feng Yang               2025-08-27  19476  		kfree(new_sl);
c9e31900b54cad Eduard Zingerman        2025-06-11  19477  		return err;
c9e31900b54cad Eduard Zingerman        2025-06-11  19478  	}
b5dc0163d8fd78 Alexei Starovoitov      2019-06-15  19479  
2589726d12a1b1 Alexei Starovoitov      2019-06-15  19480  	cur->parent = new;
b5dc0163d8fd78 Alexei Starovoitov      2019-06-15  19481  	cur->first_insn_idx = insn_idx;
2793a8b015f7f1 Eduard Zingerman        2023-10-24  19482  	cur->dfs_depth = new->dfs_depth + 1;
baaebe0928bf32 Eduard Zingerman        2025-06-11  19483  	clear_jmp_history(cur);
5564ee3abb2ebe Eduard Zingerman        2025-02-15  19484  	list_add(&new_sl->node, head);
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19485  	return 0;
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19486  }
f1bca824dabba4 Alexei Starovoitov      2014-09-29  19487  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author maintainer
#15 lore raw

Re: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness

Quoted 11 lines · Dan Carpenter

Hi Eduard,

kernel test robot noticed the following build warnings:

url:    https://github.com/intel-lab-lkp/linux/commits/Eduard-Zingerman/bpf-bpf_verifier_state-cleaned-flag-instead-of-REG_LIVE_DONE/20250911-090604
base:   https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link:    https://lore.kernel.org/r/20250911010437.2779173-10-eddyz87%40gmail.com
patch subject: [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
config: arm-randconfig-r071-20250911 (https://download.01.org/0day-ci/archive/20250912/[email protected]/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 21857ae337e0892a5522b6e7337899caa61de2a6)
Quoted 8 lines

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

Quoted 3 lines

smatch warnings:
kernel/bpf/verifier.c:19305 is_state_visited() error: uninitialized symbol 'err'.

Hi Dan,

Thank you for the report, kernel test robot already notified me here:
https://lore.kernel.org/bpf/[email protected]/T/#m103a269f4c096b34043bef16fa5f1d629b794968

Thanks,
Eduard

[...]

author maintainer
#16 lore raw

PATCH 10/10 bpf: table based bpf_insn_successors()

In reply to Eduard Zingerman

Converting bpf_insn_successors() to use lookup table makes it ~1.5 times faster.

Also remove unnecessary conditionals:
- idx + 1 < prog->len is unnecessary because after check_cfg() all jump targets are guaranteed to be within a program;
- i == 0 || succ[0] != dst is unnecessary because any client of bpf_insn_successors() can handle duplicate edges:
- compute_live_registers()
- compute_scc()

Moving bpf_insn_successors() to liveness.c allows its inlining in liveness.c:__update_stack_liveness().
Such inlining speeds up __update_stack_liveness() by ~40%. bpf_insn_successors() is used in both verifier.c and liveness.c. perf shows such move does not negatively impact users in verifier.c, as these are executed only once before main varification pass. Unlike __update_stack_liveness() which can be triggered multiple times.

---
 include/linux/bpf_verifier.h |  1 +
 kernel/bpf/liveness.c        | 51 +++++++++++++++++++++++++
 kernel/bpf/verifier.c        | 72 +-----------------------------------
 3 files changed, 53 insertions(+), 71 deletions(-)
include/linux/bpf_verifier.h +1 -0
index c7515da8500c..4c497e839526 100644
@@ -1049,6 +1049,7 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st
 		      u32 frameno);
 
 struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *env, int off);
+int bpf_jmp_offset(struct bpf_insn *insn);
 int bpf_insn_successors(struct bpf_prog *prog, u32 idx, u32 succ[2]);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx);
kernel/bpf/liveness.c +51 -0

57 lines not on this page · raw message

kernel/bpf/verifier.c +1 -71

90 lines not on this page · raw message

Signature · 1 line
2.47.3
syzbot Tested-by [email protected] on the v1 series
#17 lore raw
In reply to Eduard Zingerman
syzbot ci has tested the following series

syzbot ci has tested the following series

[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: use compute_live_registers() info in clean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundant REG_LIVE_READ check in stacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table based bpf_insn_successors()

and found the following issue:

KASAN: slab-out-of-bounds Write in compute_postorder

Full report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17

***

KASAN: slab-out-of-bounds Write in compute_postorder
tree:      bpf-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git
base:      e12873ee856ffa6f104869b8ea10c0f741606f13
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config
C repro:   https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro
syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro

==================================================================

BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840
Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991

CPU: 0 UID: 0 PID: 5991 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Call Trace:
 <TASK>
 dump_stack_lvl+0x189/0x250 lib/dump_stack.c:120
 print_address_description mm/kasan/report.c:378 [inline]
 print_report+0xca/0x240 mm/kasan/report.c:482
 kasan_report+0x118/0x150 mm/kasan/report.c:595
 compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840
 bpf_check+0x1f90/0x1d440 kernel/bpf/verifier.c:24437
 bpf_prog_load+0x1318/0x1930 kernel/bpf/syscall.c:2979
 __sys_bpf+0x528/0x870 kernel/bpf/syscall.c:6029
 __do_sys_bpf kernel/bpf/syscall.c:6139 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:6137 [inline]
 __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6137
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0x3b0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f366058eba9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffef8486b28 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
RAX: ffffffffffffffda RBX: 00007f36607d5fa0 RCX: 00007f366058eba9
RDX: 0000000000000070 RSI: 0000200000000440 RDI: 0000000000000005
RBP: 00007f3660611e19 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f36607d5fa0 R14: 00007f36607d5fa0 R15: 0000000000000003
 </TASK>

Allocated by task 5991:
 kasan_save_stack mm/kasan/common.c:47 [inline]
 kasan_save_track+0x3e/0x80 mm/kasan/common.c:68
 poison_kmalloc_redzone mm/kasan/common.c:388 [inline]
 __kasan_kmalloc+0x93/0xb0 mm/kasan/common.c:405
 kasan_kmalloc include/linux/kasan.h:260 [inline]
 __do_kmalloc_node mm/slub.c:4365 [inline]
 __kvmalloc_node_noprof+0x30d/0x5f0 mm/slub.c:5052
 kvmalloc_array_node_noprof include/linux/slab.h:1065 [inline]
 compute_postorder+0xd6/0xcb0 kernel/bpf/verifier.c:17823
 bpf_check+0x1f90/0x1d440 kernel/bpf/verifier.c:24437
 bpf_prog_load+0x1318/0x1930 kernel/bpf/syscall.c:2979
 __sys_bpf+0x528/0x870 kernel/bpf/syscall.c:6029
 __do_sys_bpf kernel/bpf/syscall.c:6139 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:6137 [inline]
 __x64_sys_bpf+0x7c/0x90 kernel/bpf/syscall.c:6137
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0x3b0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

The buggy address belongs to the object at ffff88801f1d4b80
 which belongs to the cache kmalloc-cg-32 of size 32

The buggy address is located 0 bytes to the right of
allocated 24-byte region [ffff88801f1d4b80, ffff88801f1d4b98)

The buggy address belongs to the physical page:
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0xffff88801f1d4e40 pfn:0x1f1d4
memcg:ffff888026bbb801
flags: 0xfff00000000000(node=0|zone=1|lastcpupid=0x7ff)
page_type: f5(slab)
raw: 00fff00000000000 ffff88801a449b40 dead000000000100 dead000000000122
raw: ffff88801f1d4e40 000000008040003f 00000000f5000000 ffff888026bbb801
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0x52cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP), pid 532, tgid 532 (kworker/u10:0), ts 5351847364, free_ts 0
 set_page_owner include/linux/page_owner.h:32 [inline]
 post_alloc_hook+0x240/0x2a0 mm/page_alloc.c:1851
 prep_new_page mm/page_alloc.c:1859 [inline]
 get_page_from_freelist+0x21e4/0x22c0 mm/page_alloc.c:3858
 __alloc_frozen_pages_noprof+0x181/0x370 mm/page_alloc.c:5148
 alloc_pages_mpol+0x232/0x4a0 mm/mempolicy.c:2416
 alloc_slab_page mm/slub.c:2487 [inline]
 allocate_slab+0x8a/0x370 mm/slub.c:2655
 new_slab mm/slub.c:2709 [inline]
 ___slab_alloc+0xbeb/0x1410 mm/slub.c:3891
 __slab_alloc mm/slub.c:3981 [inline]
 __slab_alloc_node mm/slub.c:4056 [inline]
 slab_alloc_node mm/slub.c:4217 [inline]
 __do_kmalloc_node mm/slub.c:4364 [inline]
 __kmalloc_noprof+0x305/0x4f0 mm/slub.c:4377
 kmalloc_noprof include/linux/slab.h:909 [inline]
 kzalloc_noprof include/linux/slab.h:1039 [inline]
 lsm_blob_alloc security/security.c:684 [inline]
 lsm_cred_alloc security/security.c:701 [inline]
 security_prepare_creds+0x52/0x390 security/security.c:3271
 prepare_kernel_cred+0x2ee/0x500 kernel/cred.c:617
 call_usermodehelper_exec_async+0xd0/0x360 kernel/umh.c:88
 ret_from_fork+0x3fc/0x770 arch/x86/kernel/process.c:148
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245

page_owner free stack trace missing

Memory state around the buggy address:
 ffff88801f1d4a80: fa fb fb fb fc fc fc fc 00 00 00 fc fc fc fc fc
 ffff88801f1d4b00: 00 00 00 fc fc fc fc fc fa fb fb fb fc fc fc fc
>ffff88801f1d4b80: 00 00 00 fc fc fc fc fc fa fb fb fb fc fc fc fc
                            ^
 ffff88801f1d4c00: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
 ffff88801f1d4c80: fa fb fb fb fc fc fc fc fa fb fb fb fc fc fc fc
==================================================================

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

author maintainer
#18 lore raw
Quoted 34 lines · syzbot ci

syzbot ci has tested the following series

[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: use compute_live_registers() info in clean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundant REG_LIVE_READ check in stacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table based bpf_insn_successors()

and found the following issue:

KASAN: slab-out-of-bounds Write in compute_postorder

Full report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17

***

KASAN: slab-out-of-bounds Write in compute_postorder
tree:      bpf-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git
base:      e12873ee856ffa6f104869b8ea10c0f741606f13
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config
C repro:   https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro
syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro
Quoted 3 lines

==================================================================

BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840
Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991

The error is caused by the following program:

(e5) if r15 (null) 0xffffffff goto pc-1 <---- absence of DISCOVERED/EXPLORED mark here
(71) r1 = *(u8 *)(r1 +70) leads to instruction being put on stack second time
(85) call pc+2
(85) call bpf_get_numa_node_id#42
(95) exit
(95) exit

And this is the fix:

kernel/bpf/verifier.c +1 -0
@@ -17840,6 +17840,7 @@ static int compute_postorder(struct bpf_verifier_env *env)
                stack_sz = 1;
                do {
                        top = stack[stack_sz - 1];
+                       state[top] |= DISCOVERED;
                        if (state[top] & EXPLORED) {
                                postorder[cur_postorder++] = top;
                                stack_sz--;

[...]

#19 lore raw
Quoted 36 lines · Eduard Zingerman
Quoted 34 lines · syzbot ci

syzbot ci has tested the following series

[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: use compute_live_registers() info in clean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundant REG_LIVE_READ check in stacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table based bpf_insn_successors()

and found the following issue:

KASAN: slab-out-of-bounds Write in compute_postorder

Full report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17

***

KASAN: slab-out-of-bounds Write in compute_postorder
tree:      bpf-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git
base:      e12873ee856ffa6f104869b8ea10c0f741606f13
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config
C repro:   https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro
syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro
Quoted 7 lines
Quoted 3 lines

==================================================================

BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840
Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991

The error is caused by the following program:

(e5) if r15 (null) 0xffffffff goto pc-1 <---- absence of DISCOVERED/EXPLORED mark here

(null) ?
Is it jset again? but insn_successors() handles it already. Or pc-1 infinite loop caused it?
but we have pc-1 selftest...

Quoted 19 lines

(71) r1 = *(u8 *)(r1 +70) leads to instruction being put on stack second time
(85) call pc+2
(85) call bpf_get_numa_node_id#42
(95) exit
(95) exit

And this is the fix:

kernel/bpf/verifier.c +1 -0
@@ -17840,6 +17840,7 @@ static int compute_postorder(struct bpf_verifier_env *env)
                stack_sz = 1;
                do {
                        top = stack[stack_sz - 1];
+                       state[top] |= DISCOVERED;
                        if (state[top] & EXPLORED) {
                                postorder[cur_postorder++] = top;
                                stack_sz--;
[...]
author maintainer
#20 lore raw
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Eduard Zingerman
Quoted 37 lines · syzbot ci

syzbot ci has tested the following series

[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: use compute_live_registers() info in clean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundant REG_LIVE_READ check in stacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table based bpf_insn_successors()

and found the following issue:

KASAN: slab-out-of-bounds Write in compute_postorder

Full report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17

***

KASAN: slab-out-of-bounds Write in compute_postorder
tree:      bpf-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git
base:      e12873ee856ffa6f104869b8ea10c0f741606f13
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config
C repro:   https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro
syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro

==================================================================

BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840
Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991
Quoted 5 lines
Quoted 3 lines

The error is caused by the following program:

(e5) if r15 (null) 0xffffffff goto pc-1 <---- absence of DISCOVERED/EXPLORED mark here

(null) ?

The code byte is 0xe5, BPF_OP(0xe5) == 0xe0, which is an invalid opcode. But opcodes are verified after check_cfg()/compute_postorder().

Quoted 3 lines

Is it jset again? but insn_successors() handles it already. Or pc-1 infinite loop caused it?
but we have pc-1 selftest...

It's not infinite, but it causes instruction to be put twice on the stack array, and this array is allocated expecting max prog->len instructions. KASAN would only catch this error if program really needs to consume full stack depth during postorder construction, as far as I understand.

[...]

#21 lore raw
Quoted 43 lines · Eduard Zingerman
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Eduard Zingerman
Quoted 37 lines · syzbot ci

syzbot ci has tested the following series

[v1] bpf: replace path-sensitive with path-insensitive live stack analysis
https://lore.kernel.org/all/[email protected]
* [PATCH bpf-next v1 01/10] bpf: bpf_verifier_state->cleaned flag instead of REG_LIVE_DONE
* [PATCH bpf-next v1 02/10] bpf: use compute_live_registers() info in clean_func_state
* [PATCH bpf-next v1 03/10] bpf: remove redundant REG_LIVE_READ check in stacksafe()
* [PATCH bpf-next v1 04/10] bpf: declare a few utility functions as internal api
* [PATCH bpf-next v1 05/10] bpf: compute instructions postorder per subprogram
* [PATCH bpf-next v1 06/10] bpf: callchain sensitive stack liveness tracking using CFG
* [PATCH bpf-next v1 07/10] bpf: enable callchain sensitive stack liveness tracking
* [PATCH bpf-next v1 08/10] bpf: signal error if old liveness is more conservative than new
* [PATCH bpf-next v1 09/10] bpf: disable and remove registers chain based liveness
* [PATCH bpf-next v1 10/10] bpf: table based bpf_insn_successors()

and found the following issue:

KASAN: slab-out-of-bounds Write in compute_postorder

Full report is available here:
https://ci.syzbot.org/series/c42e236b-f40c-4d72-8ae7-da4e21c37e17

***

KASAN: slab-out-of-bounds Write in compute_postorder
tree:      bpf-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/bpf/bpf-next.git
base:      e12873ee856ffa6f104869b8ea10c0f741606f13
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/6d2bc952-3d65-4bcd-9a84-1207b810a1b5/config
C repro:   https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/c_repro
syz repro: https://ci.syzbot.org/findings/338e6ce4-7207-484f-a508-9b00b3121701/syz_repro

==================================================================

BUG: KASAN: slab-out-of-bounds in compute_postorder+0x802/0xcb0 kernel/bpf/verifier.c:17840
Write of size 4 at addr ffff88801f1d4b98 by task syz.0.17/5991
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The error is caused by the following program:

(e5) if r15 (null) 0xffffffff goto pc-1 <---- absence of DISCOVERED/EXPLORED mark here

(null) ?

The code byte is 0xe5, BPF_OP(0xe5) == 0xe0, which is an invalid opcode. But opcodes are verified after check_cfg()/compute_postorder().

hmm.
resolve_pseudo_ldimm64() -> bpf_opcode_in_insntable() does it before check_cfg.