bpf: static stack liveness data flow analysis
Subsystems (4)
Eduard Zingerman
maintainerv2 cover bpf: static stack liveness data flow analysis
This patch set converts current dynamic stack slot liveness tracking mechanism to a static data flow analysis. The result is used during state pruning (clean_verifier_state): to zero out dead stack slots, enabling more aggressive state equivalence and pruning. To improve analysis precision live stack slot tracking is converted to 4-byte granularity.
The key ideas and the bulk of the execution behind the series belong to Alexei Starovoitov. I contributed to patch set integration with existing liveness tracking mechanism.
Due to complexity of the changes the bisectability property of the patch set is not preserved. Some selftests may fail between intermediate patches of the series.
Analysis consists of two passes:
- A forward fixed-point analysis that tracks which frame's FP each register value is derived from, and at what byte offset. This is needed because a callee can receive a pointer to its caller's stack frame (e.g. r1 = fp-16 at the call site), then do *(u64 *)(r1 + 0) inside the callee - a cross-frame stack access that the callee's local liveness must attribute to the caller's stack.
- A backward dataflow pass within each callee subprog that computes live_in = (live_out \ def) ∪ use for both local and non-local (ancestor) stack slots. The result of the analysis for callee is propagated up to the callsite.
The key idea making such analysis possible is that limited and conservative argument tracking pass is sufficient to recover most of the offsets / stack pointer arguments.
Changelog:
v1 -> v2:
liveness.c:
- Removed func_instance->callsites and replaced it with explicit spine passed through analys_subprog() calls (sashiko).
- Fixed BPF_LOAD_ACQ handling in arg_track_xfer: don't clear dst register tracking (sashiko).
- Various error threading nits highlighted by bots
(sashiko, bot+bpf-ci).
- Massaged fmt_spis_mask() to be more concise (Alexei)
verifier.c:
- Move subprog_info[i].name assignment from add_subprog_and_kfunc to check_btf_func (sashiko, bot+bpf-ci).
- Fixed inverse usage of msb/lsb halves by patch
"bpf: make liveness.c track stack with 4-byte granularity" (sashiko, bot+bpf-ci).
v1: https://lore.kernel.org/bpf/[email protected]/
Verification performance impact:
========= selftests: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF) ----------------------- ------------- --------- --------- --------------- xdp_synproxy_kern.bpf.o syncookie_tc 20363 22910 +2547 (+12.51%) xdp_synproxy_kern.bpf.o syncookie_xdp 20450 23001 +2551 (+12.47%)
Total progs: 4490 Old success: 2856 New success: 2856 total_insns diff min: -80.26% total_insns diff max: 12.51%
0 -> value: 0
value -> 0: 0total_insns abs max old: 837,487total_insns abs max new: 837,487
-85 .. -75 %: 1 -50 .. -40 %: 1 -35 .. -25 %: 1 -20 .. -10 %: 5 -10 .. 0 %: 18 0 .. 5 %: 4458 5 .. 15 %: 6
========= scx: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF) -------------- --------- --------- --------- -------------- scx_qmap.bpf.o qmap_init 20230 19022 -1208 (-5.97%)
Total progs: 376 Old success: 351 New success: 351 total_insns diff min: -27.15% total_insns diff max: 0.50%
0 -> value: 0
value -> 0: 0total_insns abs max old: 236,251total_insns abs max new: 233,669
-30 .. -20 %: 8 -20 .. -10 %: 2 -10 .. 0 %: 21 0 .. 5 %: 345
========= meta: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF) ----------------------------------------------------------------------------------------- ----------------------------- --------- --------- ----------------- ... third-party-scx-backports-6.9-1.0.13.1-scheds-rust-scx_layered-bpf_skel_genskel-bpf.bpf.o layered_dispatch 13944 13104 -840 (-6.02%) third-party-scx-backports-6.9-1.0.13.2-scheds-rust-scx_layered-bpf_skel_genskel-bpf.bpf.o layered_dispatch 13944 13104 -840 (-6.02%) third-party-scx-gefe21962f49a-__scx_layered_bpf_skel_genskel-bpf.bpf.o layered_dispatch 13825 12985 -840 (-6.08%) third-party-scx-v1.0.16-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_enqueue 15501 13602 -1899 (-12.25%) third-party-scx-v1.0.16-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_select_cpu 19814 16231 -3583 (-18.08%) third-party-scx-v1.0.17-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_enqueue 15501 13602 -1899 (-12.25%) third-party-scx-v1.0.17-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_select_cpu 19814 16231 -3583 (-18.08%) third-party-scx-v1.0.17-__scx_layered_bpf_skel_genskel-bpf.bpf.o layered_dispatch 13976 13151 -825 (-5.90%) third-party-scx-v1.0.18-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_dispatch 260628 237930 -22698 (-8.71%) third-party-scx-v1.0.18-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_enqueue 13437 12225 -1212 (-9.02%) third-party-scx-v1.0.18-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_select_cpu 17744 14730 -3014 (-16.99%) third-party-scx-v1.0.19-10-6b1958477-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_cpu_offline 19676 18418 -1258 (-6.39%) third-party-scx-v1.0.19-10-6b1958477-__scx_lavd_bpf_skel_genskel-bpf.bpf.o lavd_cpu_online 19674 18416 -1258 (-6.39%)
...
Total progs: 1540 Old success: 1492 New success: 1493 total_insns diff min: -75.83% total_insns diff max: 73.60%
0 -> value: 0
value -> 0: 0total_insns abs max old: 434,763total_insns abs max new: 666,036
-80 .. -70 %: 2 -55 .. -50 %: 7 -50 .. -45 %: 10 -45 .. -35 %: 4 -35 .. -25 %: 4 -25 .. -20 %: 8 -20 .. -15 %: 15 -15 .. -10 %: 11 -10 .. -5 %: 45 -5 .. 0 %: 112 0 .. 5 %: 1316 5 .. 15 %: 2 15 .. 25 %: 1 25 .. 35 %: 1 55 .. 65 %: 1 70 .. 75 %: 1
========= cilium: master vs patch-set =========
File Program Insns (A) Insns (B) Insns (DIFF) --------------- --------------------------------- --------- --------- ---------------- bpf_host.o cil_host_policy 45801 32027 -13774 (-30.07%) bpf_host.o cil_to_netdev 100287 69042 -31245 (-31.16%) bpf_host.o tail_handle_ipv4_cont_from_host 60911 20962 -39949 (-65.59%) bpf_host.o tail_handle_ipv4_from_netdev 59735 33155 -26580 (-44.50%) bpf_host.o tail_handle_ipv6_cont_from_host 23529 17036 -6493 (-27.60%) bpf_host.o tail_handle_ipv6_from_host 11906 10303 -1603 (-13.46%) bpf_host.o tail_handle_ipv6_from_netdev 29778 23743 -6035 (-20.27%) bpf_host.o tail_handle_snat_fwd_ipv4 61616 67463 +5847 (+9.49%) bpf_host.o tail_handle_snat_fwd_ipv6 30802 22806 -7996 (-25.96%) bpf_host.o tail_ipv4_host_policy_ingress 20017 10528 -9489 (-47.40%) bpf_host.o tail_ipv6_host_policy_ingress 20693 17301 -3392 (-16.39%) bpf_host.o tail_nodeport_nat_egress_ipv4 16455 13684 -2771 (-16.84%) bpf_host.o tail_nodeport_nat_ingress_ipv4 36174 20080 -16094 (-44.49%) bpf_host.o tail_nodeport_nat_ingress_ipv6 48039 25779 -22260 (-46.34%) bpf_lxc.o tail_handle_ipv4 13765 10001 -3764 (-27.34%) bpf_lxc.o tail_handle_ipv4_cont 96891 68725 -28166 (-29.07%) bpf_lxc.o tail_handle_ipv6_cont 21809 17697 -4112 (-18.85%) bpf_lxc.o tail_ipv4_ct_egress 15949 17746 +1797 (+11.27%) bpf_lxc.o tail_nodeport_nat_egress_ipv4 16183 13432 -2751 (-17.00%) bpf_lxc.o tail_nodeport_nat_ingress_ipv4 18532 10697 -7835 (-42.28%) bpf_overlay.o tail_handle_inter_cluster_revsnat 15708 11099 -4609 (-29.34%) bpf_overlay.o tail_handle_ipv4 105672 76108 -29564 (-27.98%) bpf_overlay.o tail_handle_ipv6 15733 19944 +4211 (+26.77%) bpf_overlay.o tail_handle_snat_fwd_ipv4 19327 26468 +7141 (+36.95%) bpf_overlay.o tail_handle_snat_fwd_ipv6 20817 12556 -8261 (-39.68%) bpf_overlay.o tail_nodeport_nat_egress_ipv4 16175 12184 -3991 (-24.67%) bpf_overlay.o tail_nodeport_nat_ingress_ipv4 20760 11951 -8809 (-42.43%) bpf_wireguard.o tail_handle_ipv4 27466 28909 +1443 (+5.25%) bpf_wireguard.o tail_nodeport_nat_egress_ipv4 15937 12094 -3843 (-24.11%) bpf_wireguard.o tail_nodeport_nat_ingress_ipv4 20624 11993 -8631 (-41.85%) bpf_xdp.o tail_lb_ipv4 42673 60855 +18182 (+42.61%) bpf_xdp.o tail_lb_ipv6 87903 108585 +20682 (+23.53%) bpf_xdp.o tail_nodeport_nat_ingress_ipv4 28787 20991 -7796 (-27.08%) bpf_xdp.o tail_nodeport_nat_ingress_ipv6 207593 152012 -55581 (-26.77%)
Total progs: 134 Old success: 134 New success: 134 total_insns diff min: -65.59% total_insns diff max: 42.61%
0 -> value: 0
value -> 0: 0total_insns abs max old: 207,593total_insns abs max new: 152,012
-70 .. -60 %: 1 -50 .. -40 %: 7 -40 .. -30 %: 9 -30 .. -25 %: 9 -25 .. -20 %: 12 -20 .. -15 %: 7 -15 .. -10 %: 14 -10 .. -5 %: 6 -5 .. 0 %: 16 0 .. 5 %: 42 5 .. 15 %: 5 15 .. 25 %: 2 25 .. 35 %: 2 35 .. 45 %: 2
---
Alexei Starovoitov (5):
bpf: Add spis_*() helpers for 4-byte stack slot bitmasks selftests/bpf: update existing tests due to liveness changes selftests/bpf: adjust verifier_log buffers selftests/bpf: add new tests for static stack liveness analysis
bpf: poison dead stack slots
Eduard Zingerman (8):
bpf: share several utility functions as internal API
bpf: save subprogram name in bpf_subprog_info
bpf: make liveness.c track stack with 4-byte granularity
bpf: 4-byte precise clean_verifier_state
bpf: prepare bpf_liveness api for use by static analysis pass
bpf: introduce forward arg-tracking dataflow analysis
bpf: simplify liveness to use (callsite, depth) keyed func_instances
bpf: change logging scheme for live stack analysis
include/linux/bpf_verifier.h | 63 +- kernel/bpf/liveness.c | 1991 ++++++++++++++---- kernel/bpf/log.c | 9 +- kernel/bpf/verifier.c | 267 ++- .../selftests/bpf/prog_tests/verifier_log.c | 6 +- .../selftests/bpf/progs/exceptions_assert.c | 6 +- tools/testing/selftests/bpf/progs/uninit_stack.c | 1 + tools/testing/selftests/bpf/progs/verifier_align.c | 10 +- .../selftests/bpf/progs/verifier_div_mod_bounds.c | 18 +- .../selftests/bpf/progs/verifier_live_stack.c | 2217 +++++++++++++++++++- .../selftests/bpf/progs/verifier_scalar_ids.c | 26 +- .../selftests/bpf/progs/verifier_spill_fill.c | 10 +- .../bpf/progs/verifier_subprog_precision.c | 8 +- 13 files changed, 4013 insertions(+), 619 deletions(-)
---
base-commit: 6a14beefab457f267b8cedc6ac697a9562ec1244
change-id: 20260408-patch-set-29efd8c0f810
Eduard Zingerman
maintainerPATCH v2 1/13 bpf: share several utility functions as internal API
Namely:
- bpf_subprog_is_global
- bpf_vlog_alignment
--- include/linux/bpf_verifier.h | 2 ++ kernel/bpf/log.c | 4 ++-- kernel/bpf/verifier.c | 10 +++++----- 3 files changed, 9 insertions(+), 7 deletions(-)
include/linux/bpf_verifier.h +2 -0
@@ -1121,12 +1121,14 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie u32 frameno, bool print_all);void print_insn_state (struct bpf_verifier_env *env,const struct bpf_verifier_state *vstate,u32 frameno); +u32 bpf_vlog_alignment (u32 pos);struct bpf_subprog_info *bpf_find_containing_subprog (struct bpf_verifier_env *env,int off);int bpf_jmp_offset (struct bpf_insn *insn);struct bpf_iarray *bpf_insn_successors (struct bpf_verifier_env *env,u32 idx);void bpf_fmt_stack_mask (char *buf,ssize_t buf_sz,u64 stack_mask);bool bpf_calls_callback (struct bpf_verifier_env *env,int insn_idx); +bool bpf_subprog_is_global (const struct bpf_verifier_env *env,int subprog);int bpf_find_subprog (struct bpf_verifier_env *env,int off);int bpf_compute_const_regs (struct bpf_verifier_env *env);
kernel/bpf/log.c +2 -2
@@ -847,7 +847,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifiemark_verifier_state_clean (env); } -static inline u32 vlog_alignment (u32 pos) +u32 bpf_vlog_alignment (u32 pos) {return round_up (max (pos +BPF_LOG_MIN_ALIGNMENT /2 ,BPF_LOG_ALIGNMENT ),BPF_LOG_MIN_ALIGNMENT ) - pos -1 ;
@@ -859,7 +859,7 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_stif (env->prev_log_pos && env->prev_log_pos == env->log.end_pos) {/* remove new line character */ bpf_vlog_reset (&env->log, env->prev_log_pos -1 ); -verbose (env,"%*c;" ,vlog_alignment (env->prev_insn_print_pos),' ' ); +verbose (env,"%*c;" ,bpf_vlog_alignment (env->prev_insn_print_pos),' ' ); }else {verbose (env,"%d:" , env->insn_idx); }
kernel/bpf/verifier.c +5 -5
@@ -421,7 +421,7 @@ static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg)return rec; } -static bool subprog_is_global (const struct bpf_verifier_env *env,int subprog) +bool bpf_subprog_is_global (const struct bpf_verifier_env *env,int subprog) {struct bpf_func_info_aux *aux = env->prog->aux->func_info_aux;
@@ -4610,7 +4610,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,if (subprog <0 )return -EFAULT ; -if (subprog_is_global (env, subprog)) { +if (bpf_subprog_is_global (env, subprog)) {/* check that jump history doesn't have any * extra instructions from subprog; the next * instruction after call to global subprog
@@ -6971,7 +6971,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,if (subprog[idx].has_tail_call) tail_call_reachable = true; - frame =subprog_is_global (env, idx) ?0 : frame +1 ; + frame =bpf_subprog_is_global (env, idx) ?0 : frame +1 ;if (frame >=MAX_CALL_FRAMES ) {verbose (env,"the call stack of %d frames is too deep !\n" , frame);
@@ -11040,7 +11040,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, err =btf_check_subprog_call (env, subprog, caller->regs);if (err == -EFAULT )return err; -if (subprog_is_global (env, subprog)) { +if (bpf_subprog_is_global (env, subprog)) {const char *sub_name =subprog_name (env, subprog);if (env->cur_state->active_locks) {
@@ -25186,7 +25186,7 @@ static int do_check_subprogs(struct bpf_verifier_env *env) again: new_cnt =0 ;for (i =1 ; i < env->subprog_cnt; i++) { -if (!subprog_is_global (env, i)) +if (!bpf_subprog_is_global (env, i))continue ; sub_aux =subprog_aux (env, i);
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 2/13 bpf: save subprogram name in bpf_subprog_info
In reply to Eduard ZingermanSubprogram name can be computed from function info and BTF, but it is convenient to have the name readily available for logging purposes.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 1 + 2 files changed, 2 insertions(+)
include/linux/bpf_verifier.h +1 -0
@@ -664,6 +664,7 @@ enum priv_stack_mode { };struct bpf_subprog_info { +const char *name;/* 'start' has to be the first field otherwise find_subprog() won't work */ u32 start;/* insn idx of function entry point */ u32 linfo_idx;/* The idx to the main_prog->aux->linfo */
kernel/bpf/verifier.c +1 -0
@@ -19682,6 +19682,7 @@ static int check_btf_func(struct bpf_verifier_env *env, goto err_free; } + env->subprog_info[i].name =btf_name_by_offset (btf, type->name_off);bpfptr_add (&urecord, urec_size); }
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v2 02/13] bpf: save subprogram name in bpf_subprog_info
With name added before start, the comment claiming start has to be the first field is now inaccurate. Looking at find_subprog() and bpf_find_containing_subprog(), the binary search accesses the…
include/linux/bpf_verifier.h enum priv_stack_mode { · line 669
Quoted 11 lines
include/linux/bpf_verifier.h +1
-0@@ -664,6 +664,7 @@ enum priv_stack_mode { };struct bpf_subprog_info { +const char *name;/* 'start' has to be the first field otherwise find_subprog() won't work */ u32 start;/* insn idx of function entry point */
With name added before start, the comment claiming start has to be the first field is now inaccurate. Looking at find_subprog() and bpf_find_containing_subprog(), the binary search accesses the field by name (vals[m].start), not by raw offset, so field ordering doesn't actually matter:
bpf_find_containing_subprog() {
...
while (l < r) {
m = l + (r - l + 1) / 2;
if (vals[m].start <= off)
l = m;
else
r = m - 1;
}
...
}Should the comment be updated or removed to avoid confusing future readers?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 02/13] bpf: save subprogram name in bpf_subprog_info
Quoted 30 lines · On Fri, 2026-04-10 at 01:50 +0000, [email protected] wrote:
Quoted 11 lines
include/linux/bpf_verifier.h +1
-0@@ -664,6 +664,7 @@ enum priv_stack_mode { }; struct bpf_subprog_info { +const char *name; /* 'start' has to be the first field otherwise find_subprog() won't work */u32 start; /* insn idx of function entry point */
With name added before start, the comment claiming start has to be the first field is now inaccurate. Looking at
find_subprog()andbpf_find_containing_subprog(), the binary search accesses the field by name (vals[m].start), not by raw offset, so field ordering doesn't actually matter:
bpf_find_containing_subprog(){
...
while (l < r) {
m = l + (r - l + 1) / 2;
if (vals[m].start <= off)
l = m;
else
r = m - 1;
}
...
}
Quoted 2 lines
Should the comment be updated or removed to avoid confusing future readers?
Comment does not seem to be accurate indeed.
I'll move the field and update the comment.
Quoted 7 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerPATCH v2 3/13 bpf: Add spis_*() helpers for 4-byte stack slot bitmasks
In reply to Eduard ZingermanFrom: Alexei Starovoitov <[email protected]>
Add helper functions for manipulating u64[2] bitmasks that represent 4-byte stack slot liveness. The 512-byte BPF stack is divided into 128 4-byte slots, requiring 128 bits (two u64s) to track.
These will be used by the static stack liveness analysis in the next commit.
--- include/linux/bpf_verifier.h | 67 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
include/linux/bpf_verifier.h +67 -0
@@ -224,6 +224,73 @@ enum bpf_stack_slot_type {#define BPF_REG_SIZE 8/* size of eBPF register in bytes */ +/* 4-byte stack slot granularity for liveness analysis */ +#define BPF_HALF_REG_SIZE 4 +#define STACK_SLOTS (MAX_BPF_STACK / BPF_HALF_REG_SIZE)/* 128 */ + +typedef struct { +u64 v[2 ]; +}spis_t ; + +#define SPIS_ZERO ((spis_t){}) +#define SPIS_ALL ((spis_t){{ U64_MAX, U64_MAX }}) + +static inline bool spis_is_zero (spis_t s) +{ +return s.v[0 ] ==0 && s.v[1 ] ==0 ; +} + +static inline bool spis_equal (spis_t a,spis_t b) +{ +return a.v[0 ] == b.v[0 ] && a.v[1 ] == b.v[1 ]; +} + +static inline spis_t spis_or (spis_t a,spis_t b) +{ +return (spis_t ){{ a.v[0 ] | b.v[0 ], a.v[1 ] | b.v[1 ] }}; +} + +static inline spis_t spis_and (spis_t a,spis_t b) +{ +return (spis_t ){{ a.v[0 ] & b.v[0 ], a.v[1 ] & b.v[1 ] }}; +} + +static inline spis_t spis_xor (spis_t a,spis_t b) +{ +return (spis_t ){{ a.v[0 ] ^ b.v[0 ], a.v[1 ] ^ b.v[1 ] }}; +} + +static inline spis_t spis_not (spis_t s) +{ +return (spis_t ){{ ~s.v[0 ], ~s.v[1 ] }}; +} + +static inline bool spis_test_bit (spis_t s,u32 slot) +{ +return s.v[slot /64 ] &BIT_ULL (slot %64 ); +} + +static inline void spis_or_range (spis_t *mask,u32 lo,u32 hi) +{ +u32 w; + +for (w = lo; w <= hi && w <STACK_SLOTS ; w++) + mask->v[w /64 ] |=BIT_ULL (w %64 ); +} + +static inline spis_t spis_one_bit (u32 slot) +{ +if (slot <64 ) +return (spis_t ){{BIT (slot),0 }}; +else +return (spis_t ){{0 ,BIT (slot -64 ) }}; +} + +static inline spis_t spis_single_slot (u32 spi) +{ +return spis_or (spis_one_bit (spi *2 ),spis_one_bit (spi *2 +1 )); +} +#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \ (1 << BPF_REG_3) | (1 << BPF_REG_4) | \ (1 << BPF_REG_5))
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v2 03/13] bpf: Add spis_*() helpers for 4-byte stack slot bitmasks
Should BIT() here be BIT_ULL() to match spis_test_bit() and spis_or_range() above?
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0
[ ... ]
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+static inline bool spis_test_bit (spis_t s,u32 slot) +{ +return s.v[slot /64 ] &BIT_ULL (slot %64 ); +} + +static inline void spis_or_range (spis_t *mask,u32 lo,u32 hi) +{ +u32 w; + +for (w = lo; w <= hi && w <STACK_SLOTS ; w++)
include/linux/bpf_verifier.h enum bpf_stack_slot_type { · line 287
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+ mask->v[w /64 ] |=BIT_ULL (w %64 ); +} + +static inline spis_t spis_one_bit (u32 slot) +{ +if (slot <64 ) +return (spis_t ){{BIT (slot),0 }}; +else +return (spis_t ){{0 ,BIT (slot -64 ) }}; +}
Should BIT() here be BIT_ULL() to match spis_test_bit() and spis_or_range() above?
The bitmask is stored in u64 v[2], and BIT() expands to (1UL << (nr)) which is 32-bit wide on 32-bit architectures. For slot values >= 32, this would be undefined behavior.
spis_single_slot() passes values up to 127 through here via spis_one_bit(spi * 2 + 1), so on a 32-bit build both branches would hit shifts >= 32.
Note: this function is removed later in the series by commit 6314c1b64c52 ("bpf: simplify liveness to use (callsite, depth) keyed func_instances"), but it is used in the intermediate commit 3199040a2cb9 ("bpf: make liveness.c track stack with 4-byte granularity") via spis_single_slot().
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 03/13] bpf: Add spis_*() helpers for 4-byte stack slot bitmasks
Quoted 39 lines · On Fri, 2026-04-10 at 01:50 +0000, [email protected] wrote:
Quoted 4 lines
include/linux/bpf_verifier.h +0
-0[ ... ]
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+static inline bool spis_test_bit (spis_t s,u32 slot) +{ +return s.v[slot /64 ] &BIT_ULL (slot %64 ); +} + +static inline void spis_or_range (spis_t *mask,u32 lo,u32 hi) +{ +u32 w; + +for (w = lo; w <= hi && w <STACK_SLOTS ; w++)include/linux/bpf_verifier.h enum bpf_stack_slot_type { · line 287
Quoted 10 lines
include/linux/bpf_verifier.h +10
-0+ mask->v[w /64 ] |=BIT_ULL (w %64 ); +} + +static inline spis_t spis_one_bit (u32 slot) +{ +if (slot <64 ) +return (spis_t ){{BIT (slot),0 }}; +else +return (spis_t ){{0 ,BIT (slot -64 ) }}; +}Should
BIT()here beBIT_ULL()to matchspis_test_bit()andspis_or_range()above?The bitmask is stored in u64 v[2], and
BIT()expands to (1UL << (nr)) which is 32-bit wide on 32-bit architectures. For slot values >= 32, this would be undefined behavior.
spis_single_slot()passes values up to 127 through here viaspis_one_bit(spi * 2 + 1), so on a 32-bit build both branches would hit shifts >= 32.
Quoted 5 lines
Note: this function is removed later in the series by commit
6314c1b64c52("bpf: simplify liveness to use (callsite, depth) keyedfunc_instances"), but it is used in the intermediate commit3199040a2cb9("bpf: makeliveness.ctrack stack with 4-byte granularity") viaspis_single_slot().
Yeap, BIT_ULL is needed
Quoted 6 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerPATCH v2 4/13 bpf: make liveness.c track stack with 4-byte granularity
In reply to Eduard ZingermanConvert liveness bitmask type from u64 to spis_t, doubling the number of trackable stack slots from 64 to 128 to support 4-byte granularity.
Each 8-byte SPI now maps to two consecutive 4-byte sub-slots in the
bitmask: spi*2 half and spi*2+1 half. In verifier.c, check_stack_write_fixed_off() now reports 4-byte aligned writes of 4-byte writes as half-slot marks and 8-byte aligned 8-byte writes as two slots. Similar logic applied in check_stack_read_fixed_off().
Queries (is_live_before) are not yet migrated to half-slot granularity.
--- include/linux/bpf_verifier.h | 4 +- kernel/bpf/liveness.c | 122 ++++++++++++++++++++++++++++--------------- kernel/bpf/verifier.c | 55 +++++++++++-------- 3 files changed, 115 insertions(+), 66 deletions(-)
include/linux/bpf_verifier.h +2 -2
@@ -1218,8 +1218,8 @@ s64 bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env,int bpf_stack_liveness_init (struct bpf_verifier_env *env);void bpf_stack_liveness_free (struct bpf_verifier_env *env);int bpf_update_live_stack (struct bpf_verifier_env *env); -int bpf_mark_stack_read (struct bpf_verifier_env *env,u32 frameno,u32 insn_idx,u64 mask); -void bpf_mark_stack_write (struct bpf_verifier_env *env,u32 frameno,u64 mask); +int bpf_mark_stack_read (struct bpf_verifier_env *env,u32 frameno,u32 insn_idx,spis_t mask); +void bpf_mark_stack_write (struct bpf_verifier_env *env,u32 frameno,spis_t mask);int bpf_reset_stack_write_marks (struct bpf_verifier_env *env,u32 insn_idx);int bpf_commit_stack_write_marks (struct bpf_verifier_env *env);int bpf_live_stack_query_init (struct bpf_verifier_env *env,struct bpf_verifier_state *st);
kernel/bpf/liveness.c +79 -43
234 lines not on this page · raw message
kernel/bpf/verifier.c +34 -21
127 lines not on this page · raw message
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 5/13 bpf: 4-byte precise clean_verifier_state
In reply to Eduard ZingermanMigrate clean_verifier_state() and its liveness queries from 8-byte SPI granularity to 4-byte half-slot granularity.
In __clean_func_state(), each SPI is cleaned in two independent halves:
- half_spi 2*i (lo): slot_type[0..3]
- half_spi 2*i+1 (hi): slot_type[4..7]
Slot types STACK_DYNPTR, STACK_ITER and STACK_IRQ_FLAG are never cleaned, as their slot type markers are required by destroy_if_dynptr_stack_slot(), is_iter_reg_valid_uninit() and is_irq_flag_reg_valid_uninit() for correctness.
When only the hi half is dead, spilled_ptr metadata is destroyed and the lo half's STACK_SPILL bytes are downgraded to STACK_MISC or STACK_ZERO. When only the lo half is dead, spilled_ptr is preserved because the hi half may still need it for state comparison.
--- kernel/bpf/liveness.c | 14 ++++---- kernel/bpf/verifier.c | 94 ++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 85 insertions(+), 23 deletions(-)
kernel/bpf/liveness.c +7 -7
@@ -732,13 +732,12 @@ int bpf_update_live_stack(struct bpf_verifier_env *env)return 0 ; } -static bool is_live_before (struct func_instance *instance,u32 insn_idx,u32 frameno,u32 spi) +static bool is_live_before (struct func_instance *instance,u32 insn_idx,u32 frameno,u32 half_spi) {struct per_frame_masks *masks; masks =get_frame_masks (instance, frameno, insn_idx); -return masks && (spis_test_bit (masks->live_before, spi *2 ) || -spis_test_bit (masks->live_before, spi *2 +1 )); +return masks &&spis_test_bit (masks->live_before, half_spi); }int bpf_live_stack_query_init (struct bpf_verifier_env *env,struct bpf_verifier_state *st)
@@ -759,7 +758,7 @@ int bpf_live_stack_query_init(struct bpf_verifier_env *env, struct bpf_verifier_return 0 ; } -bool bpf_stack_slot_alive (struct bpf_verifier_env *env,u32 frameno,u32 spi) +bool bpf_stack_slot_alive (struct bpf_verifier_env *env,u32 frameno,u32 half_spi) {/* * Slot is alive if it is read before q->st->insn_idx in current func instance,
@@ -773,15 +772,16 @@ bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 spi)bool alive; curframe_instance = q->instances[q->curframe]; -if (is_live_before (curframe_instance, q->insn_idx, frameno, spi)) + alive =is_live_before (curframe_instance, q->insn_idx, frameno, half_spi); +if (alive)return true;for (i = frameno; i < q->curframe; i++) { callsite = curframe_instance->callchain.callsites[i]; instance = q->instances[i]; alive =bpf_calls_callback (env, callsite) - ?is_live_before (instance, callsite, frameno, spi) - :is_live_before (instance, callsite +1 , frameno, spi); + ?is_live_before (instance, callsite, frameno, half_spi) + :is_live_before (instance, callsite +1 , frameno, half_spi);if (alive)return true; }
kernel/bpf/verifier.c +78 -16
141 lines not on this page · raw message
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 6/13 bpf: prepare bpf_liveness api for use by static analysis pass
In reply to Eduard Zingermanbpf: prepare liveness internal API for static analysis pass
Move the updated check and reset from bpf_update_live_stack() into update_instance() itself, so callers outside the main loop can reuse it. Similarly, move write_insn_idx assignment out of reset_stack_write_marks() into its public caller, and thread insn_idx as a parameter to commit_stack_write_marks() instead of reading it from liveness->write_insn_idx. Drop the unused env parameter from alloc_frame_masks() and mark_stack_read().
--- kernel/bpf/liveness.c | 48 +++++++++++++++++++++++------------------------- 1 file changed, 23 insertions(+), 25 deletions(-)
kernel/bpf/liveness.c +23 -25
@@ -259,8 +259,7 @@ static struct per_frame_masks *get_frame_masks(struct func_instance *instance,return &instance->frames[frame][relative_idx (instance, insn_idx)]; } -static struct per_frame_masks *alloc_frame_masks (struct bpf_verifier_env *env, -struct func_instance *instance, +static struct per_frame_masks *alloc_frame_masks (struct func_instance *instance,u32 frame,u32 insn_idx) {struct per_frame_masks *arr;
@@ -298,13 +297,12 @@ static int ensure_cur_instance(struct bpf_verifier_env *env) }/* Accumulate may_read masks for @frame at @insn_idx */ -static int mark_stack_read (struct bpf_verifier_env *env, -struct func_instance *instance,u32 frame,u32 insn_idx,spis_t mask) +static int mark_stack_read (struct func_instance *instance,u32 frame,u32 insn_idx,spis_t mask) {struct per_frame_masks *masks;spis_t new_may_read; - masks =alloc_frame_masks (env, instance, frame, insn_idx); + masks =alloc_frame_masks (instance, frame, insn_idx);if (IS_ERR (masks))return PTR_ERR (masks); new_may_read =spis_or (masks->may_read, mask);
@@ -321,17 +319,15 @@ int bpf_mark_stack_read(struct bpf_verifier_env *env, u32 frame, u32 insn_idx, sint err; err =ensure_cur_instance (env); - err = err ?:mark_stack_read (env, env->liveness->cur_instance, frame, insn_idx, mask); + err = err ?:mark_stack_read (env->liveness->cur_instance, frame, insn_idx, mask);return err; } -static void reset_stack_write_marks (struct bpf_verifier_env *env, -struct func_instance *instance,u32 insn_idx) +static void reset_stack_write_marks (struct bpf_verifier_env *env,struct func_instance *instance) {struct bpf_liveness *liveness = env->liveness;int i; - liveness->write_insn_idx = insn_idx;for (i =0 ; i <= instance->callchain.curframe; i++) liveness->write_masks_acc[i] =SPIS_ZERO ; }
@@ -345,7 +341,8 @@ int bpf_reset_stack_write_marks(struct bpf_verifier_env *env, u32 insn_idx)if (err)return err; -reset_stack_write_marks (env, liveness->cur_instance, insn_idx); + liveness->write_insn_idx = insn_idx; +reset_stack_write_marks (env, liveness->cur_instance);return 0 ; }
@@ -355,7 +352,8 @@ void bpf_mark_stack_write(struct bpf_verifier_env *env, u32 frame, spis_t mask) }static int commit_stack_write_marks (struct bpf_verifier_env *env, -struct func_instance *instance) +struct func_instance *instance, +u32 insn_idx) {struct bpf_liveness *liveness = env->liveness;u32 idx, frame, curframe;
@@ -366,13 +364,13 @@ static int commit_stack_write_marks(struct bpf_verifier_env *env,return 0 ; curframe = instance->callchain.curframe; - idx =relative_idx (instance, liveness->write_insn_idx); + idx =relative_idx (instance, insn_idx);for (frame =0 ; frame <= curframe; frame++) { mask = liveness->write_masks_acc[frame];/* avoid allocating frames for zero masks */ if (spis_is_zero (mask) && !instance->must_write_set[idx])continue ; - masks =alloc_frame_masks (env, instance, frame, liveness->write_insn_idx); + masks =alloc_frame_masks (instance, frame, insn_idx);if (IS_ERR (masks))return PTR_ERR (masks); old_must_write = masks->must_write;
@@ -402,7 +400,7 @@ static int commit_stack_write_marks(struct bpf_verifier_env *env,*/ int bpf_commit_stack_write_marks (struct bpf_verifier_env *env) { -return commit_stack_write_marks (env, env->liveness->cur_instance); +return commit_stack_write_marks (env, env->liveness->cur_instance, env->liveness->write_insn_idx); }static char *fmt_callchain (struct bpf_verifier_env *env,struct callchain *callchain)
@@ -576,18 +574,17 @@ static int propagate_to_outer_instance(struct bpf_verifier_env *env,if (IS_ERR (outer_instance))return PTR_ERR (outer_instance); callsite = callchain->callsites[callchain->curframe -1 ]; - -reset_stack_write_marks (env, outer_instance, callsite); +reset_stack_write_marks (env, outer_instance);for (frame =0 ; frame < callchain->curframe; frame++) { insn =get_frame_masks (instance, frame, this_subprog_start);if (!insn)continue ;bpf_mark_stack_write (env, frame, insn->must_write_acc); - err =mark_stack_read (env, outer_instance, frame, callsite, insn->live_before); + err =mark_stack_read (outer_instance, frame, callsite, insn->live_before);if (err)return err; } -commit_stack_write_marks (env, outer_instance); +commit_stack_write_marks (env, outer_instance, callsite);return 0 ; }
@@ -654,6 +651,9 @@ static int update_instance(struct bpf_verifier_env *env, struct func_instance *ibool changed;int err; +if (!instance->updated) +return 0 ; + this_subprog_start =callchain_subprog_start (callchain);/* * If must_write marks were updated must_write_acc needs to be reset
@@ -699,6 +699,8 @@ static int update_instance(struct bpf_verifier_env *env, struct func_instance *ireturn err; } + instance->updated = false; + instance->must_write_dropped = false;return 0 ; }
@@ -721,13 +723,9 @@ int bpf_update_live_stack(struct bpf_verifier_env *env)if (IS_ERR (instance))return PTR_ERR (instance); -if (instance->updated) { - err =update_instance (env, instance); -if (err) -return err; - instance->updated = false; - instance->must_write_dropped = false; - } + err =update_instance (env, instance); +if (err) +return err; }return 0 ; }
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 7/13 bpf: introduce forward arg-tracking dataflow analysis
In reply to Eduard ZingermanThe analysis is a basis for static liveness tracking mechanism introduced by the next two commits.
A forward fixed-point analysis that tracks which frame's FP each register value is derived from, and at what byte offset. This is needed because a callee can receive a pointer to its caller's stack frame (e.g. r1 = fp-16 at the call site), then do *(u64 *)(r1 + 0) inside the callee — a cross-frame stack access that the callee's local liveness must attribute to the caller's stack.
Each register holds an arg_track value from a three-level lattice:
- Precise {frame=N, off=[o1,o2,...]} — known frame index and up to 4 concrete byte offsets
- Offset-imprecise {frame=N, off_cnt=0} — known frame, unknown offset
- Fully-imprecise {frame=ARG_IMPRECISE, mask=bitmask} — unknown frame, mask says which frames might be involved
At CFG merge points the lattice moves toward imprecision (same frame+offset stays precise, same frame different offsets merges offset sets or becomes offset-imprecise, different frames become fully-imprecise with OR'd bitmask).
The analysis also tracks spills/fills to the callee's own stack (at_stack_in/out), so FP derived values spilled and reloaded.
This pass is run recursively per call site: when subprog A calls B with specific FP-derived arguments, B is re-analyzed with those entry args. The recursion follows analyze_subprog -> compute_subprog_args -> (for each call insn) -> analyze_subprog. Subprogs that receive no FP-derived args are skipped during recursion and analyzed independently at depth 0.
--- include/linux/bpf_verifier.h | 4 + kernel/bpf/liveness.c | 1293 ++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 16 +- 3 files changed, 1311 insertions(+), 2 deletions(-)
include/linux/bpf_verifier.h +4 -0
@@ -226,6 +226,7 @@ enum bpf_stack_slot_type {/* 4-byte stack slot granularity for liveness analysis */ #define BPF_HALF_REG_SIZE 4 +#define STACK_SLOT_SZ 4#define STACK_SLOTS (MAX_BPF_STACK / BPF_HALF_REG_SIZE)/* 128 */ typedef struct {
@@ -887,6 +888,8 @@ struct bpf_verifier_env { } cfg;struct backtrack_state bt;struct bpf_jmp_history_entry *cur_hist_ent; +/* Per-callsite copy of parent's converged at_stack_in for cross-frame fills. */ +struct arg_track **callsite_at_stack;u32 pass_cnt;/* number of times do_check() was called */ u32 subprog_cnt;/* number of instructions analyzed by the verifier */
@@ -1214,6 +1217,7 @@ s64 bpf_helper_stack_access_bytes(struct bpf_verifier_env *env,s64 bpf_kfunc_stack_access_bytes (struct bpf_verifier_env *env,struct bpf_insn *insn,int arg,int insn_idx); +int bpf_compute_subprog_arg_access (struct bpf_verifier_env *env);int bpf_stack_liveness_init (struct bpf_verifier_env *env);void bpf_stack_liveness_free (struct bpf_verifier_env *env);
kernel/bpf/liveness.c +1293 -0
1,323 lines not on this page · raw message
kernel/bpf/verifier.c +14 -2
@@ -20256,6 +20256,15 @@ static int clean_live_states(struct bpf_verifier_env *env, int insn,struct list_head *pos, *head;int err; +/* keep cleaning the current state as registers/stack become dead */ + err =clean_verifier_state (env, cur); +if (err) +return err; + +/* +* can simply return here, since cached states will also be clean, +* but keep old logic for the sake of dynamic liveness. +*/ head =explored_state (env, insn);list_for_each (pos, head) { sl = container_of(pos, struct bpf_verifier_state_list, node);
@@ -20267,8 +20276,6 @@ static int clean_live_states(struct bpf_verifier_env *env, int insn,if (sl->state.cleaned)/* all regs in this state in all frames were already marked */ continue ; -if (incomplete_read_marks (env, &sl->state)) -continue ; err =clean_verifier_state (env, &sl->state);if (err)return err;
@@ -26301,6 +26308,11 @@ static int compute_live_registers(struct bpf_verifier_env *env)for (i =0 ; i < insn_cnt; ++i)compute_insn_live_regs (env, &insns[i], &state[i]); +/* Forward pass: resolve stack access through FP-derived pointers */ + err =bpf_compute_subprog_arg_access (env); +if (err) + goto out; + changed = true;while (changed) { changed = false;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v2 07/13] bpf: introduce forward arg-tracking dataflow analysis
With 4-byte slot granularity (128 slots), spis_t is a 128-bit type (u64 v[2]) and SPIS_ALL is {{ U64_MAX, U64_MAX }}. Should this comment say SPIS_ALL rather than U64_MAX?
Quoted 4 lines
kernel/bpf/liveness.c +0
-0
[ ... ]
Quoted 3 lines
+ * At memory access sites (LDX/STX/ST), offset-imprecise marks only
+ * the known frame's access mask asU64_MAX, while fully-imprecise
+ * iterates bits in the bitmask and routes each frame to its target.
With 4-byte slot granularity (128 slots), spis_t is a 128-bit type (u64 v[2]) and SPIS_ALL is {{ U64_MAX, U64_MAX }}. Should this comment say SPIS_ALL rather than U64_MAX?
Quoted 1 line
+ */
[ ... ]
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 989
Quoted 7 lines
kernel/bpf/liveness.c +7
-0+/* +* Collapse to ARG_IMPRECISE with a bitmask of which arg +* identities are involved. This lets memory access sites +* mark only the relevant args' access masks and only set +* stack_use when AT_CURRENT (bit 0) is in the bitmask. +*/ +static struct arg_track arg_join_imprecise (struct arg_track a,struct arg_track b)
Two references here that look stale. AT_CURRENT is not defined anywhere in the tree, and the current frame is BIT(depth), not necessarily bit 0. Similarly, "stack_use" doesn't appear as a variable or field in this file -- the code uses may_read.
Were these left over from an earlier iteration of the design?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerPATCH v2 8/13 bpf: simplify liveness to use (callsite, depth) keyed func_instances
In reply to Eduard ZingermanRework func_instance identification and remove the dynamic liveness API, completing the transition to fully static stack liveness analysis.
Replace callchain-based func_instance keys with (callsite, depth) pairs. The full callchain (all ancestor callsites) is no longer part of the hash key; only the immediate callsite and the call depth matter. This does not lose precision in practice and simplifies the data structure significantly: struct callchain is removed entirely, func_instance stores just callsite, depth, and a callsites[] array for outer frame lookups.
Drop must_write_acc propagation. Previously, must_write marks were accumulated across successors and propagated to the caller via propagate_to_outer_instance(). Instead, callee entry liveness (live_before at subprog start) is pulled directly back to the caller's callsite in analyze_subprog() after each callee returns.
Skip recursive descent into callees that receive no FP-derived arguments (has_fp_args() check). This is needed because global subprogram calls can push depth beyond MAX_CALL_FRAMES (max depth is 64 for global calls but only 8 frames are accommodated for FP passing). It also handles the case where a callback subprog cannot be determined by argument tracking: such callbacks will be processed by analyze_subprog() at depth 0 independently.
Update lookup_instance() (used by is_live_before queries) to search for the func_instance with maximal depth at the corresponding callsite, walking depth downward from frameno to 0. This accounts for the fact that instance depth no longer corresponds 1:1 to bpf_verifier_state->curframe, since skipped non-FP calls create gaps.
Remove the dynamic public liveness API from verifier.c:
- bpf_mark_stack_{read,write}(), bpf_reset/commit_stack_write_marks()
- bpf_update_live_stack(), bpf_reset_live_stack_callchain()
- All call sites in check_stack_{read,write}_fixed_off(), check_stack_range_initialized(), mark_stack_slot_obj_read(), mark/unmark_stack_slots_{dynptr,iter,irq_flag}()
- The per-instruction write mark accumulation in do_check()
- The bpf_update_live_stack() call in prepare_func_exit()
mark_stack_read() and mark_stack_write() become static functions in liveness.c, called only from the static analysis pass. The func_instance->updated and must_write_dropped flags are removed. Remove spis_single_slot(), spis_msb_half_slot(), spis_lsb_half_slot() helpers from bpf_verifier.h as they are no longer used.
--- include/linux/bpf_verifier.h | 19 -- kernel/bpf/liveness.c | 558 ++++++++++++------------------------------- kernel/bpf/verifier.c | 79 +----- 3 files changed, 159 insertions(+), 497 deletions(-)
include/linux/bpf_verifier.h +0 -19
@@ -279,19 +279,6 @@ static inline void spis_or_range(spis_t *mask, u32 lo, u32 hi) mask->v[w /64 ] |=BIT_ULL (w %64 ); } -static inline spis_t spis_one_bit (u32 slot) -{ -if (slot <64 ) -return (spis_t ){{BIT (slot),0 }}; -else -return (spis_t ){{0 ,BIT (slot -64 ) }}; -} - -static inline spis_t spis_single_slot (u32 spi) -{ -return spis_or (spis_one_bit (spi *2 ),spis_one_bit (spi *2 +1 )); -} -#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \ (1 << BPF_REG_3) | (1 << BPF_REG_4) | \ (1 << BPF_REG_5))
@@ -1221,13 +1208,7 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env);int bpf_stack_liveness_init (struct bpf_verifier_env *env);void bpf_stack_liveness_free (struct bpf_verifier_env *env); -int bpf_update_live_stack (struct bpf_verifier_env *env); -int bpf_mark_stack_read (struct bpf_verifier_env *env,u32 frameno,u32 insn_idx,spis_t mask); -void bpf_mark_stack_write (struct bpf_verifier_env *env,u32 frameno,spis_t mask); -int bpf_reset_stack_write_marks (struct bpf_verifier_env *env,u32 insn_idx); -int bpf_commit_stack_write_marks (struct bpf_verifier_env *env);int bpf_live_stack_query_init (struct bpf_verifier_env *env,struct bpf_verifier_state *st);bool bpf_stack_slot_alive (struct bpf_verifier_env *env,u32 frameno,u32 spi); -void bpf_reset_live_stack_callchain (struct bpf_verifier_env *env);#endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/liveness.c +155 -403
850 lines not on this page · raw message
kernel/bpf/verifier.c +4 -75
184 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v2 08/13] bpf: simplify liveness to use (callsite, depth) keyed func_instances
The struct func_instance does not have a callsites[] array. The callsites[] array lives in struct live_stack_query, not func_instance. Should this say "live_stack_query stores a callsites[] array…
Quoted 2 lines
func_instancestores just callsite, depth, and a callsites[] array for outer frame lookups.
The struct func_instance does not have a callsites[] array. The callsites[] array lives in struct live_stack_query, not func_instance. Should this say "live_stack_query stores a callsites[] array for outer frame lookups" instead?
Quoted 2 lines
Remove
spis_single_slot(),spis_msb_half_slot(),spis_lsb_half_slot()helpers frombpf_verifier.has they are no longer used.
The diff removes spis_one_bit() and spis_single_slot(). There are no spis_msb_half_slot() or spis_lsb_half_slot() functions in the tree (checked back through the entire series). Should this say "spis_one_bit() and spis_single_slot()"?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerPATCH v2 9/13 bpf: change logging scheme for live stack analysis
In reply to Eduard ZingermanInstead of breadcrumps like:
(d2,cs15) frame 0 insn 18 +live -16
(d2,cs15) frame 0 insn 17 +live -16
Print final accumulated stack use/def data per-func_instance per-instruction. printed func_instance's are ordered by callsite and depth. For example:
stack use/def subprog#0 shared_instance_must_write_overwrite (d0,cs0):
0: (b7) r1 = 1
1: (7b) *(u64 *)(r10 -8) = r1 ; def: fp0-8
2: (7b) *(u64 *)(r10 -16) = r1 ; def: fp0-16
3: (bf) r1 = r10
4: (07) r1 += -8
5: (bf) r2 = r10
6: (07) r2 += -16
7: (85) call pc+7 ; use: fp0-8 fp0-16
8: (bf) r1 = r10
9: (07) r1 += -16 10: (bf) r2 = r10
11: (07) r2 += -8
12: (85) call pc+2 ; use: fp0-8 fp0-16
13: (b7) r0 = 0
14: (95) exit
stack use/def subprog#1 forwarding_rw (d1,cs7):
15: (85) call pc+1 ; use: fp0-8 fp0-16
16: (95) exit
stack use/def subprog#1 forwarding_rw (d1,cs12):
15: (85) call pc+1 ; use: fp0-8 fp0-16
16: (95) exit
stack use/def subprog#2 write_first_read_second (d2,cs15):
17: (7a) *(u64 *)(r1 +0) = 42
18: (79) r0 = *(u64 *)(r2 +0) ; use: fp0-8 fp0-16
19: (95) exit
For groups of three or more consecutive stack slots, abbreviate as follows:
25: (85) call bpf_loop#181 ; use: fp2-8..-512 fp1-8..-512 fp0-8..-512
--- include/linux/bpf_verifier.h | 5 - kernel/bpf/liveness.c | 238 +++++++++++++++++++++++++++++-------------- 2 files changed, 162 insertions(+), 81 deletions(-)
include/linux/bpf_verifier.h +0 -5
@@ -256,11 +256,6 @@ static inline spis_t spis_and(spis_t a, spis_t b)return (spis_t ){{ a.v[0 ] & b.v[0 ], a.v[1 ] & b.v[1 ] }}; } -static inline spis_t spis_xor (spis_t a,spis_t b) -{ -return (spis_t ){{ a.v[0 ] ^ b.v[0 ], a.v[1 ] ^ b.v[1 ] }}; -} -static inline spis_t spis_not (spis_t s) {return (spis_t ){{ ~s.v[0 ], ~s.v[1 ] }};
kernel/bpf/liveness.c +162 -76
286 lines not on this page · raw message
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v2 09/13] bpf: change logging scheme for live stack analysis
Should pos and insn_pos be u64 here? They store env->log.end_pos which is u64, and are later passed to bpf_vlog_reset() which takes a u64 new_pos parameter:
Quoted 4 lines
kernel/bpf/liveness.c +0
-0
[ ... ]
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 547
Quoted 6 lines
kernel/bpf/liveness.c +6
-0+static void print_instance (struct bpf_verifier_env *env,struct func_instance *instance) +{ +int start = env->subprog_info[instance->subprog].start; +struct bpf_insn *insns = env->prog->insnsi; +struct per_frame_masks *masks; +int insn_idx, frame, pos, insn_pos, i;
Should pos and insn_pos be u64 here? They store env->log.end_pos which is u64, and are later passed to bpf_vlog_reset() which takes a u64 new_pos parameter:
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 560
Quoted 13 lines
kernel/bpf/liveness.c +13
-0+int len = instance->insn_cnt; +bool has_use, has_def; + +if (!(env->log.level &BPF_LOG_LEVEL2 )) +return ; + +verbose (env,"stack use/def %s " ,fmt_subprog (env, instance->subprog)); +verbose (env,"%s:\n" ,fmt_instance (env, instance)); +for (i =0 ; i < len; i++) { + insn_idx = start + i; + has_use = false; + has_def = false; + pos = env->log.end_pos;
Here end_pos (u64) is truncated to int.
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 564
Quoted 4 lines
kernel/bpf/liveness.c +4
-0+verbose (env,"%3d: " , insn_idx); +bpf_verbose_insn (env, &insns[insn_idx]); +bpf_vlog_reset (&env->log, env->log.end_pos -1 );/* remove \n */ + insn_pos = env->log.end_pos;
Same truncation for insn_pos.
Quoted 3 lines
kernel/bpf/liveness.c +3
-0+verbose (env,"%*c;" ,bpf_vlog_alignment (insn_pos - pos),' ' ); + pos = env->log.end_pos; +verbose (env," use: " );
[ ... ]
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 576
Quoted 2 lines
kernel/bpf/liveness.c +2
-0+if (!has_use) +bpf_vlog_reset (&env->log, pos);
When end_pos has exceeded INT_MAX, the truncated pos is promoted back to u64 via sign extension, producing an incorrect position. For example, if end_pos was 0x100000005 at the time pos was saved, pos holds 5, and bpf_vlog_reset() receives 5 instead of 0x100000005, rewinding the log to the wrong position.
With BPF_LOG_LEVEL2 the verifier can produce large volumes of output (per-state info at every prune point), so end_pos can realistically grow past INT_MAX for complex programs. The existing convention in bpf_verifier_env uses u64 for prev_log_pos, and print_subprog_arg_access() in this same file avoids storing end_pos in a local variable, passing end_pos - 1 directly to bpf_vlog_reset().
Quoted 2 lines
kernel/bpf/liveness.c +2
-0+ pos = env->log.end_pos; +verbose (env," def: " );
[ ... ]
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 587
Quoted 2 lines
kernel/bpf/liveness.c +2
-0+if (!has_def) +bpf_vlog_reset (&env->log, has_use ? pos : insn_pos);
Same issue here with both pos and insn_pos.
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 592
Quoted 5 lines
kernel/bpf/liveness.c +5
-0+verbose (env,"\n" ); +if (bpf_is_ldimm64 (&insns[insn_idx])) + i++; + } +}
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 09/13] bpf: change logging scheme for live stack analysis
Quoted 7 lines · On Fri, 2026-04-10 at 02:02 +0000, [email protected] wrote:
Quoted 4 lines
kernel/bpf/liveness.c +0
-0[ ... ]
Quoted 10 lines
kernel/bpf/liveness.c bool bpf_stack_slot_alive(struct bpf_verifier_env *env, u32 frameno, u32 half_sp · line 547
Quoted 6 lines
kernel/bpf/liveness.c +6
-0+static void print_instance (struct bpf_verifier_env *env,struct func_instance *instance) +{ +int start = env->subprog_info[instance->subprog].start; +struct bpf_insn *insns = env->prog->insnsi; +struct per_frame_masks *masks; +int insn_idx, frame, pos, insn_pos, i;Should pos and
insn_posbe u64 here? They storeenv->log.end_poswhich is u64, and are later passed tobpf_vlog_reset()which takes a u64new_posparameter:
Yeap, u64 is needed.
[...]
Eduard Zingerman
maintainerPATCH v2 10/13 selftests/bpf: update existing tests due to liveness changes
In reply to Eduard ZingermanFrom: Alexei Starovoitov <[email protected]>
The verifier cleans all dead registers and stack slots in the current state. Adjust expected output in tests or insert dummy stack/register reads. Also update verifier_live_stack tests to adhere to new logging scheme.
--- .../selftests/bpf/progs/exceptions_assert.c | 6 +- tools/testing/selftests/bpf/progs/uninit_stack.c | 1 + tools/testing/selftests/bpf/progs/verifier_align.c | 10 +-- .../selftests/bpf/progs/verifier_div_mod_bounds.c | 18 +++-- .../selftests/bpf/progs/verifier_live_stack.c | 88 +++++++++------------- .../selftests/bpf/progs/verifier_scalar_ids.c | 26 ++++--- .../selftests/bpf/progs/verifier_spill_fill.c | 8 +- .../bpf/progs/verifier_subprog_precision.c | 8 +- 8 files changed, 79 insertions(+), 86 deletions(-)
tools/testing/selftests/bpf/progs/exceptions_assert.c +3 -3
@@ -59,7 +59,7 @@ check_assert(s64, >=, ge_neg, INT_MIN); SEC("?tc") __log_level(2) __failure -__msg(": R0=0 R1=ctx() R2=scalar(smin=0xffffffff80000002,smax=smax32=0x7ffffffd,smin32=0x80000002) R10=fp0") +__msg(": R1=ctx() R2=scalar(smin=0xffffffff80000002,smax=smax32=0x7ffffffd,smin32=0x80000002) R10=fp0")int check_assert_range_s64 (struct __sk_buff *ctx) {struct bpf_sock *sk = ctx->sk;
@@ -86,7 +86,7 @@ int check_assert_range_u64(struct __sk_buff *ctx) SEC("?tc") __log_level(2) __failure -__msg(": R0=0 R1=ctx() R2=4096 R10=fp0") +__msg(": R1=ctx() R2=4096 R10=fp0")int check_assert_single_range_s64 (struct __sk_buff *ctx) {struct bpf_sock *sk = ctx->sk;
@@ -114,7 +114,7 @@ int check_assert_single_range_u64(struct __sk_buff *ctx) SEC("?tc") __log_level(2) __failure -__msg(": R1=pkt(r=64,imm=64) R2=pkt_end() R6=pkt(r=64) R10=fp0") +__msg(": R6=pkt(r=64) R10=fp0")int check_assert_generic (struct __sk_buff *ctx) {u8 *data_end = (void *)(long )ctx->data_end;
tools/testing/selftests/bpf/progs/uninit_stack.c +1 -0
@@ -76,6 +76,7 @@ __naked int helper_uninit_to_misc(void *ctx)* thus showing the stack state, matched by __msg(). \ */ \ call %[dummy]; \ + r1 = *(u64 *)(r10 -104 ); \ r0 =0 ; \ exit; \ "
tools/testing/selftests/bpf/progs/verifier_align.c +5 -5
@@ -131,7 +131,7 @@ LBL ":" \ SEC("tc") __success __log_level(2) __flag(BPF_F_ANY_ALIGNMENT) -__msg("6: R0=pkt(r=8,imm=8)") +__msg("6: {{.*}} R2=pkt(r=8)") __msg("6: {{.*}} R3={{[^)]*}}var_off=(0x0; 0xff)") __msg("7: {{.*}} R3={{[^)]*}}var_off=(0x0; 0x1fe)")__msg ("8: {{.*}} R3={{[^)]*}}var_off=(0x0; 0x3fc)" )
@@ -205,7 +205,7 @@ __success __log_level(2) __msg("2: {{.*}} R5=pkt(r=0)") __msg("4: {{.*}} R5=pkt(r=0,imm=14)") __msg("5: {{.*}} R4=pkt(r=0,imm=14)") -__msg("9: {{.*}} R2=pkt(r=18)") +__msg("9: {{.*}} R5=pkt(r=18,imm=14)") __msg("10: {{.*}} R4={{[^)]*}}var_off=(0x0; 0xff){{.*}} R5=pkt(r=18,imm=14)") __msg("13: {{.*}} R4={{[^)]*}}var_off=(0x0; 0xffff)")__msg ("14: {{.*}} R4={{[^)]*}}var_off=(0x0; 0xffff)" )
@@ -254,7 +254,7 @@ __msg("11: {{.*}} R5=pkt(id=1,{{[^)]*}},var_off=(0x2; 0x7fc)")* offset is considered using reg->aux_off_align which * is 4 and meets the load's requirements. */ -__msg ("15: {{.*}} R4={{[^)]*}}var_off=(0x2; 0x7fc){{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)" ) +__msg ("15: {{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)" )/* Variable offset is added to R5 packet pointer, * resulting in auxiliary alignment of 4. To avoid BPF * verifier's precision backtracking logging
@@ -273,7 +273,7 @@ __msg("19: {{.*}} R5=pkt(id=2,{{[^)]*}}var_off=(0x2; 0x7fc)")* aligned, so the total offset is 4-byte aligned and * meets the load's requirements. */ -__msg ("24: {{.*}} R4={{[^)]*}}var_off=(0x2; 0x7fc){{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)" ) +__msg ("24: {{.*}} R5={{[^)]*}}var_off=(0x2; 0x7fc)" )/* Constant offset is added to R5 packet pointer, * resulting in reg->off value of 14. */
@@ -296,7 +296,7 @@ __msg("31: {{.*}} R4={{[^)]*}}var_off=(0x2; 0xffc){{.*}} R5={{[^)]*}}var_off=(0x* the total offset is 4-byte aligned and meets the * load's requirements. */ -__msg("35: {{.*}} R4={{[^)]*}}var_off=(0x2; 0xffc){{.*}} R5={{[^)]*}}var_off=(0x2; 0xffc)") +__msg("35: {{.*}} R5={{[^)]*}}var_off=(0x2; 0xffc)")__naked void packet_variable_offset (void ) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_div_mod_bounds.c +10 -8
@@ -36,7 +36,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("UDIV32, zero divisor") __success __retval(0) __log_level(2) -__msg("w1 /= w2 {{.*}}; R1=0 R2=0") +__msg("w1 /= w2 {{.*}}; R1=0")__naked void udiv32_zero_divisor (void ) { asm volatile (" \
@@ -81,7 +81,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("UDIV64, zero divisor") __success __retval(0) __log_level(2) -__msg("r1 /= r2 {{.*}}; R1=0 R2=0") +__msg("r1 /= r2 {{.*}}; R1=0")__naked void udiv64_zero_divisor (void ) { asm volatile (" \
@@ -242,7 +242,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("SDIV32, zero divisor") __success __retval(0) __log_level(2) -__msg("w1 s/= w2 {{.*}}; R1=0 R2=0") +__msg("w1 s/= w2 {{.*}}; R1=0")__naked void sdiv32_zero_divisor (void ) { asm volatile (" \
@@ -275,6 +275,7 @@ __naked void sdiv32_overflow_1(void) w2 +=10 ; \ if w1 s> w2 goto l0_%=; \ w1 s/= -1; \ + r2 = r1; \ l0_%=: r0 = 0; \ exit; \ " :
@@ -443,7 +444,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("SDIV64, zero divisor") __success __retval(0) __log_level(2) -__msg("r1 s/= r2 {{.*}}; R1=0 R2=0") +__msg("r1 s/= r2 {{.*}}; R1=0")__naked void sdiv64_zero_divisor (void ) { asm volatile (" \
@@ -476,6 +477,7 @@ __naked void sdiv64_overflow_1(void) r2 +=10 ; \ if r1 s> r2 goto l0_%=; \ r1 s/= -1; \ + r2 = r1; \ l0_%=: r0 = 0; \ exit; \ " :
@@ -553,7 +555,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("UMOD32, zero divisor") __success __retval(0) __log_level(2) -__msg("w1 %= w2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8)) R2=0") +__msg("w1 %= w2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8))")__naked void umod32_zero_divisor (void ) { asm volatile (" \
@@ -624,7 +626,7 @@ l0_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("UMOD64, zero divisor") __success __retval(0) __log_level(2) -__msg("r1 %= r2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8)) R2=0") +__msg("r1 %= r2 {{.*}}; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=9,var_off=(0x1; 0x8))")__naked void umod64_zero_divisor (void ) { asm volatile (" \
@@ -833,7 +835,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("SMOD32, zero divisor") __success __retval(0) __log_level(2) -__msg("w1 s%= w2 {{.*}}; R1=scalar(smin=0,smax=umax=0xffffffff,smin32=-8,smax32=10,var_off=(0x0; 0xffffffff)) R2=0") +__msg("w1 s%= w2 {{.*}}; R1=scalar(smin=0,smax=umax=0xffffffff,smin32=-8,smax32=10,var_off=(0x0; 0xffffffff))")__naked void smod32_zero_divisor (void ) { asm volatile (" \
@@ -1084,7 +1086,7 @@ l1_%=: r0 = *(u64 *)(r1 + 0); \ SEC("socket") __description("SMOD64, zero divisor") __success __retval(0) __log_level(2) -__msg("r1 s%= r2 {{.*}}; R1=scalar(smin=smin32=-8,smax=smax32=10) R2=0") +__msg("r1 s%= r2 {{.*}}; R1=scalar(smin=smin32=-8,smax=smax32=10)")__naked void smod64_zero_divisor (void ) { asm volatile (" \
tools/testing/selftests/bpf/progs/verifier_live_stack.c +35 -53
139 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +17 -9
@@ -264,13 +264,13 @@ void precision_many_frames__bar(void)*/ SEC("socket") __success __log_level(2) -__msg("11: (0f) r2 += r1") +__msg("12: (0f) r2 += r1")/* foo frame */ -__msg("frame1: regs=r1 stack= before 10: (bf) r2 = r10") -__msg("frame1: regs=r1 stack= before 9: (25) if r1 > 0x7 goto pc+0") -__msg("frame1: regs=r1 stack=-8,-16 before 8: (7b) *(u64 *)(r10 -16) = r1") -__msg("frame1: regs=r1 stack=-8 before 7: (7b) *(u64 *)(r10 -8) = r1") -__msg ("frame1: regs=r1 stack= before 4: (85) call pc+2" ) +__msg("frame1: regs=r1 stack= before 11: (bf) r2 = r10") +__msg("frame1: regs=r1 stack= before 10: (25) if r1 > 0x7 goto pc+0") +__msg("frame1: regs=r1 stack=-8,-16 before 9: (7b) *(u64 *)(r10 -16) = r1") +__msg("frame1: regs=r1 stack=-8 before 8: (7b) *(u64 *)(r10 -8) = r1") +__msg ("frame1: regs=r1 stack= before 4: (85) call pc+3" ) /* main frame */ __msg("frame0: regs=r1 stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r1")__msg ("frame0: regs=r1 stack= before 2: (bf) r1 = r0" )
@@ -286,6 +286,7 @@ __naked void precision_stack(void)"r1 = r0;" "*(u64*)(r10 - 8) = r1;" "call precision_stack__foo;" +"r0 = *(u64*)(r10 - 8);" "r0 = 0;" "exit;" :
@@ -309,6 +310,8 @@ void precision_stack__foo(void)*/ "r2 = r10;" "r2 += r1;" +"r0 = *(u64*)(r10 - 8);" +"r0 = *(u64*)(r10 - 16);" "exit" ::: __clobber_all); }
@@ -802,9 +805,9 @@ __success __log_level(2)/* The exit instruction should be reachable from two states, * use two matches and "processed .. insns" to ensure this. */ -__msg("15: (95) exit") -__msg("15: (95) exit") -__msg("processed 20 insns") +__msg("16: (95) exit") +__msg("16: (95) exit") +__msg("processed 22 insns")__flag (BPF_F_TEST_STATE_FREQ) __naked void two_old_ids_one_cur_id(void) {
@@ -835,6 +838,11 @@ __naked void two_old_ids_one_cur_id(void)"r2 = r10;" "r2 += r6;" "r2 += r7;" +/* +* keep r8 and r9 live, otherwise r6->id and r7->id +* will become singular and reset to zero before if r6 > r7 +*/ +"r9 += r8;" "exit;" : : __imm(bpf_ktime_get_ns)
tools/testing/selftests/bpf/progs/verifier_spill_fill.c +4 -4
@@ -650,7 +650,7 @@ __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1") __msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")__msg ("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6" ) /* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */ -__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1") +__msg("mark_precise: frame0: parent state regs= stack=-8: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")__msg ("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0" )
@@ -668,7 +668,7 @@ __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2") __msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")__msg ("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6" ) /* now both fp-8 and fp-16 are precise, very good */ -__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1") +__msg("mark_precise: frame0: parent state regs= stack=-16: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")__msg ("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0" )
@@ -726,7 +726,7 @@ __msg("9: (0f) r1 += r2") __msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1") __msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)") __msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6") -__msg("mark_precise: frame0: parent state regs= stack=-8: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1") +__msg("mark_precise: frame0: parent state regs= stack=-8: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")__msg ("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0" )
@@ -743,7 +743,7 @@ __msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2 __msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2") __msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)") __msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6") -__msg("mark_precise: frame0: parent state regs= stack=-16: R0=1 R1=ctx() R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1") +__msg("mark_precise: frame0: parent state regs= stack=-16: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1") __msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")__msg ("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0" )
tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +4 -4
@@ -282,7 +282,7 @@ __msg("mark_precise: frame0: regs=r0,r6 stack= before 10: (bf) r6 = r0")__msg ("mark_precise: frame0: regs=r0 stack= before 9: (85) call bpf_loop" ) /* State entering callback body popped from states stack */ __msg("from 9 to 17: frame1:") -__msg("17: frame1: R1=scalar() R2=0 R10=fp0 cb") +__msg("17: frame1: R10=fp0 cb") __msg("17: (b7) r0 = 0") __msg("18: (95) exit")__msg ("returning from callee:" )
@@ -411,7 +411,7 @@ __msg("mark_precise: frame0: regs=r6 stack= before 5: (b7) r1 = 1")__msg ("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3" ) /* State entering callback body popped from states stack */ __msg("from 9 to 15: frame1:") -__msg("15: frame1: R1=scalar() R2=0 R10=fp0 cb") +__msg("15: frame1: R10=fp0 cb") __msg("15: (b7) r0 = 0") __msg("16: (95) exit")__msg ("returning from callee:" )
@@ -567,7 +567,7 @@ __msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -8) = r6__msg ("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3" ) /* State entering callback body popped from states stack */ __msg("from 10 to 17: frame1:") -__msg("17: frame1: R1=scalar() R2=0 R10=fp0 cb") +__msg("17: frame1: R10=fp0 cb") __msg("17: (b7) r0 = 0") __msg("18: (95) exit")__msg ("returning from callee:" )
@@ -681,7 +681,7 @@ __msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1") __msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8") __msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4") __msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)") -__msg("mark_precise: frame0: parent state regs= stack=-8: R0=2 R6=1 R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1") +__msg("mark_precise: frame0: parent state regs= stack=-8: R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1") __msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7") __msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")__msg ("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2" )
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 11/13 selftests/bpf: adjust verifier_log buffers
In reply to Eduard ZingermanFrom: Alexei Starovoitov <[email protected]>
The new liveness analysis in liveness.c adds verbose output at BPF_LOG_LEVEL2, making the verifier log for good_prog exceed the 1024-byte reference buffer. When the reference is truncated in fixed mode, the rolling mode captures the actual tail of the full log, which doesn't match the truncated reference.
The fix is to increase the buffer sizes in the test.
--- tools/testing/selftests/bpf/prog_tests/verifier_log.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/bpf/prog_tests/verifier_log.c +3 -3
@@ -25,10 +25,10 @@ static bool check_prog_load(int prog_fd, bool expect_err, const char *tag)static struct {/* strategically placed before others to avoid accidental modification by kernel */ -char filler[1024 ]; -char buf[1024 ]; +char filler[16384 ]; +char buf[16384 ];/* strategically placed after buf[] to catch more accidental corruptions */ -char reference[1024 ]; +char reference[16384 ]; } logs;static const struct bpf_insn *insns;static size_t insn_cnt;
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 12/13 selftests/bpf: add new tests for static stack liveness analysis
In reply to Eduard ZingermanFrom: Alexei Starovoitov <[email protected]>
Add a bunch of new tests to verify the static stack liveness analysis.
--- .../selftests/bpf/progs/verifier_live_stack.c | 2129 ++++++++++++++++++++ 1 file changed, 2129 insertions(+)
tools/testing/selftests/bpf/progs/verifier_live_stack.c +2129 -0
@@ -3,6 +3,7 @@#include <linux/bpf.h> #include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" #include "bpf_misc.h" char _license[] SEC("license") = "GPL";
@@ -13,6 +14,15 @@ struct { __type(value, long long); } map SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, __u64); +} array_map_8b SEC(".maps"); + +const char snprintf_u64_fmt[] ="%llu" ; + SEC("socket") __log_level(2)__msg ("0: (79) r1 = *(u64 *)(r10 -8) ; use: fp0-8" )
@@ -154,6 +164,33 @@ static __used __naked void read_first_param(void) ::: __clobber_all); } +SEC("socket") +__success +__naked void arg_track_join_convergence (void ) +{ + asmvolatile ( +"r1 = 1;" +"r2 = 2;" +"call arg_track_join_convergence_subprog;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void arg_track_join_convergence_subprog (void ) +{ + asmvolatile ( +"if r1 == 0 goto 1f;" +"r0 = r1;" +"goto 2f;" +"1:" +"r0 = r2;" +"2:" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + SEC("socket")__flag (BPF_F_TEST_STATE_FREQ) __log_level(2)
@@ -324,3 +361,2095 @@ static __used __naked unsigned long write_tail_call(void)__imm_addr (map_array) : __clobber_all); } + +/* Test precise subprog stack access analysis. +* Caller passes fp-32 (SPI 3) to callee that only accesses arg+0 and arg+8 +* (SPIs 3 and 2). Slots 0 and 1 should NOT be live at the call site. +* +* Insn layout: +* 0: *(u64*)(r10 - 8) = 0 write SPI 0 +* 1: *(u64*)(r10 - 16) = 0 write SPI 1 +* 2: *(u64*)(r10 - 24) = 0 write SPI 2 +* 3: *(u64*)(r10 - 32) = 0 write SPI 3 +* 4: r1 = r10 +* 5: r1 += -32 +* 6: call precise_read_two passes fp-32 (SPI 3) +* 7: r0 = 0 +* 8: exit +* +* At insn 6 only SPIs 2,3 should be live (slots 4-7, 0xf0). +* SPIs 0,1 are written but never read → dead. +*/ +SEC("socket") +__log_level(2) +__msg("6: (85) call pc+{{.*}} ; use: fp0-24 fp0-32{{$}}") +__naked void subprog_precise_stack_access (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u64 *)(r10 - 16) = 0;" +"*(u64 *)(r10 - 24) = 0;" +"*(u64 *)(r10 - 32) = 0;" +"r1 = r10;" +"r1 += -32;" +"call precise_read_two;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Callee reads only at arg+0 (SPI 3) and arg+8 (SPI 2) */ +static __used __naked void precise_read_two (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" +"r2 = *(u64 *)(r1 + 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that multi-level subprog calls (callee passes arg-derived ptr +* to another BPF subprog) are analyzed precisely. +* +* Caller passes fp-32 (SPI 3). The callee forwards it to inner_callee. +* inner_callee only reads at offset 0 from the pointer. +* The analysis recurses into forward_to_inner -> inner_callee and +* determines only SPI 3 is accessed (slots 6-7, 0xc0), not all of SPIs 0-3. +* +* Insn layout: +* 0: *(u64*)(r10 - 8) = 0 write SPI 0 +* 1: *(u64*)(r10 - 16) = 0 write SPI 1 +* 2: *(u64*)(r10 - 24) = 0 write SPI 2 +* 3: *(u64*)(r10 - 32) = 0 write SPI 3 +* 4: r1 = r10 +* 5: r1 += -32 +* 6: call forward_to_inner passes fp-32 (SPI 3) +* 7: r0 = 0 +* 8: exit +*/ +SEC("socket") +__log_level(2) +__msg("6: (85) call pc+{{.*}} ; use: fp0-32{{$}}") +__naked void subprog_multilevel_conservative (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u64 *)(r10 - 16) = 0;" +"*(u64 *)(r10 - 24) = 0;" +"*(u64 *)(r10 - 32) = 0;" +"r1 = r10;" +"r1 += -32;" +"call forward_to_inner;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Forwards arg to another subprog */ +static __used __naked void forward_to_inner (void ) +{ + asmvolatile ( +"call inner_callee;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void inner_callee (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test multi-frame precision loss: callee consumes caller stack early, +* but static liveness keeps it live at pruning points inside callee. +* +* Caller stores map_ptr or scalar(42) at fp-8, then calls +* consume_and_call_inner. The callee reads fp0-8 at entry (consuming +* the slot), then calls do_nothing2. After do_nothing2 returns (a +* pruning point), fp-8 should be dead -- the read already happened. +* But because the call instruction's stack_use includes SPI 0, the +* static live_stack_before at insn 7 is 0x1, keeping fp-8 live inside +* the callee and preventing state pruning between the two paths. +* +* Insn layout: +* 0: call bpf_get_prandom_u32 +* 1: if r0 == 42 goto pc+2 -> insn 4 +* 2: r0 = map ll (ldimm64 part1) +* 3: (ldimm64 part2) +* 4: *(u64)(r10 - 8) = r0 fp-8 = map_ptr OR scalar(42) +* 5: r1 = r10 +* 6: r1 += -8 +* 7: call consume_and_call_inner +* 8: r0 = 0 +* 9: exit +* +* At insn 7, live_stack_before = 0x3 (slots 0-1 live due to stack_use). +* At insn 8, live_stack_before = 0x0 (SPI 0 dead, caller doesn't need it). +*/ +SEC("socket") +__flag(BPF_F_TEST_STATE_FREQ) +__log_level(2) +__success +__msg(" 7: (85) call pc+{{.*}} ; use: fp0-8") +__msg(" 8: {{.*}} (b7)") +__naked void callee_consumed_caller_stack(void) +{ + asmvolatile ( +"call %[bpf_get_prandom_u32];" +"if r0 == 42 goto 1f;" +"r0 = %[map] ll;" +"1:" +"*(u64 *)(r10 - 8) = r0;" +"r1 = r10;" +"r1 += -8;" +"call consume_and_call_inner;" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm_addr (map) + : __clobber_all); +} + +static __used __naked void consume_and_call_inner (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" /* read fp[0]-8 into caller-saved r0 */ +"call do_nothing2;" /* inner call clobbers r0 */ +"r0 = 0;" +"goto +0;" /* checkpoint */ +"r0 = 0;" +"goto +0;" /* checkpoint */ +"r0 = 0;" +"goto +0;" /* checkpoint */ +"r0 = 0;" +"goto +0;" /* checkpoint */ +"exit;" + ::: __clobber_all); +} + +static __used __naked void do_nothing2 (void ) +{ + asmvolatile ( +"r0 = 0;" +"r0 = 0;" +"r0 = 0;" +"r0 = 0;" +"r0 = 0;" +"r0 = 0;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Reproducer for unsound pruning when clean_verifier_state() promotes +* live STACK_ZERO bytes to STACK_MISC. +* +* Program shape: +* - Build key at fp-4: +* - path A keeps key byte as STACK_ZERO; +* - path B writes unknown byte making it STACK_MISC. +* - Branches merge at a prune point before map_lookup. +* - map_lookup on ARRAY map is value-sensitive to constant zero key: +* - path A: const key 0 => PTR_TO_MAP_VALUE (non-NULL); +* - path B: non-const key => PTR_TO_MAP_VALUE_OR_NULL. +* - Dereference lookup result without null check. +* +* Note this behavior won't trigger at fp-8, since the verifier will +* track 32-bit scalar spill differently as spilled_ptr. +* +* Correct verifier behavior: reject (path B unsafe). +* With blanket STACK_ZERO->STACK_MISC promotion on live slots, cached path A +* state can be generalized and incorrectly prune path B, making program load. +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'map_value_or_null'") +__naked void stack_zero_to_misc_unsound_array_lookup (void ) +{ + asmvolatile ( +/* key at fp-4: all bytes STACK_ZERO */ +"*(u32 *)(r10 - 4) = 0;" +"call %[bpf_get_prandom_u32];" +/* fall-through (path A) explored first */ +"if r0 != 0 goto l_nonconst%=;" +/* path A: keep key constant zero */ +"goto l_lookup%=;" +"l_nonconst%=:" +/* path B: key byte turns to STACK_MISC, key no longer const */ +"*(u8 *)(r10 - 4) = r0;" +"l_lookup%=:" +/* value-sensitive lookup */ +"r2 = r10;" +"r2 += -4;" +"r1 = %[array_map_8b] ll;" +"call %[bpf_map_lookup_elem];" +/* unsafe when lookup result is map_value_or_null */ +"r0 = *(u64 *)(r0 + 0);" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_map_lookup_elem), +__imm_addr (array_map_8b) + : __clobber_all); +} + +/* +* Subprog variant of stack_zero_to_misc_unsound_array_lookup. +* +* Check unsound pruning when a callee modifies the caller's +* stack through a pointer argument. +* +* Program shape: +* main: +* *(u32)(fp - 4) = 0 key = 0 (all bytes STACK_ZERO) +* r1 = fp - 4 +* call maybe_clobber_key may overwrite key[0] with scalar +* <-- prune point: two states meet here --> +* r2 = fp - 4 +* r1 = array_map_8b +* call bpf_map_lookup_elem value-sensitive on const-zero key +* r0 = *(u64)(r0 + 0) deref without null check +* exit +* +* maybe_clobber_key(r1): +* r6 = r1 save &key +* call bpf_get_prandom_u32 +* if r0 == 0 goto skip path A: key stays STACK_ZERO +* *(u8)(r6 + 0) = r0 path B: key[0] becomes STACK_MISC +* skip: +* r0 = 0 +* exit +* +* Path A: const-zero key => array lookup => PTR_TO_MAP_VALUE => deref OK. +* Path B: non-const key => array lookup => PTR_TO_MAP_VALUE_OR_NULL => UNSAFE. +* +* If the cleaner collapses STACK_ZERO -> STACK_MISC for the live key +* slot, path A's cached state matches path B, pruning the unsafe path. +* +* Correct verifier behaviour: reject. +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'map_value_or_null'") +__naked void subprog_stack_zero_to_misc_unsound (void ) +{ + asmvolatile ( +/* key at fp-4: all bytes STACK_ZERO */ +"*(u32 *)(r10 - 4) = 0;" +/* subprog may clobber key[0] with a scalar byte */ +"r1 = r10;" +"r1 += -4;" +"call maybe_clobber_key;" +/* value-sensitive array lookup */ +"r2 = r10;" +"r2 += -4;" +"r1 = %[array_map_8b] ll;" +"call %[bpf_map_lookup_elem];" +/* unsafe when result is map_value_or_null (path B) */ +"r0 = *(u64 *)(r0 + 0);" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm_addr (array_map_8b) + : __clobber_all); +} + +static __used __naked void maybe_clobber_key (void ) +{ + asmvolatile ( +"r6 = r1;" +"call %[bpf_get_prandom_u32];" +/* path A (r0==0): key stays STACK_ZERO, explored first */ +"if r0 == 0 goto 1f;" +/* path B (r0!=0): overwrite key[0] with scalar */ +"*(u8 *)(r6 + 0) = r0;" +"1:" +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* +* Demonstrate that subprog arg spill/reload breaks arg tracking, +* inflating caller stack liveness and preventing state pruning. +* +* modifier2(fp-24) has two paths: one writes a scalar to *(r1+8) +* = caller fp-16, the other leaves it as zero. After modifier2 +* returns, fp-16 is never read again — it is dead. +* +* spill_reload_reader2(fp-24) only reads caller fp-8 via +* *(r1+16), but it spills r1 across a helper call. This +* breaks compute_subprog_arg_access(): the reload from callee +* stack cannot be connected back to arg1, so arg1 access goes +* "all (conservative)". At the call site (r1 = fp-24, slot 5) +* apply_callee_stack_access() marks slots 0..5 as stack_use — +* pulling fp-16 (slots 2-3) into live_stack_before even though +* the reader never touches it. +* +* Result: at modifier2's return point two states with different +* fp-16 values cannot be pruned. +* +* With correct (or old dynamic) liveness fp-16 is dead at that +* point and the states prune → "6: safe" appears in the log. +*/ +SEC("socket") +__flag(BPF_F_TEST_STATE_FREQ) +__log_level(2) +__success +__msg("6: safe") +__naked void spill_reload_inflates_stack_liveness(void) +{ + asm volatile ( +/* struct at fp-24: { ctx; ptr; tail; } */ +"*(u64 *)(r10 - 24) = r1;" /* fp-24 = ctx */ +"*(u64 *)(r10 - 16) = r1;" /* fp-16 = ctx (STACK_SPILL ptr) */ +"*(u64 *)(r10 - 8) = 0;" /* fp-8 = tail */ +/* modifier2 writes different values to fp-16 on two paths */ +"r1 = r10;" +"r1 += -24;" +"call modifier2;" +/* insn 6: prune point — two states with different fp-16 +* path A: fp-16 = STACK_MISC (scalar overwrote pointer) +* path B: fp-16 = STACK_SPILL (original ctx pointer) +* STACK_MISC does NOT subsume STACK_SPILL(ptr), +* so pruning fails unless fp-16 is cleaned (dead). +*/ +"r1 = r10;" +"r1 += -24;" +"call spill_reload_reader2;" /* reads fp-8 via *(r1+16) */ +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Two paths: one writes a scalar to *(r1+8) = caller fp-16, +* the other leaves it unchanged. Both return 0 via separate +* exits to prevent pruning inside the subprog at the merge. +*/ +static __used __naked void modifier2 (void ) +{ + asmvolatile ( +"r6 = r1;" +"call %[bpf_get_prandom_u32];" +"if r0 == 0 goto 1f;" +"*(u64 *)(r6 + 8) = r0;" /* fp-16 = random */ +"r0 = 0;" +"exit;" /* path A exit */ +"1:" +"r0 = 0;" +"exit;" /* path B exit */ + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* Receives r1 = caller fp-24. Only reads *(r1+16) = fp-8. +* Spills r1 across a helper call → arg tracking goes conservative → +* slots 0..5 all appear used instead of just slot 1 (fp-8). +*/ +static __used __naked void spill_reload_reader2 (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = r1;" /* spill arg1 */ +"call %[bpf_get_prandom_u32];" /* clobbers r1-r5 */ +"r1 = *(u64 *)(r10 - 8);" /* reload arg1 */ +"r0 = *(u64 *)(r1 + 16);" /* read caller fp-8 */ +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* BTF FUNC records are not generated for kfuncs referenced +* from inline assembly. These records are necessary for +* libbpf to link the program. The function below is a hack +* to ensure that BTF FUNC records are generated. +*/ +void __kfunc_btf_root (void ) +{ +bpf_iter_num_new (0 ,0 ,0 ); +bpf_iter_num_next (0 ); +bpf_iter_num_destroy (0 ); +} + +/* Test that open-coded iterator kfunc arguments get precise stack +* liveness tracking. struct bpf_iter_num is 8 bytes (1 SPI). +* +* Insn layout: +* 0: *(u64*)(r10 - 8) = 0 write SPI 0 (dead) +* 1: *(u64*)(r10 - 16) = 0 write SPI 1 (dead) +* 2: r1 = r10 +* 3: r1 += -24 iter state at fp-24 (SPI 2) +* 4: r2 = 0 +* 5: r3 = 10 +* 6: call bpf_iter_num_new defines SPI 2 (KF_ITER_NEW) → 0x0 +* 7-8: r1 = fp-24 +* 9: call bpf_iter_num_next uses SPI 2 → 0x30 +* 10: if r0 == 0 goto 2f +* 11: goto 1b +* 12-13: r1 = fp-24 +* 14: call bpf_iter_num_destroy uses SPI 2 → 0x30 +* 15: r0 = 0 +* 16: exit +* +* At insn 6, SPI 2 is defined (KF_ITER_NEW initializes, doesn't read), +* so it kills liveness from successors. live_stack_before = 0x0. +* At insns 9 and 14, SPI 2 is used (iter_next/destroy read the state), +* so live_stack_before = 0x30. +*/ +SEC("socket") +__success __log_level(2) +__msg(" 6: (85) call bpf_iter_num_new{{.*}} ; def: fp0-24{{$}}") +__msg(" 9: (85) call bpf_iter_num_next{{.*}} ; use: fp0-24{{$}}") +__msg("14: (85) call bpf_iter_num_destroy{{.*}} ; use: fp0-24{{$}}") +__naked void kfunc_iter_stack_liveness (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" /* SPI 0 - dead */ +"*(u64 *)(r10 - 16) = 0;" /* SPI 1 - dead */ +"r1 = r10;" +"r1 += -24;" +"r2 = 0;" +"r3 = 10;" +"call %[bpf_iter_num_new];" +"1:" +"r1 = r10;" +"r1 += -24;" +"call %[bpf_iter_num_next];" +"if r0 == 0 goto 2f;" +"goto 1b;" +"2:" +"r1 = r10;" +"r1 += -24;" +"call %[bpf_iter_num_destroy];" +"r0 = 0;" +"exit;" + :: __imm(bpf_iter_num_new), +__imm (bpf_iter_num_next), +__imm (bpf_iter_num_destroy) + : __clobber_all); +} + +/* +* Test for soundness bug in static stack liveness analysis. +* +* The static pre-pass tracks FP-derived register offsets to determine +* which stack slots are accessed. When a PTR_TO_STACK is spilled to +* the stack and later reloaded, the reload (BPF_LDX) kills FP-derived +* tracking, making subsequent accesses through the reloaded pointer +* invisible to the static analysis. +* +* This causes the analysis to incorrectly mark SPI 0 as dead at the +* merge point. clean_verifier_state() zeros it in the cached state, +* and stacksafe() accepts the new state against STACK_INVALID, +* enabling incorrect pruning. +* +* Path A (verified first): stores PTR_TO_MAP_VALUE in SPI 0 +* Path B (verified second): stores scalar 42 in SPI 0 +* After merge: reads SPI 0 through spilled/reloaded PTR_TO_STACK +* and dereferences the result as a pointer. +* +* Correct behavior: reject (path B dereferences a scalar) +* Bug behavior: accept (path B is incorrectly pruned) +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'scalar'") +__naked void spill_ptr_liveness_type_confusion (void ) +{ + asmvolatile ( +/* Map lookup to get PTR_TO_MAP_VALUE */ +"r1 = %[map] ll;" +"*(u32 *)(r10 - 32) = 0;" +"r2 = r10;" +"r2 += -32;" +"call %[bpf_map_lookup_elem];" +"if r0 == 0 goto l_exit%=;" +/* r6 = PTR_TO_MAP_VALUE (callee-saved) */ +"r6 = r0;" +/* Branch: fall-through (path A) verified first */ +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto l_scalar%=;" +/* Path A: store map value ptr at SPI 0 */ +"*(u64 *)(r10 - 8) = r6;" +"goto l_merge%=;" +"l_scalar%=:" +/* Path B: store scalar at SPI 0 */ +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"l_merge%=:" +/* +* Spill PTR_TO_STACK{off=-8} to SPI 1, then reload. +* Reload kills FP-derived tracking, hiding the +* subsequent SPI 0 access from the static analysis. +*/ +"r1 = r10;" +"r1 += -8;" +"*(u64 *)(r10 - 16) = r1;" +"goto +0;" /* checkpoint */ +"goto +0;" /* checkpoint */ +"goto +0;" /* checkpoint */ +"r1 = *(u64 *)(r10 - 16);" +/* Read SPI 0 through reloaded pointer */ +"r0 = *(u64 *)(r1 + 0);" +/* Dereference: safe for map value (path A), +* unsafe for scalar (path B). +*/ +"r0 = *(u64 *)(r0 + 0);" +"exit;" +"l_exit%=:" +"r0 = 0;" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm (bpf_get_prandom_u32), +__imm_addr (map) + : __clobber_all); +} + +/* === Tests for 4-byte stack slot liveness granularity === */ + +/* Test that a 4-byte aligned write is stack_def and kills liveness. +* +* 0: *(u64 *)(r10 - 8) = 0 def slots 0,1 (full SPI 0) +* 1: *(u32 *)(r10 - 8) = 0 def slot 1 (4-byte write kills slot 1) +* 2: r0 = *(u64 *)(r10 - 8) use slots 0,1 +* 3: r0 = 0 +* 4: exit +* +* At insn 1, the 4-byte write defines slot 1. Slot 0 still flows +* backward from insn 2's read: live_stack_before = 0x1. +*/ +SEC("socket") +__log_level(2) +__msg("1: (62) *(u32 *)(r10 -8) = 0 ; def: fp0-8h") +__naked void four_byte_write_kills_slot (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u32 *)(r10 - 8) = 0;" +"r0 = *(u64 *)(r10 - 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that a write to the upper half of an SPI is dead when only +* the lower half is read. This was impossible at SPI granularity +* where any read of the SPI kept the entire SPI live. +* +* 0: *(u32 *)(r10 - 8) = 0 def slot 1 (DEAD: never read) +* 1: *(u32 *)(r10 - 4) = 0 def slot 0 +* 2: r0 = *(u32 *)(r10 - 4) use slot 0 only +* 3: r0 = 0 +* 4: exit +* +* At insn 0, nothing is live (0x0). Previously at SPI granularity, +* the read at insn 2 would mark the full SPI 0 as live and the +* 4-byte writes wouldn't count as def, so insn 0 would have had +* SPI 0 live (0x1). +*/ +SEC("socket") +__log_level(2) +__msg("0: (62) *(u32 *)(r10 -8) = 0 ; def: fp0-8h") +__msg("2: (61) r0 = *(u32 *)(r10 -4) ; use: fp0-4h") +__naked void dead_half_spi_write (void ) +{ + asmvolatile ( +"*(u32 *)(r10 - 8) = 0;" +"*(u32 *)(r10 - 4) = 0;" +"r0 = *(u32 *)(r10 - 4);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that a 4-byte read from the upper half of SPI 0 makes only +* slot 1 live (0x2), not the full SPI (0x3). +* +* 0: *(u64 *)(r10 - 8) = 0 def slots 0,1 +* 1: r0 = *(u32 *)(r10 - 8) use slot 1 only (upper half) +* 2: r0 = 0 +* 3: exit +* +* At insn 1, live_stack_before = 0x2 (slot 1 only). +*/ +SEC("socket") +__log_level(2) +__msg("1: (61) r0 = *(u32 *)(r10 -8) ; use: fp0-8h") +__naked void four_byte_read_upper_half (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"r0 = *(u32 *)(r10 - 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that a 2-byte write does NOT count as stack_def. +* Sub-4-byte writes don't fully cover a 4-byte slot, +* so liveness passes through. +* +* 0: *(u64 *)(r10 - 8) = 0 def slots 0,1 +* 1: *(u16 *)(r10 - 4) = 0 NOT stack_def (2 < 4 bytes) +* 2: r0 = *(u32 *)(r10 - 4) use slot 0 +* 3: r0 = 0 +* 4: exit +* +* At insn 1, slot 0 still live (0x1) because 2-byte write +* didn't kill it. +*/ +SEC("socket") +__log_level(2) +__msg("0: (7a) *(u64 *)(r10 -8) = 0 ; def: fp0-8") +__msg("1: (6a) *(u16 *)(r10 -4) = 0{{$}}") +__msg("2: (61) r0 = *(u32 *)(r10 -4) ; use: fp0-4h") +__naked void two_byte_write_no_kill (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u16 *)(r10 - 4) = 0;" +"r0 = *(u32 *)(r10 - 4);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that a 1-byte write does NOT count as stack_def. +* +* 0: *(u64 *)(r10 - 8) = 0 def slots 0,1 +* 1: *(u8 *)(r10 - 4) = 0 NOT stack_def (1 < 4 bytes) +* 2: r0 = *(u32 *)(r10 - 4) use slot 0 +* 3: r0 = 0 +* 4: exit +* +* At insn 1, slot 0 still live (0x1). +*/ +SEC("socket") +__log_level(2) +__msg("0: (7a) *(u64 *)(r10 -8) = 0 ; def: fp0-8") +__msg("1: (72) *(u8 *)(r10 -4) = 0") +__msg("2: (61) r0 = *(u32 *)(r10 -4) ; use: fp0-4h") +__naked void one_byte_write_no_kill (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u8 *)(r10 - 4) = 0;" +"r0 = *(u32 *)(r10 - 4);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test stack access beyond fp-256 exercising the second bitmask word. +* fp-264 is SPI 32, slots 64-65, which are bits 0-1 of live_stack[1]. +* +* 0: *(u64 *)(r10 - 264) = 0 def slots 64,65 +* 1: r0 = *(u64 *)(r10 - 264) use slots 64,65 +* 2: r0 = 0 +* 3: exit +* +* At insn 1, live_stack high word has bits 0,1 set: 0x3:0x0. +*/ +SEC("socket") +__log_level(2) +__msg("1: (79) r0 = *(u64 *)(r10 -264) ; use: fp0-264") +__naked void high_stack_second_bitmask_word (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 264) = 0;" +"r0 = *(u64 *)(r10 - 264);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that two separate 4-byte writes to each half of an SPI +* together kill liveness for the full SPI. +* +* 0: *(u32 *)(r10 - 8) = 0 def slot 1 (upper half) +* 1: *(u32 *)(r10 - 4) = 0 def slot 0 (lower half) +* 2: r0 = *(u64 *)(r10 - 8) use slots 0,1 +* 3: r0 = 0 +* 4: exit +* +* At insn 0: live_stack_before = 0x0 (both slots killed by insns 0,1). +* At insn 1: live_stack_before = 0x2 (slot 1 still live, slot 0 killed here). +*/ +SEC("socket") +__log_level(2) +__msg("0: (62) *(u32 *)(r10 -8) = 0 ; def: fp0-8h") +__msg("1: (62) *(u32 *)(r10 -4) = 0 ; def: fp0-4h") +__naked void two_four_byte_writes_kill_full_spi (void ) +{ + asmvolatile ( +"*(u32 *)(r10 - 8) = 0;" +"*(u32 *)(r10 - 4) = 0;" +"r0 = *(u64 *)(r10 - 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Test that 4-byte writes on both branches kill a slot at the +* join point. Previously at SPI granularity, a 4-byte write was +* not stack_def, so liveness would flow backward through the +* branch that only had a 4-byte write. +* +* 0: call bpf_get_prandom_u32 +* 1: if r0 != 0 goto 1f +* 2: *(u64 *)(r10 - 8) = 0 path A: def slots 0,1 +* 3: goto 2f +* 1:4: *(u32 *)(r10 - 4) = 0 path B: def slot 0 +* 2:5: r0 = *(u32 *)(r10 - 4) use slot 0 +* 6: r0 = 0 +* 7: exit +* +* Both paths define slot 0 before the read. At insn 1 (branch), +* live_stack_before = 0x0 because slot 0 is killed on both paths. +*/ +SEC("socket") +__log_level(2) +__msg("1: (55) if r0 != 0x0 goto pc+2") +__msg("2: (7a) *(u64 *)(r10 -8) = 0 ; def: fp0-8") +__msg("3: (05) goto pc+1") +__msg("4: (62) *(u32 *)(r10 -4) = 0 ; def: fp0-4h") +__msg("5: (61) r0 = *(u32 *)(r10 -4) ; use: fp0-4h") +__naked void both_branches_kill_slot (void ) +{ + asmvolatile ( +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto 1f;" +"*(u64 *)(r10 - 8) = 0;" +"goto 2f;" +"1:" +"*(u32 *)(r10 - 4) = 0;" +"2:" +"r0 = *(u32 *)(r10 - 4);" +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* Soundness: cleaning the dead upper half of an SPI must not +* affect the live lower half's type information for pruning. +* +* Both halves of SPI 0 are written separately. Only the lower +* half (slot 0) is used as a 4-byte map key. The upper half +* (slot 1) is dead and cleaned to STACK_INVALID. +* +* Path A: key stays 0 (STACK_ZERO) → non-null array lookup +* Path B: key byte turns STACK_MISC → may-null array lookup +* Deref without null check: safe for A, unsafe for B. +* +* If half-SPI cleaning incorrectly corrupted the live half's +* type info, path A's cached state could generalize and unsoundly +* prune path B. +* +* Expected: reject (path B unsafe). +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'map_value_or_null'") +__naked void half_spi_clean_preserves_stack_zero (void ) +{ + asmvolatile ( +"*(u32 *)(r10 - 4) = 0;" /* slot 0: STACK_ZERO */ +"*(u32 *)(r10 - 8) = 0;" /* slot 1: STACK_ZERO (dead) */ +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto l_nonconst%=;" +"goto l_lookup%=;" +"l_nonconst%=:" +"*(u8 *)(r10 - 4) = r0;" /* slot 0: STACK_MISC */ +"l_lookup%=:" +"r2 = r10;" +"r2 += -4;" +"r1 = %[array_map_8b] ll;" +"call %[bpf_map_lookup_elem];" +"r0 = *(u64 *)(r0 + 0);" /* unsafe if null */ +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_map_lookup_elem), +__imm_addr (array_map_8b) + : __clobber_all); +} + +/* +* Model of scx_lavd's pick_idle_cpu_at_cpdom iat block: +* conditional block with helper call and temporary stack spill, +* spill dead after merge. +* +* Path A (fall-through): spill r6 to fp-8 across helper call +* Path B (branch taken): skip the block entirely +* At merge (insn 6): fp-8 is dead (never read after merge) +* +* Static liveness marks fp-8 dead at merge. clean_verifier_state() +* converts path A's STACK_SPILL to STACK_INVALID. Path B has +* STACK_INVALID. stacksafe() matches -> path B pruned -> "6: safe". +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__success +__log_level(2) +__msg("6: safe") +__naked void dead_spill_at_merge_enables_pruning (void ) +{ + asmvolatile ( +"call %[bpf_get_prandom_u32];" +"r6 = 7;" +"if r0 != 0 goto l_skip%=;" +/* conditional block: spill, call, reload */ +"*(u64 *)(r10 - 8) = r6;" +"call %[bpf_get_prandom_u32];" +"r6 = *(u64 *)(r10 - 8);" +"l_skip%=:" +/* fp-8 dead. Path B pruned here -> "6: safe" */ +"r0 = r6;" +"exit;" + : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* +* FP-offset tracking loses precision on second ADD, killing all liveness. +* +* fp_off_insn_xfer() handles "FP itself + negative imm" precisely +* (e.g. r6 = r10; r6 += -24 -> slot 5). But any subsequent ADD/SUB +* on a register that already has non-zero spis falls through to +* spis_set_all(), because the code only handles the FP-itself case. +* +* A write through this imprecise register enters the non-zero-spis +* branch of set_indirect_stack_access(), which OR's the all-ones +* mask into stack_def. The backward liveness equation +* +* stack_in = (stack_out & ~stack_def) | stack_use +* +* sees ~ALL = 0, killing ALL slot liveness at that instruction. +* +* At the merge pruning point, live_stack_before is empty. +* clean_verifier_state() marks fp-8 as STACK_INVALID. +* stacksafe() skips STACK_INVALID (line "continue"), so pruning +* succeeds regardless of the current state's fp-8 value. +* Path B is pruned, its null deref is never explored. +* +* Correct behavior: reject (path B dereferences NULL). +* Bug behavior: accept (path B pruned away). +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R1 invalid mem access 'scalar'") +__naked void fp_add_loses_precision_kills_liveness (void ) +{ + asmvolatile ( +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto l_pathB%=;" + +/* Path A (fall-through, explored first): fp-8 = 0 */ +"r1 = 0;" +"*(u64 *)(r10 - 8) = r1;" +"goto l_merge%=;" + +"l_pathB%=:" +/* Path B (explored second): fp-8 = 42 */ +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" + +"l_merge%=:" +/* +* Create imprecise FP-derived register. +* r6 = r10 - 24 gets precise slot 5. +* r6 += 8 hits the else branch (spis non-zero, delta > 0) +* and sets spis to ALL. r6 is actually r10-16. +*/ +"r6 = r10;" +"r6 += -24;" +"r6 += 8;" + +/* +* Write through imprecise r6. Actually writes to fp-16 +* (does NOT touch fp-8), but liveness marks ALL slots +* as stack_def, killing fp-8's liveness. +*/ +"r7 = 0;" +"*(u64 *)(r6 + 0) = r7;" + +/* Read fp-8: liveness says dead, but value is needed. */ +"r2 = *(u64 *)(r10 - 8);" +"if r2 == 42 goto l_danger%=;" + +/* r2 != 42 (path A: r2 == 0): safe exit */ +"r0 = 0;" +"exit;" + +"l_danger%=:" +/* Only reachable from path B (r2 == 42): null deref */ +"r1 = 0;" +"r0 = *(u64 *)(r1 + 0);" +"exit;" + : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R1 invalid mem access 'scalar'") +__naked void fp_spill_loses_precision_kills_liveness (void ) +{ + asmvolatile ( +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto l_pathB%=;" + +"r1 = 0;" +"*(u64 *)(r10 - 8) = r1;" +"goto l_merge%=;" + +"l_pathB%=:" +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" + +"l_merge%=:" +"r6 = r10;" +"r6 += -64;" +"*(u64 *)(r10 - 160) = r6;" +"r6 = *(u64 *)(r10 - 160);" + +"r7 = 0;" +"*(u64 *)(r6 + 0) = r7;" + +"r2 = *(u64 *)(r10 - 8);" +"if r2 == 42 goto l_danger%=;" + +"r0 = *(u64 *)(r10 - 56);" +"exit;" + +"l_danger%=:" +"r1 = 0;" +"r0 = *(u64 *)(r1 + 0);" +"exit;" + : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* === Tests for frame-based AT_FP tracking === */ + +/* +* Test 1: conditional_stx_in_subprog +* Subprog conditionally writes caller's slot. +* Verify slot stays live (backward pass handles conditional def via CFG). +* +* Main writes fp-8=42, calls cond_writer(fp-8), reads fp-8. +* cond_writer only writes on one path → parent_def only on that path. +* The backward parent_live correctly keeps fp-8 live at entry +* (conditional write doesn't kill liveness at the join). +*/ +SEC("socket") +__log_level (2 ) +/* fp-8 live at call (callee conditionally writes → slot not killed) */ +__msg("1: (7b) *(u64 *)(r10 -8) = r1 ; def: fp0-8") +__msg("4: (85) call pc+2{{$}}") +__msg("5: (79) r0 = *(u64 *)(r10 -8) ; use: fp0-8") +__naked void conditional_stx_in_subprog (void ) +{ + asmvolatile ( +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"r1 = r10;" +"r1 += -8;" +"call cond_writer;" +"r0 = *(u64 *)(r10 - 8);" +"exit;" + ::: __clobber_all); +} + +/* Conditionally writes to *(r1+0) */ +static __used __naked void cond_writer (void ) +{ + asmvolatile ( +"r6 = r1;" +"call %[bpf_get_prandom_u32];" +"if r0 == 0 goto 1f;" +"*(u64 *)(r6 + 0) = r0;" +"1:" +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__log_level(2) +__msg("4: (85) call pc+{{.*}} ; use: fp0-16") +__msg("7: (85) call pc+{{.*}} ; use: fp0-32") +__naked void multiple_callsites_different_offsets (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 16) = 0;" +"*(u64 *)(r10 - 32) = 0;" +"r1 = r10;" +"r1 += -16;" +"call read_first_param;" +"r1 = r10;" +"r1 += -32;" +"call read_first_param;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Test 3: nested_fp_passthrough +* main→A→B, main's FP forwarded to B. B accesses main's stack. +* Verify liveness propagates through. +* +* Main passes fp-32 to outer_forwarder, which passes it to inner_reader. +* inner_reader reads at arg+0 (= main's fp-32). +* parent_live propagates transitively: inner→outer→main. +*/ +SEC("socket") +__log_level (2 ) +/* At call to outer_forwarder: main's fp-32 (slots 6,7) should be live */ +__msg("6: (85) call pc+{{.*}} ; use: fp0-32") +__naked void nested_fp_passthrough (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u64 *)(r10 - 16) = 0;" +"*(u64 *)(r10 - 24) = 0;" +"*(u64 *)(r10 - 32) = 0;" +"r1 = r10;" +"r1 += -32;" +"call outer_forwarder;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Forwards arg to inner_reader */ +static __used __naked void outer_forwarder (void ) +{ + asmvolatile ( +"call inner_reader;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void inner_reader (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Test 4: callee_must_write_before_read +* Callee unconditionally writes parent slot before reading. +* Verify slot is NOT live at call site (parent_def kills it). +*/ +SEC("socket") +__log_level (2 ) +/* fp-8 NOT live at call: callee writes before reading (parent_def kills it) */ +__msg("2: .12345.... (85) call pc+") +__naked void callee_must_write_before_read (void ) +{ + asmvolatile ( +"r1 = r10;" +"r1 += -8;" +"call write_then_read;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Unconditionally writes *(r1+0), then reads it back */ +static __used __naked void write_then_read (void ) +{ + asmvolatile ( +"r6 = r1;" +"r7 = 99;" +"*(u64 *)(r6 + 0) = r7;" +"r0 = *(u64 *)(r6 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Test 5: return_site_liveness_bleeding +* Main calls subprog twice. Slot used after one call but not the other. +* Context-insensitive: slot conservatively live at both. +* +* After first call: read fp-8. +* After second call: don't read fp-8. +* Since parent_live is per-subprog (not per call-site), +* fp-8 is live at both call sites. +*/ +SEC("socket") +__log_level (2 ) +/* Both calls have fp-8 live due to context-insensitive parent_live */ +__msg("3: (85) call pc+{{.*}} ; use: fp0-8") +__msg("7: (85) call pc+{{.*}} ; use: fp0-8") +__naked void return_site_liveness_bleeding (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"r1 = r10;" +"r1 += -8;" +"call read_first_param;" +"r0 = *(u64 *)(r10 - 8);" +"r1 = r10;" +"r1 += -8;" +"call read_first_param;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +SEC("socket") +__log_level(2) +__msg("9: (85) call bpf_loop#181 ; use: fp0-16") +__naked void callback_conditional_read_beyond_ctx (void ) +{ + asmvolatile ( +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"*(u64 *)(r10 - 16) = r1;" +"r1 = 2;" +"r2 = cb_cond_read ll;" +"r3 = r10;" +"r3 += -8;" +"r4 = 0;" +"call %[bpf_loop];" +"r0 = 0;" +"exit;" + :: __imm(bpf_loop) + : __clobber_all); +} + +/* Callback conditionally reads *(ctx - 8) = caller fp-16 */ +static __used __naked void cb_cond_read (void ) +{ + asmvolatile ( +"r6 = r2;" +"call %[bpf_get_prandom_u32];" +"if r0 == 0 goto 1f;" +"r0 = *(u64 *)(r6 - 8);" +"1:" +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__log_level(2) +__msg("14: (7b) *(u64 *)(r6 -8) = r7 ; def: fp0-16") +__msg("15: (79) r0 = *(u64 *)(r6 -8) ; use: fp0-16") +__naked void callback_write_before_read_kills (void ) +{ + asmvolatile ( +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"*(u64 *)(r10 - 16) = r1;" +"r1 = 2;" +"r2 = cb_write_read ll;" +"r3 = r10;" +"r3 += -8;" +"r4 = 0;" +"call %[bpf_loop];" +"r0 = 0;" +"exit;" + :: __imm(bpf_loop) + : __clobber_all); +} + +/* Callback unconditionally writes *(ctx-8), then reads it back. +* The write (parent_def) kills liveness before entry. +*/ +static __used __naked void cb_write_read (void ) +{ + asmvolatile ( +"r6 = r2;" +"r7 = 99;" +"*(u64 *)(r6 - 8) = r7;" +"r0 = *(u64 *)(r6 - 8);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* bpf_loop callback conditionally writes fp-16 then unconditionally +* reads it. The conditional write does NOT kill liveness +*/ +SEC("socket") +__log_level(2) +__msg("9: (85) call bpf_loop#181 ; use: fp0-16") +__naked void callback_conditional_write_preserves (void ) +{ + asmvolatile ( +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"*(u64 *)(r10 - 16) = r1;" +"r1 = 2;" +"r2 = cb_cond_write_read ll;" +"r3 = r10;" +"r3 += -8;" +"r4 = 0;" +"call %[bpf_loop];" +"r0 = 0;" +"exit;" + :: __imm(bpf_loop) + : __clobber_all); +} + +static __used __naked void cb_cond_write_read (void ) +{ + asmvolatile ( +"r6 = r2;" +"call %[bpf_get_prandom_u32];" +"if r0 == 0 goto 1f;" +"*(u64 *)(r6 - 8) = r0;" +"1:" +"r0 = *(u64 *)(r6 - 8);" +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* +* Two bpf_loop calls with the same callback but different ctx pointers. +* +* First call: ctx=fp-8, second call: ctx=fp-24. +*/ +SEC("socket") +__log_level(2) +__msg(" 8: (85) call bpf_loop{{.*}} ; use: fp0-8") +__msg("15: (85) call bpf_loop{{.*}} ; use: fp0-24") +__naked void callback_two_calls_different_ctx (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = 0;" +"*(u64 *)(r10 - 24) = 0;" +"r1 = 1;" +"r2 = cb_read_ctx ll;" +"r3 = r10;" +"r3 += -8;" +"r4 = 0;" +"call %[bpf_loop];" +"r1 = 1;" +"r2 = cb_read_ctx ll;" +"r3 = r10;" +"r3 += -24;" +"r4 = 0;" +"call %[bpf_loop];" +"r0 = 0;" +"exit;" + :: __imm(bpf_loop) + : __clobber_all); +} + +/* Callback reads at ctx+0 unconditionally */ +static __used __naked void cb_read_ctx (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r2 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Reproducer for unsound pruning in refined_caller_live_stack(). +* +* Three-level call chain: main → mid_fwd → grandchild_deref. +* Main passes &fp-8 to mid_fwd, which forwards R1 to grandchild_deref. +* grandchild_deref reads main's fp-8 through the forwarded pointer +* and dereferences the result. +* +* refined_caller_live_stack() has a callee_offset++ when mid_fwd +* (frame 1) is mid-call. This drops the transitive parent_live +* contribution at mid_fwd's call instruction — the only place +* where grandchild_deref's read of main's fp-8 is recorded. +* As a result, main's fp-8 is cleaned to STACK_INVALID at the +* pruning point inside grandchild_deref, and path B is +* incorrectly pruned against path A. +* +* Path A: main stores PTR_TO_MAP_VALUE at fp-8 +* Path B: main stores scalar 42 at fp-8 +* +* Correct behavior: reject (path B dereferences scalar) +* Bug behavior: accept (path B pruned against cleaned path A) +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'scalar'") +__naked void transitive_parent_stack_read_unsound (void ) +{ + asmvolatile ( +/* Map lookup to get PTR_TO_MAP_VALUE */ +"r1 = %[map] ll;" +"*(u32 *)(r10 - 32) = 0;" +"r2 = r10;" +"r2 += -32;" +"call %[bpf_map_lookup_elem];" +"if r0 == 0 goto l_exit%=;" +"r6 = r0;" +/* Branch: path A (fall-through) explored first */ +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto l_scalar%=;" +/* Path A: fp-8 = PTR_TO_MAP_VALUE */ +"*(u64 *)(r10 - 8) = r6;" +"goto l_merge%=;" +"l_scalar%=:" +/* Path B: fp-8 = scalar 42 */ +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"l_merge%=:" +/* Pass &fp-8 to mid_fwd → grandchild_deref */ +"r1 = r10;" +"r1 += -8;" +"call mid_fwd;" +"r0 = 0;" +"exit;" +"l_exit%=:" +"r0 = 0;" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm (bpf_get_prandom_u32), +__imm_addr (map) + : __clobber_all); +} + +/* Forwards R1 (ptr to main's fp-8) to grandchild_deref */ +static __used __naked void mid_fwd (void ) +{ + asmvolatile ( +"call grandchild_deref;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Reads main's fp-8 through forwarded pointer, dereferences result */ +static __used __naked void grandchild_deref (void ) +{ + asmvolatile ( +"goto +0;" /* checkpoint */ +"goto +0;" /* checkpoint */ +/* read main's fp-8: map_ptr (path A) or scalar (path B) */ +"r0 = *(u64 *)(r1 + 0);" +/* dereference: safe for map_ptr, unsafe for scalar */ +"r0 = *(u64 *)(r0 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +SEC("socket") +__log_level(2) +__success +__msg("14: (79) r1 = *(u64 *)(r10 -8) // r6=fp0-8 r7=fp1-16 fp-8=fp1-16 fp-16=fp0-8") +__msg("15: (79) r0 = *(u64 *)(r1 +0) // r1=fp1-16 r6=fp0-8 r7=fp1-16 fp-8=fp1-16 fp-16=fp0-8") +__msg("stack use/def subprog#1 mid_two_fp_threshold (d1,cs2):") +__msg("14: (79) r1 = *(u64 *)(r10 -8) ; use: fp1-8") +__msg("15: (79) r0 = *(u64 *)(r1 +0) ; use: fp1-16") +__naked void two_fp_clear_stack_threshold (void ) +{ + asmvolatile ( +"r1 = r10;" +"r1 += -8;" +"call mid_two_fp_threshold;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void mid_two_fp_threshold (void ) +{ + asmvolatile ( +"r6 = r1;" +"r7 = r10;" +"r7 += -16;" +"*(u64 *)(r10 - 8) = r7;" +"*(u64 *)(r10 - 16) = r6;" +"r1 = r10;" +"r1 += -8;" +"r2 = r6;" +"call inner_nop_fptest;" +"r1 = *(u64 *)(r10 - 8);" +"r0 = *(u64 *)(r1 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void inner_nop_fptest (void ) +{ + asmvolatile ( +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +SEC("socket") +__log_level(2) +__success +__msg("13: (79) r1 = *(u64 *)(r10 -8) // r6=fp0-8 r7=fp1-16 fp-8=fp1-16 fp-16=fp0-8") +__msg("14: (79) r0 = *(u64 *)(r1 +0) // r1=fp1-16 r6=fp0-8 r7=fp1-16 fp-8=fp1-16 fp-16=fp0-8") +__msg("stack use/def subprog#1 mid_one_fp_threshold (d1,cs2):") +__msg("13: (79) r1 = *(u64 *)(r10 -8) ; use: fp1-8") +__msg("14: (79) r0 = *(u64 *)(r1 +0) ; use: fp1-16") +__naked void one_fp_clear_stack_threshold (void ) +{ + asmvolatile ( +"r1 = r10;" +"r1 += -8;" +"call mid_one_fp_threshold;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void mid_one_fp_threshold (void ) +{ + asmvolatile ( +"r6 = r1;" +"r7 = r10;" +"r7 += -16;" +"*(u64 *)(r10 - 8) = r7;" +"*(u64 *)(r10 - 16) = r6;" +"r1 = r10;" +"r1 += -8;" +"call inner_nop_fptest;" +"r1 = *(u64 *)(r10 - 8);" +"r0 = *(u64 *)(r1 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Reproducer for unsound pruning when a subprog forwards a parent +* stack pointer (AT_PARENT) to a helper with a memory argument. +* +* set_call_stack_access_at() previously only tracked AT_CURRENT args, +* skipping AT_PARENT entirely. This meant helper reads through parent +* stack pointers did not set parent_use, letting the slot appear dead +* at pruning checkpoints inside the subprog. +* +* Program shape: +* main: +* *(u32)(fp-4) = 0 key = STACK_ZERO (const 0) +* call bpf_get_prandom_u32 +* if r0 != 0 goto clobber path A (fall-through) first +* goto merge +* clobber: +* *(u8)(fp-4) = r0 path B: key[0] = STACK_MISC +* merge: +* r1 = fp - 4 +* call fwd_parent_key_to_helper +* r0 = 0 +* exit +* +* fwd_parent_key_to_helper(r1 = &caller_fp-4): +* goto +0 checkpoint +* r2 = r1 R2 = AT_PARENT ptr to caller fp-4 +* r1 = array_map_8b ll R1 = array map +* call bpf_map_lookup_elem reads key_size(4) from parent fp-4 +* r0 = *(u64 *)(r0 + 0) deref without null check +* r0 = 0 +* exit +* +* Path A: STACK_ZERO key = const 0 -> array lookup -> PTR_TO_MAP_VALUE +* (non-NULL for in-bounds const key) -> deref OK. +* Path B: STACK_MISC key = unknown -> array lookup -> +* PTR_TO_MAP_VALUE_OR_NULL -> deref UNSAFE. +* +* Bug: AT_PARENT R2 arg to bpf_map_lookup_elem skipped -> parent_use +* not set -> fp-4 cleaned at checkpoint -> STACK_ZERO collapses +* to STACK_INVALID -> path B pruned -> deref never checked. +* +* Correct verifier behavior: reject (path B deref of map_value_or_null). +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'map_value_or_null'") +__naked void helper_parent_stack_read_unsound (void ) +{ + asmvolatile ( +/* key at fp-4: all bytes STACK_ZERO */ +"*(u32 *)(r10 - 4) = 0;" +"call %[bpf_get_prandom_u32];" +/* fall-through (path A) explored first */ +"if r0 != 0 goto l_clobber%=;" +/* path A: key stays constant zero */ +"goto l_merge%=;" +"l_clobber%=:" +/* path B: key[0] becomes STACK_MISC, key no longer const */ +"*(u8 *)(r10 - 4) = r0;" +"l_merge%=:" +"r1 = r10;" +"r1 += -4;" +"call fwd_parent_key_to_helper;" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* +* Subprog forwards parent stack pointer to bpf_map_lookup_elem as key +* on an array map, then dereferences the result without a null check. +* R1 = &parent_fp-4 (AT_PARENT in this frame). +* +* The helper reads key_size(4) bytes from parent stack. The deref of +* R0 reads the map value, NOT parent stack, so record_insn_mem_accesses +* does not set parent_use for it. The ONLY parent stack access is +* through the helper's R2 arg. +*/ +static __used __naked void fwd_parent_key_to_helper (void ) +{ + asmvolatile ( +"goto +0;" /* checkpoint */ +"r2 = r1;" /* R2 = parent ptr (AT_PARENT) */ +"r1 = %[array_map_8b] ll;" /* R1 = array map */ +"call %[bpf_map_lookup_elem];" /* reads 4 bytes from parent fp-4 */ +/* deref without null check: safe for PTR_TO_MAP_VALUE, +* unsafe for PTR_TO_MAP_VALUE_OR_NULL +*/ +"r0 = *(u64 *)(r0 + 0);" +"r0 = 0;" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm_addr (array_map_8b) + : __clobber_all); +} + +/* +* Regression for keeping later helper args after a whole-stack fallback +* on an earlier local arg. The first bpf_snprintf() arg is a local +* frame-derived pointer with offset-imprecise tracking (`fp1 ?`), which +* conservatively marks the whole local stack live. The fourth arg still +* forwards &parent_fp-8 and must contribute nonlocal_use[0]=0:3. +*/ +SEC("socket") +__log_level(2) +__success +__msg("call bpf_snprintf{{.*}} ; use: fp1-8..-512 fp0-8") +__naked void helper_arg_fallback_keeps_scanning(void) +{ + asmvolatile ( +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"r1 = r10;" +"r1 += -8;" +"call helper_snprintf_parent_after_local_fallback;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void helper_snprintf_parent_after_local_fallback (void ) +{ + asmvolatile ( +"r6 = r1;" /* save &parent_fp-8 */ +"call %[bpf_get_prandom_u32];" +"r0 &= 8;" +"r1 = r10;" +"r1 += -16;" +"r1 += r0;" /* local fp, offset-imprecise */ +"r2 = 8;" +"r3 = %[snprintf_u64_fmt] ll;" +"r4 = r6;" /* later arg: parent fp-8 */ +"r5 = 8;" +"call %[bpf_snprintf];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_prandom_u32), +__imm (bpf_snprintf), +__imm_addr (snprintf_u64_fmt) + : __clobber_all); +} + +/* +* Test that propagate_callee_ancestor() correctly chains ancestor +* liveness across sequential calls within a single frame. +* +* main → mid_seq_touch → {nop_callee, deref_ancestor} +* +* mid_seq_touch receives two pointers: R1 = &main_fp-8 (forwarded to +* deref_ancestor) and R2 = &main_fp-16 (read directly by mid_seq_touch). +* The direct read of fp-16 forces ensure_anc_arrays() to allocate +* ancestor_live[0] for mid_seq_touch, so refined_caller_live_stack() +* uses the refined path (not the conservative fallback). +* +* mid_seq_touch calls nop_callee first (no-op, creates a pruning point), +* then calls deref_ancestor which reads main's fp-8 and dereferences it. +* +* propagate_callee_ancestor() propagates deref_ancestor's entry +* ancestor_live[0] into mid_seq_touch's anc_use[0] at the call-to-deref +* instruction. mid_seq_touch's backward pass flows this backward so +* ancestor_live[0] includes fp-8 at the pruning point between the calls. +* +* Without propagation, mid_seq_touch's ancestor_live[0] only has fp-16 +* (from the direct read) — fp-8 is missing. refined_caller_live_stack() +* Term 1 says fp-8 is dead, the verifier cleans it, and path B +* (scalar 42) is incorrectly pruned against path A (MAP_VALUE). +* +* Path A: main stores PTR_TO_MAP_VALUE at fp-8 → deref succeeds +* Path B: main stores scalar 42 at fp-8 → deref must fail +* +* Correct: reject (path B dereferences scalar) +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__failure __msg("R0 invalid mem access 'scalar'") +__naked void propagate_callee_ancestor_chain (void ) +{ + asmvolatile ( +/* Map lookup to get PTR_TO_MAP_VALUE */ +"r1 = %[map] ll;" +"*(u32 *)(r10 - 32) = 0;" +"r2 = r10;" +"r2 += -32;" +"call %[bpf_map_lookup_elem];" +"if r0 == 0 goto l_exit%=;" +"r6 = r0;" +/* Branch: path A (fall-through) explored first */ +"call %[bpf_get_prandom_u32];" +"if r0 != 0 goto l_scalar%=;" +/* Path A: fp-8 = PTR_TO_MAP_VALUE */ +"*(u64 *)(r10 - 8) = r6;" +"goto l_merge%=;" +"l_scalar%=:" +/* Path B: fp-8 = scalar 42 */ +"r1 = 42;" +"*(u64 *)(r10 - 8) = r1;" +"l_merge%=:" +/* fp-16 = dummy value (mid_seq_touch reads it directly) */ +"r1 = 99;" +"*(u64 *)(r10 - 16) = r1;" +/* R1 = &fp-8 (for deref_ancestor), R2 = &fp-16 (for mid_seq_touch) */ +"r1 = r10;" +"r1 += -8;" +"r2 = r10;" +"r2 += -16;" +"call mid_seq_touch;" +"r0 = 0;" +"exit;" +"l_exit%=:" +"r0 = 0;" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm (bpf_get_prandom_u32), +__imm_addr (map) + : __clobber_all); +} + +/* +* R1 = &main_fp-8 (forwarded to deref_ancestor) +* R2 = &main_fp-16 (read directly here → allocates ancestor_live[0]) +* +* Reads main's fp-16 to force ancestor_live[0] allocation, then +* calls nop_callee (pruning point), then deref_ancestor. +*/ +static __used __naked void mid_seq_touch (void ) +{ + asmvolatile ( +"r6 = r1;" /* save &main_fp-8 in callee-saved */ +"r0 = *(u64 *)(r2 + 0);" /* read main's fp-16: triggers anc_use[0] */ +"call nop_callee;" /* no-op, creates pruning point after */ +"r1 = r6;" /* restore ptr to &main_fp-8 */ +"call deref_ancestor;" /* reads main's fp-8, dereferences */ +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void nop_callee (void ) +{ + asmvolatile ( +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Reads main's fp-8 through forwarded pointer, dereferences result */ +static __used __naked void deref_ancestor (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" /* read main's fp-8 */ +"r0 = *(u64 *)(r0 + 0);" /* deref: safe for map_ptr, unsafe for scalar */ +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Test: callee loads an fp-derived pointer from caller's stack, then +* reads through it to access another caller stack slot. +* +* main stores PTR_TO_MAP_VALUE at fp-24, stores &fp-24 (an fp-derived +* pointer) at fp-8, passes &fp-8 through mid_fwd_spilled_ptr to +* load_ptr_deref_grandchild. The leaf loads the pointer from main's +* fp-8, then reads main's fp-24 through the loaded pointer. +* +* fill_from_stack() in arg_track_xfer() only handles local-frame +* FP-derived loads (src_is_local_fp check requires frame == depth). +* When a callee loads from a parent-frame pointer (frame < depth), +* the loaded value gets ARG_NONE instead of being recognized as +* fp-derived. Subsequent reads through that loaded pointer are +* invisible to liveness — nonlocal_use is never set for fp-24. +* +* clean_live_states() cleans the current state at every prune point. +* Because liveness misses fp-24, refined_caller_live_stack() tells +* __clean_func_state() that fp-24 is dead, which destroys the +* PTR_TO_MAP_VALUE spill before the grandchild can read it. +* The grandchild then reads STACK_INVALID → scalar, and the deref +* is rejected with "R0 invalid mem access 'scalar'" — even though +* fp-24 is genuinely live and holds a valid map pointer. +* +* This is a false positive: a valid program incorrectly rejected. +*/ +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__success +__naked void spilled_fp_cross_frame_deref (void ) +{ + asmvolatile ( +/* Map lookup to get PTR_TO_MAP_VALUE */ +"r1 = %[map] ll;" +"*(u32 *)(r10 - 32) = 0;" +"r2 = r10;" +"r2 += -32;" +"call %[bpf_map_lookup_elem];" +"if r0 == 0 goto l_exit%=;" +/* fp-24 = PTR_TO_MAP_VALUE */ +"*(u64 *)(r10 - 24) = r0;" +/* Store pointer to fp-24 at fp-8 */ +"r1 = r10;" +"r1 += -24;" +"*(u64 *)(r10 - 8) = r1;" +/* R1 = &fp-8: pointer to the spilled ptr */ +"r1 = r10;" +"r1 += -8;" +"call mid_fwd_spilled_ptr;" +"r0 = 0;" +"exit;" +"l_exit%=:" +"r0 = 0;" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm_addr (map) + : __clobber_all); +} + +/* Forwards R1 (ptr to main's fp-8, which holds &main_fp-24) to leaf */ +static __used __naked void mid_fwd_spilled_ptr (void ) +{ + asmvolatile ( +"call load_ptr_deref_grandchild;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* R1 = &main_fp-8 (where main stored ptr to fp-24) +* Loads the ptr from main's fp-8, reads main's fp-24 through it, +* then dereferences the result. +*/ +static __used __naked void load_ptr_deref_grandchild (void ) +{ + asmvolatile ( +/* Load ptr from main's fp-8 → r2 = &main_fp-24 */ +"r2 = *(u64 *)(r1 + 0);" +/* Read main's fp-24 through loaded ptr */ +"r0 = *(u64 *)(r2 + 0);" +/* Dereference: safe for map_ptr */ +"r0 = *(u64 *)(r0 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Exercise merge_nonlocal_live(). +* +* merge_shared_mid is analyzed twice (once from each wrapper), so the +* callsite within merge_shared_mid that calls merge_leaf_read gets its +* nonlocal_live info merged twice via merge_nonlocal_live(). +*/ +SEC("socket") +__log_level(2) +__success +__msg("14: (85) call pc+2 r1: fp0-16") +__msg("17: (79) r0 = *(u64 *)(r1 +0) // r1=fp0-16") +__msg("14: (85) call pc+2 r1: fp0-8") +__msg("17: (79) r0 = *(u64 *)(r1 +0) // r1=fp0-8") +__msg("5: (85) call pc+{{.*}} ; use: fp0-8 fp0-16") +__naked void test_merge_nonlocal_live (void ) +{ + asmvolatile ( +"r1 = 0;" +"*(u64 *)(r10 - 8) = r1;" +"*(u64 *)(r10 - 16) = r1;" +"r1 = r10;" +"r1 += -8;" +"call merge_wrapper_a;" +"r1 = r10;" +"r1 += -16;" +"call merge_wrapper_b;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void merge_wrapper_a (void ) +{ + asmvolatile ( +"call merge_shared_mid;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void merge_wrapper_b (void ) +{ + asmvolatile ( +"call merge_shared_mid;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void merge_shared_mid (void ) +{ + asmvolatile ( +"call merge_leaf_read;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void merge_leaf_read (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* Same bpf_loop instruction calls different callbacks depending on branch. */ +SEC("socket") +__log_level(2) +__success +__msg("call bpf_loop#181 ; use: fp2-8..-512 fp1-8..-512 fp0-8..-512") +__naked void bpf_loop_two_callbacks(void) +{ + asmvolatile ( +"r1 = 0;" +"*(u64 *)(r10 - 8) = r1;" +"*(u64 *)(r10 - 16) = r1;" +"r1 = r10;" +"r1 += -8;" +"call dyn_wrapper_a;" +"r1 = r10;" +"r1 += -16;" +"call dyn_wrapper_b;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void dyn_wrapper_a (void ) +{ + asmvolatile ( +"call mid_dynamic_cb;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void dyn_wrapper_b (void ) +{ + asmvolatile ( +"call mid_dynamic_cb;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void mid_dynamic_cb (void ) +{ + asmvolatile ( +"r6 = r1;" +"call %[bpf_get_prandom_u32];" +"if r0 == 0 goto 1f;" +"r2 = dyn_cb_a ll;" +"goto 2f;" +"1:" +"r2 = dyn_cb_b ll;" +"2:" +"r1 = 1;" +"r3 = r6;" /* ctx = fp-derived ptr from parent */ +"r4 = 0;" +"call %[bpf_loop];" +"r0 = 0;" +"exit;" + :: __imm(bpf_get_prandom_u32), +__imm (bpf_loop) + : __clobber_all); +} + +/* Callback A/B: read parent stack through ctx */ +static __used __naked void dyn_cb_a (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r2 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void dyn_cb_b (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r2 + 0);" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +/* +* Path A: r0 = map_lookup result (non-FP, ARG_NONE for stack tracking) +* Path B: r0 = fp-8 (FP-derived, frame=0, off=-8) +* At the join: r0 is not guaranteed to be a frame pointer. +*/ +SEC("socket") +__log_level(2) +__msg("10: (79) r0 = *(u64 *)(r10 -8) // r0=fp0-8|fp0+0") +__naked void stack_or_non_stack_write (void ) +{ + asmvolatile ( +/* initial write to fp-8 */ +"*(u64 *)(r10 - 8) = 0;" +/* map lookup to get a non-FP pointer */ +"r2 = r10;" +"r2 += -4;" +"r1 = %[map] ll;" +"call %[bpf_map_lookup_elem];" +/* r0 = map_value (ARG_NONE) */ +"if r0 != 0 goto 1f;" +/* path B: r0 = fp-8 */ +"r0 = r10;" +"r0 += -8;" +"1:" +/* join: the write is not a def for fp[0]-8 */ +"*(u64 *)(r0 + 0) = 7;" +/* read fp-8: should be non-poisoned */ +"r0 = *(u64 *)(r10 - 8);" +"exit;" + : + : __imm(bpf_map_lookup_elem), +__imm_addr (map) + : __clobber_all); +} + +SEC("socket") +__log_level(2) +__flag(BPF_F_TEST_STATE_FREQ) +__msg("subprog#2 write_first_read_second:") +__msg("17: (7a) *(u64 *)(r1 +0) = 42{{$}}") +__msg("18: (79) r0 = *(u64 *)(r2 +0) // r1=fp0-8 r2=fp0-16{{$}}") +__msg("stack use/def subprog#2 write_first_read_second (d2,cs15):") +__msg("17: (7a) *(u64 *)(r1 +0) = 42{{$}}") +__msg("18: (79) r0 = *(u64 *)(r2 +0) ; use: fp0-8 fp0-16") +__naked void shared_instance_must_write_overwrite (void ) +{ + asmvolatile ( +"r1 = 1;" +"*(u64 *)(r10 - 8) = r1;" +"*(u64 *)(r10 - 16) = r1;" +/* Call 1: write_first_read_second(&fp[-8], &fp[-16]) */ +"r1 = r10;" +"r1 += -8;" +"r2 = r10;" +"r2 += -16;" +"call forwarding_rw;" +/* Call 2: write_first_read_second(&fp[-16], &fp[-8]) */ +"r1 = r10;" +"r1 += -16;" +"r2 = r10;" +"r2 += -8;" +"call forwarding_rw;" +"r0 = 0;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void forwarding_rw (void ) +{ + asmvolatile ( +"call write_first_read_second;" +"exit;" + ::: __clobber_all); +} + +static __used __naked void write_first_read_second (void ) +{ + asmvolatile ( +"*(u64 *)(r1 + 0) = 42;" +"r0 = *(u64 *)(r2 + 0);" +"exit;" + ::: __clobber_all); +} + +#ifdef CAN_USE_LOAD_ACQ_STORE_REL + +SEC("socket") +__log_level(2) +__success +__msg("*(u64 *)(r0 +0) = 42 ; def: fp0-16") +__naked void load_acquire_dont_clear_dst(void) +{ + asmvolatile ( +"r0 = r10;" +"r0 += -16;" +"*(u64 *)(r0 + 0) = r0;" /* fp[-16] == &fp[-16] */ +".8byte %[load_acquire_insn];" /* load_acquire is a special case for BPF_STX, */ +"r0 = *(u64 *)(r10 - 16);" /* it shouldn't clear tracking info for */ +"*(u64 *)(r0 + 0) = 42;" /* dst register, r0 in this case. */ +"r0 = 0;" +"exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_0, 0)) + : __clobber_all); +} + +#endif /* CAN_USE_LOAD_ACQ_STORE_REL */
Signature · 1 line
2.53.0
Eduard Zingerman
maintainerPATCH v2 13/13 bpf: poison dead stack slots
In reply to Eduard ZingermanFrom: Alexei Starovoitov <[email protected]>
As a sanity check poison stack slots that stack liveness determined to be dead, so that any read from such slots will cause program rejection. If stack liveness logic is incorrect the poison can cause valid program to be rejected, but it also will prevent unsafe program to be accepted.
Allow global subprogs "read" poisoned stack slots.
The static stack liveness determined that subprog doesn't read certain stack slots, but sizeof(arg_type) based global subprog validation isn't accurate enough to know which slots will actually be read by the callee, so it needs to check full sizeof(arg_type) at the caller.
--- include/linux/bpf_verifier.h | 1 + kernel/bpf/log.c | 5 +- kernel/bpf/verifier.c | 80 ++++++++++++++++------ .../selftests/bpf/progs/verifier_spill_fill.c | 2 + 4 files changed, 64 insertions(+), 24 deletions(-)
include/linux/bpf_verifier.h +1 -0
@@ -220,6 +220,7 @@ enum bpf_stack_slot_type {STACK_DYNPTR ,STACK_ITER ,STACK_IRQ_FLAG , + STACK_POISON, };#define BPF_REG_SIZE 8/* size of eBPF register in bytes */
kernel/bpf/log.c +3 -2
@@ -542,7 +542,8 @@ static char slot_type_char[] = { [STACK_ZERO] = '0', [STACK_DYNPTR] = 'd', [STACK_ITER] = 'i', - [STACK_IRQ_FLAG] = 'f' + [STACK_IRQ_FLAG] = 'f', + [STACK_POISON] = 'p', };#define UNUM_MAX_DECIMAL U16_MAX
@@ -779,7 +780,7 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifiefor (j =0 ; j <BPF_REG_SIZE ; j++) { slot_type = state->stack[i].slot_type[j]; -if (slot_type !=STACK_INVALID ) +if (slot_type !=STACK_INVALID && slot_type !=STACK_POISON ) valid = true; types_buf[j] = slot_type_char[slot_type]; }
kernel/bpf/verifier.c +58 -22
165 lines not on this page · raw message
tools/testing/selftests/bpf/progs/verifier_spill_fill.c +2 -0
@@ -780,6 +780,8 @@ __naked void stack_load_preserves_const_precision_subreg(void)"r1 += r2;" "*(u8 *)(r1 + 0) = r2;" /* this should be fine */ +"r2 = *(u64 *)(r10 -8);" /* keep slots alive */ +"r2 = *(u64 *)(r10 -16);" "r0 = 0;" "exit;" :
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v2 13/13] bpf: poison dead stack slots
The function header comment for mark_stack_slot_misc() says:
Quoted 4 lines
kernel/bpf/verifier.c +0
-0
[ ... ]
kernel/bpf/verifier.c static void mark_stack_slot_misc(struct bpf_verifier_env *env, u8 *stype) · line 1397
Quoted 9 lines
kernel/bpf/verifier.c +1
-1@@ -1390,14 +1391,14 @@ static void mark_stack_slot_misc(struct bpf_verifier_env *env, u8 *stype) {if (*stype ==STACK_ZERO )return ; -if (*stype ==STACK_INVALID ) +if (*stype ==STACK_INVALID || *stype ==STACK_POISON )return ; *stype =STACK_MISC ; }
The function header comment for mark_stack_slot_misc() says:
Mark stack slot as STACK_MISC, unless it is already STACK_INVALID, in which case they are equivalent ... we won't promote STACK_INVALID to STACK_MISC. In privileged case it is unnecessary as both are considered equivalent when loading data and pruning
Should this comment mention STACK_POISON as well? Unlike STACK_INVALID, STACK_POISON is not equivalent to STACK_MISC -- it is deliberately more restrictive (reads are rejected even in privileged mode). The existing reasoning about equivalence doesn't extend to the new type, so it might be worth a brief note explaining why STACK_POISON is also preserved here.
[ ... ]
kernel/bpf/verifier.c static int check_stack_write_var_off(struct bpf_verifier_env *env, · line 5593
Quoted 7 lines
kernel/bpf/verifier.c +2
-1@@ -5588,7 +5589,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,* that may or may not be written because, if we're reject * them, the error would be too confusing. */ -if (*stype ==STACK_INVALID && !env->allow_uninit_stack) { +if ((*stype ==STACK_INVALID || *stype ==STACK_POISON ) && + !env->allow_uninit_stack) {
The comment just above says "If the slot is STACK_INVALID" but the condition now also covers STACK_POISON. Should the comment be updated to mention it?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24221779050