bpf: infer zext_dst based on static register liveness analysis
Subsystems (5)
Eduard Zingerman
maintainerv4 cover bpf: infer zext_dst based on static register liveness analysis
Min-gyu Kim reported a bug in 32-bit operations zero extension handling [1]. The same issue was independently identified by STAR Labs SG. The bug was introduced by the commit [2]. The tl;dr of the bug is that the verifier does not carry zero extension marks across pruning points. The detailed mechanism is described in patch #3.
Fix this by reworking zero extension logic to avoid main-path based subreg_def tracking, and instead extend the live registers analysis to track upper register halves' liveness.
As noted in the commit message for patch #3:
There is one notable drop in precision: whenever a BPF subprogram is called, all 64 bits of parameter registers are presumed to be used. The assumption is that such a drop in precision would not inflict noticeable performance penalty.
Ilya, could you please help with testing this conjecture? Min-gyu, could you please test the proposed fix against your reproducer?
Side note:
Patch #1 is a small refactoring for disasm.c, as patch #3 adds a new location where bpf_verbose_insn() is called and there as well I have to wrangle with the newline added by the disasm code.
[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness")
Changelog:
v1 -> v2:
- fixed BPF_JMP32 handling, these no longer for zero extension (sashiko)
- removed dead code in print_insn_for_graph() (bot+bpf-ci)
- reworked bpf_is_reg64() to drop dead code (sashiko, bot+bpf-ci)
v2 -> v3:
- fix for proper s390x calling convention all call parameters are now zero extended if necessary (sashiko)
- fix for PTR_TO_ARENA alu operations to set zext_dst when needs_zext is set, in order for such operations to be zero extended (sashiko)
- updated bpf_is_reg64() to handle BPR_PROBE_MEM{,32} loads in order to avoid a false positive from sashiko.
v3 -> v4:
- fix to properly handle address space cast instructions for arena maps with BPF_F_NO_USER_CONV (sashiko):
- extracted is_addr_space_cast32() utility function,
for use in bpf_do_misc_fixups() and bpf_is_reg64();
- made bpf_is_reg64() aware of such address space casts.
- added a note about report from STAR Labs SG.
v1: https://lore.kernel.org/bpf/[email protected]/T/
v2: https://lore.kernel.org/bpf/[email protected]/T/
v3: https://lore.kernel.org/bpf/[email protected]/T/
---
Eduard Zingerman (7):
bpf: do not print a newline after disassembly in bpf_verbose_insn()
bpf: extract is_addr_space_cast32() utility function
bpf: move bpf_is_reg64() to fixups.c
bpf: track upper 32-bit register halves' liveness in compute_live_registers()
bpf: infer zext_dst based on static register liveness analysis
bpf: simplify the bpf_is_reg64()
selftests/bpf: verify zext_dst annotations for various instructions
include/linux/bpf_verifier.h | 7 +- kernel/bpf/backtrack.c | 1 + kernel/bpf/disasm.c | 68 ++-- kernel/bpf/fixups.c | 100 ++++-- kernel/bpf/liveness.c | 109 ++++-- kernel/bpf/verifier.c | 204 +---------- tools/bpf/bpftool/xlated_dumper.c | 19 +- tools/testing/selftests/bpf/disasm_helpers.c | 3 +- tools/testing/selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/verifier_zext.c | 392 ++++++++++++++++++++++ 10 files changed, 596 insertions(+), 309 deletions(-)
---
base-commit: 41c129fdc28b6414d259da72679567c5e72a55dd
change-id: 20260731-static-zext-938cd128273c
Eduard Zingerman
maintainerPATCH v4 1/7 bpf: do not print a newline after disassembly in bpf_verbose_insn()
At the moment there are more callsites that want bpf_verbose_insn() to not print a newline after the instruction, than callsites that want a newline. Drop '\n' from disasm.c. Non-functional change.
The changes in bpftool are verified by writing a bpf program using a variety of instructions and comparing prog dump xlated output in the following modes: plain, opcodes, visual, visual opcodes. The output before and after the changes is identical.
--- kernel/bpf/backtrack.c | 1 + kernel/bpf/disasm.c | 68 ++++++++++++++-------------- kernel/bpf/liveness.c | 6 +-- kernel/bpf/verifier.c | 1 + tools/bpf/bpftool/xlated_dumper.c | 19 ++------ tools/testing/selftests/bpf/disasm_helpers.c | 3 +- 6 files changed, 43 insertions(+), 55 deletions(-)
kernel/bpf/backtrack.c +1 -0
@@ -285,6 +285,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,verbose (env,"stack=%s before " , env->tmp_str_buf);verbose (env,"%d: " , idx);bpf_verbose_insn (env, insn); +verbose (env,"\n" ); }/* If there is a history record that some registers gained range at this insn,
kernel/bpf/disasm.c +34 -34
219 lines not on this page · raw message
kernel/bpf/liveness.c +2 -4
@@ -497,7 +497,6 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i pos = env->log.end_pos;verbose (env,"%3d: " , insn_idx);bpf_verbose_insn (env, &insns[insn_idx]); -bpf_vlog_reset (&env->log, env->log.end_pos -1 );/* remove \n */ insn_pos = env->log.end_pos;verbose (env,"%*c;" ,bpf_vlog_alignment (insn_pos - pos),' ' ); pos = env->log.end_pos;
@@ -1043,7 +1042,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, iif (!printed) {verbose (env,"%3d: " , idx);bpf_verbose_insn (env, insn); -bpf_vlog_reset (&env->log, env->log.end_pos -1 ); printed = true; }verbose (env,"\tr%d: " , i);verbose_arg_track (env, &at_in[i]);
@@ -1058,7 +1056,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, iif (!printed) {verbose (env,"%3d: " , idx);bpf_verbose_insn (env, insn); -bpf_vlog_reset (&env->log, env->log.end_pos -1 ); printed = true; }verbose (env,"\tsa%d: " , i);verbose_arg_track (env, &at_in[ai]);
@@ -1070,7 +1067,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, iif (!printed) {verbose (env,"%3d: " , idx);bpf_verbose_insn (env, insn); -bpf_vlog_reset (&env->log, env->log.end_pos -1 ); printed = true; }verbose (env,"\tfp%+d: " , -(i +1 ) *8 );verbose_arg_track (env, &at_stack_in[i]);
@@ -1545,6 +1541,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,verbose (env,"%3d: " , idx);bpf_verbose_insn (env, &insns[idx]); +verbose (env,"\n" );/* Collect what needs printing */ if (is_ldx_stx_call &&
@@ -2285,6 +2282,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)verbose (env,"." );verbose (env," " );bpf_verbose_insn (env, &insns[i]); +verbose (env,"\n" );if (bpf_is_ldimm64 (&insns[i])) i++; }
kernel/bpf/verifier.c +1 -0
@@ -17538,6 +17538,7 @@ static int do_check(struct bpf_verifier_env *env) env->prev_log_pos = env->log.end_pos;verbose (env,"%d: " , env->insn_idx);bpf_verbose_insn (env, insn); +verbose (env,"\n" ); env->prev_insn_print_pos = env->log.end_pos - env->prev_log_pos; env->prev_log_pos = env->log.end_pos; }
tools/bpf/bpftool/xlated_dumper.c +4 -15
@@ -107,14 +107,7 @@ print_insn_for_graph(void *private_data, const char *fmt, ...) p = buf;while (*p !='\0' ) { -if (*p =='\n' ) { -memmove (p +3 , p,strlen (buf) +1 - (p - buf)); -/* Align each instruction dump row left. */ - *p++ ='\\' ; - *p++ ='l' ; -/* Output multiline concatenation. */ - *p++ ='\\' ; - }else if (*p =='<' || *p =='>' || *p =='|' || *p =='&' ) { +if (*p =='<' || *p =='>' || *p =='|' || *p =='&' ) {memmove (p +1 , p,strlen (buf) +1 - (p - buf));/* Escape special character. */ *p++ ='\\' ;
@@ -129,16 +122,10 @@ print_insn_for_graph(void *private_data, const char *fmt, ...) static void __printf(2, 3) print_insn_json(void *private_data, const char *fmt, ...) { -unsigned int l =strlen (fmt); -char chomped_fmt[l];va_list args;va_start (args, fmt); -if (l >0 ) { -strncpy (chomped_fmt, fmt, l -1 ); - chomped_fmt[l -1 ] ='\0' ; - } -jsonw_vprintf_enquote (json_wtr, chomped_fmt, args); +jsonw_vprintf_enquote (json_wtr, fmt, args);va_end (args); }
@@ -351,6 +338,7 @@ void dump_xlated_plain(struct dump_data *dd, void *buf, unsigned int len,printf ("%4u: " , i);print_bpf_insn (&cbs, insn + i, true); +printf ("\n" );if (opcodes) {printf (" " );
@@ -417,6 +405,7 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,printf ("%u: " , insn_off);print_bpf_insn (&cbs, cur, true); +printf ("\\l\\\n" );if (opcodes) {printf ("\\ \\ \\ \\ " );
tools/testing/selftests/bpf/disasm_helpers.c +1 -2
@@ -55,10 +55,9 @@ struct bpf_insn *disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz)* for each instruction (FF stands for instruction `code` byte). * Remove the prefix inplace, and also simplify call instructions. * E.g.: "(85) call foo#10" -> "call foo". -* Also remove newline in the end (the 'max(strlen(buf) - 1, 0)' thing). */ pfx_end = buf +5 ; - sfx_start = buf +max ((int )strlen (buf) -1 ,0 ); + sfx_start = buf + (int )strlen (buf);if (strncmp (pfx_end,"call " ,5 ) ==0 && (tmp =strrchr (buf,'#' ))) sfx_start = tmp; len = sfx_start - pfx_end;
Signature · 1 line
2.55.0
Eduard Zingerman
maintainerPATCH v4 2/7 bpf: extract is_addr_space_cast32() utility function
In reply to Eduard Zingermanbpf_do_misc_fixups() converts the following address space cast instructions to 32-bit moves:
- cast from address space 1 (user) to address space 0 (kernel)
- cast from address space 0 (kernel) to address space 1 (user) iff associated arena map has a BPF_F_NO_USER_CONV flag.
Extract a predicate detecting such instructions for use in the following patches.
--- kernel/bpf/fixups.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-)
kernel/bpf/fixups.c +26 -9
@@ -20,6 +20,26 @@ static bool is_cmpxchg_insn(const struct bpf_insn *insn) insn->imm ==BPF_CMPXCHG ; } +/* Returns true if 'insn' is an address space cast instruction translated as BPF_ALU op */ +static bool is_addr_space_cast32 (struct bpf_prog *prog,const struct bpf_insn *insn) +{ +struct bpf_map *arena = (struct bpf_map *)prog->aux->arena; + +if (insn->code != (BPF_ALU64 |BPF_MOV |BPF_X ) || insn->off !=BPF_ADDR_SPACE_CAST ) +return false; + +/* cast from as(1) to as(0) */ +if (insn->imm ==1 ) +return true; + +/* cast from as(0) to as(1) */ +if (insn->imm ==1 <<16 ) +return arena && arena->map_flags &BPF_F_NO_USER_CONV ; + +/* non-BPF_F_NO_USER_CONV cast from as(0) to as(1) should be handled by JIT */ +return false; +} +/* Return the regno defined by the insn, or -1. */ static int insn_def_regno (const struct bpf_insn *insn) {
@@ -1513,15 +1533,12 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env) }for (i =0 ; i < insn_cnt;) { -if (insn->code == (BPF_ALU64 |BPF_MOV |BPF_X ) && insn->imm) { -if ((insn->off ==BPF_ADDR_SPACE_CAST && insn->imm ==1 ) || - (((struct bpf_map *)env->prog->aux->arena)->map_flags &BPF_F_NO_USER_CONV )) { -/* convert to 32-bit mov that clears upper 32-bit */ - insn->code =BPF_ALU |BPF_MOV |BPF_X ; -/* clear off and imm, so it's a normal 'wX = wY' from JIT pov */ - insn->off =0 ; - insn->imm =0 ; - }/* cast from as(0) to as(1) should be handled by JIT */ +if (is_addr_space_cast32 (env->prog, insn)) { +/* convert to 32-bit mov that clears upper 32-bit */ + insn->code =BPF_ALU |BPF_MOV |BPF_X ; +/* clear off and imm, so it's a normal 'wX = wY' from JIT pov */ + insn->off =0 ; + insn->imm =0 ; goto next_insn; }
Signature · 1 line
2.55.0
Eduard Zingerman
maintainerPATCH v4 3/7 bpf: move bpf_is_reg64() to fixups.c
In reply to Eduard ZingermanThe following patches are going to remove bpf_is_reg64() users from everywhere except fixups.c, and also make it dependent on functions local to fixups.c. Move the function before hand to simplify the review. Non functional change.
--- kernel/bpf/fixups.c | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 90 --------------------------------------------------- 2 files changed, 90 insertions(+), 90 deletions(-)
kernel/bpf/fixups.c +90 -0
@@ -64,6 +64,96 @@ static int insn_def_regno(const struct bpf_insn *insn) } } +/* This function is supposed to be used by the zero extension optimization +* code only. It returns TRUE if the source or destination register operates +* on 64-bit, otherwise return FALSE. +*/ +bool bpf_is_reg64 (struct bpf_insn *insn, +u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t) +{ +u8 code, class, op; + + code = insn->code; + class =BPF_CLASS (code); + op =BPF_OP (code); +if (class ==BPF_JMP ) { +/* BPF_EXIT for "main" will reach here. Return TRUE +* conservatively. +*/ +if (op ==BPF_EXIT ) +return true; +if (op ==BPF_CALL ) { +/* BPF to BPF call will reach here because of marking +* caller saved clobber with DST_OP_NO_MARK for which we +* don't care the register def because they are anyway +* marked as NOT_INIT already. +*/ +if (insn->src_reg ==BPF_PSEUDO_CALL ) +return false; +/* Helper call will reach here because of arg type +* check, conservatively return TRUE. +*/ +if (t ==SRC_OP ) +return true; + +return false; + } + } + +if (class ==BPF_ALU64 && op ==BPF_END && (insn->imm ==16 || insn->imm ==32 )) +return false; + +if (class ==BPF_ALU64 || class ==BPF_JMP || + (class ==BPF_ALU && op ==BPF_END && insn->imm ==64 )) +return true; + +if (class ==BPF_ALU || class ==BPF_JMP32 ) +return false; + +if (class ==BPF_LDX ) { +if (t !=SRC_OP ) +return BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX ; +/* LDX source must be ptr. */ +return true; + } + +if (class ==BPF_STX ) { +/* BPF_STX (including atomic variants) has one or more source +* operands, one of which is a ptr. Check whether the caller is +* asking about it. +*/ +if (t ==SRC_OP && reg->type !=SCALAR_VALUE ) +return true; +return BPF_SIZE (code) ==BPF_DW ; + } + +if (class ==BPF_LD ) { +u8 mode =BPF_MODE (code); + +/* LD_IMM64 */ +if (mode ==BPF_IMM ) +return true; + +/* Both LD_IND and LD_ABS return 32-bit data. */ +if (t !=SRC_OP ) +return false; + +/* Implicit ctx ptr. */ +if (regno ==BPF_REG_6 ) +return true; + +/* Explicit source could be any width. */ +return true; + } + +if (class ==BPF_ST ) +/* The only source register for BPF_ST is a ptr. */ +return true; + +/* Conservatively return true at default. */ +return true; +} +/* Return TRUE if INSN has defined any 32-bit value explicitly. */ static bool insn_has_def32 (struct bpf_insn *insn) {
kernel/bpf/verifier.c +0 -90
96 lines not on this page · raw message
Signature · 1 line
2.55.0
[email protected]
AI reviewRe: [PATCH bpf-next v4 3/7] bpf: move bpf_is_reg64() to fixups.c
A subsystem pattern flags this as potentially concerning: the BPF subsystem guide requires multi-line comments in kernel/bpf/ to follow the modern style with the opening /* on its own line. This…
kernel/bpf/fixups.c static int insn_def_regno(const struct bpf_insn *insn) · line 70
Quoted 12 lines
kernel/bpf/fixups.c +4
-0@@ -64,6 +64,96 @@ static int insn_def_regno(const struct bpf_insn *insn) } } +/* This function is supposed to be used by the zero extension optimization +* code only. It returns TRUE if the source or destination register operates +* on 64-bit, otherwise return FALSE. +*/
A subsystem pattern flags this as potentially concerning: the BPF subsystem guide requires multi-line comments in kernel/bpf/ to follow the modern style with the opening /* on its own line. This comment puts text on the same line as the opening delimiter.
The guide states this applies even when surrounding code uses the old style, and this comment's first line is modified by this patch ('the following 32-bit optimization code only' -> 'the zero extension optimization code only'), so it is a modified line rather than untouched pre-existing text.
I noticed a later commit in the series (87fabf17dd53) rewrites this comment into the compliant form, so the deviation is transient within the series. Is it worth adjusting now, or acceptable to fix in the later patch?
Quoted 3 lines
kernel/bpf/fixups.c +3
-0+bool bpf_is_reg64 (struct bpf_insn *insn, +u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t) +{
[ ... ]
Quoted 5 lines
kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static void mark_stack_slots_scratched(struct bpf_verifier_env *env, · line 3067
Quoted 10 lines
kernel/bpf/verifier.c +0
-7@@ -3058,96 +3058,6 @@ static void mark_stack_slots_scratched(struct bpf_verifier_env *env,mark_stack_slot_scratched (env, spi - i); } -/* This function is supposed to be used by the following 32-bit optimization -* code only. It returns TRUE if the source or destination register operates -* on 64-bit, otherwise return FALSE. -*/ -bool bpf_is_reg64 (struct bpf_insn *insn, -u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t) -{
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31219172145
Eduard Zingerman
maintainerPATCH v4 4/7 bpf: track upper 32-bit register halves' liveness in compute_live_registers()
In reply to Eduard ZingermanExtend compute_live_registers() to track upper and lower register halves' liveness separately. This is mostly straightforward:
- use/def masks are extended to track 2 bits per register;
- compute_insn_live_regs() is updated to properly track these 2 bits according to the instruction semantics.
--- kernel/bpf/liveness.c | 89 ++++++++++++++++++++++++++++++++++----------------- kernel/bpf/verifier.c | 1 - 2 files changed, 59 insertions(+), 31 deletions(-)
kernel/bpf/liveness.c +59 -30
@@ -2047,29 +2047,38 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)/* Each field is a register bitmask */ struct insn_live_regs { -u16 use;/* registers read by instruction */ -u16 def;/* registers written by instruction */ -u16 in;/* registers that may be alive before instruction */ -u16 out;/* registers that may be alive after instruction */ +u32 use;/* registers read by instruction */ +u32 def;/* registers written by instruction */ +u32 in;/* registers that may be alive before instruction */ +u32 out;/* registers that may be alive after instruction */ };/* Bitmask with 1s for all caller saved registers */ #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1) +static inline u32 reg32_mask (u32 n) {return BIT (n); } +static inline u32 reg64_mask (u32 n) {return BIT (n) |BIT (n +16 ); } +static inline u32 mask_widen (u32 m) {return m | (m <<16 ); } +static inline u16 mask_lo (u32 m) {return (u16 )m; } +static inline u16 mask_hi (u32 m) {return (u16)(m >>16 ); } +/* Compute info->{use,def} fields for the instruction */ static void compute_insn_live_regs (struct bpf_verifier_env *env,struct bpf_insn *insn,struct insn_live_regs *info) {struct bpf_call_summary cs; -u8 class =BPF_CLASS (insn->code); -u8 code =BPF_OP (insn->code); -u8 mode =BPF_MODE (insn->code); -u16 src =BIT (insn->src_reg); -u16 dst =BIT (insn->dst_reg); -u16 r0 =BIT (0 ); -u16 def =0 ; -u16 use =0xffff ; +const u8 class =BPF_CLASS (insn->code); +const u8 code =BPF_OP (insn->code); +const u8 mode =BPF_MODE (insn->code); +const u8 size =BPF_SIZE (insn->code); +const u32 src =reg64_mask (insn->src_reg); +const u32 dst =reg64_mask (insn->dst_reg); +const u32 src32 =mask_lo (src); +const u32 dst32 =mask_lo (dst); +const u32 r0 =reg64_mask (0 ); +u32 def =0 ; +u32 use =U32_MAX ;switch (class) {case BPF_LD :
@@ -2080,8 +2089,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, use =0 ; }break ; -case BPF_LD |BPF_ABS : -case BPF_LD |BPF_IND : +case BPF_ABS : +case BPF_IND :/* stick with defaults */ break ; }
@@ -2089,7 +2098,15 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,case BPF_LDX :switch (mode) {case BPF_MEM : +/* a narrow load still redefines the whole register */ + def = dst; + use = src; +break ;case BPF_MEMSX : +/* +* sign extension defines the whole register; +* src holds a pointer, hence is used as 64-bit. +*/ def = dst; use = src;break ;
@@ -2107,12 +2124,19 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,switch (mode) {case BPF_MEM : def =0 ; - use = dst | src; + use = dst | (size ==BPF_DW ? src : src32);break ; -case BPF_ATOMIC : +case BPF_ATOMIC : { +/* +* dst holds a pointer and is always used as 64-bit; +* the value operand and r0 are read as 32-bit for BPF_W atomics. +*/ +u32 srcv = size ==BPF_DW ? src : src32; +u32 r0v = size ==BPF_DW ? r0 :mask_lo (r0); +switch (insn->imm) {case BPF_CMPXCHG : - use = r0 | dst | src; + use = r0v | dst | srcv; def = r0;break ;case BPF_LOAD_ACQ :
@@ -2121,10 +2145,10 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,break ;case BPF_STORE_REL : def =0 ; - use = dst | src; + use = dst | srcv;break ;default : - use = dst | src; + use = dst | srcv;if (insn->imm &BPF_FETCH ) def = src;else
@@ -2132,6 +2156,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, }break ; } + }break ;case BPF_ALU :case BPF_ALU64 :
@@ -2145,14 +2170,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,if (BPF_SRC (insn->code) ==BPF_K ) use =0 ;else - use = src; + use = class ==BPF_ALU64 ? src : src32;break ;default : def = dst;if (BPF_SRC (insn->code) ==BPF_K ) - use = dst; + use = class ==BPF_ALU64 ? dst : dst32;else - use = dst | src; + use = class ==BPF_ALU64 ? (dst | src) : (dst32 | src32); }break ;case BPF_JMP :
@@ -2178,13 +2203,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, use = def & ~BIT (BPF_REG_0 );if (bpf_get_call_summary (env, insn, &cs)) use =GENMASK (min_t (u8, cs.num_params,MAX_BPF_FUNC_REG_ARGS ),1 ); + def =mask_widen (def); + use =mask_widen (use);break ;default : def =0 ; -if (BPF_SRC (insn->code) ==BPF_K ) - use = dst; -else - use = dst | src; + use = class ==BPF_JMP ? dst : dst32; +if (BPF_SRC (insn->code) ==BPF_X ) + use |= class ==BPF_JMP ? src : src32; }break ; }
@@ -2249,8 +2275,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)int insn_idx = env->cfg.insn_postorder[i];struct insn_live_regs *live = &state[insn_idx];struct bpf_iarray *succ; -u16 new_out =0 ; -u16 new_in =0 ; +u32 new_out =0 ; +u32 new_in =0 ; succ =bpf_insn_successors (env, insn_idx);for (int s =0 ; s < succ->cnt; ++s)
@@ -2264,8 +2290,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) } } -for (i =0 ; i < insn_cnt; ++i) - insn_aux[i].live_regs_before = state[i].in; +for (i =0 ; i < insn_cnt; ++i) { +u32 in = state[i].in; + + insn_aux[i].live_regs_before =mask_lo (in) |mask_hi (in); + }if (env->log.level &BPF_LOG_LEVEL2 ) {verbose (env,"Live regs before insn:\n" );
kernel/bpf/verifier.c +0 -1
@@ -16703,7 +16703,6 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,int i;if (bpf_helper_call (call)) { -if (bpf_get_helper_proto (env, call->imm, &fn) <0 )/* error would be reported later */ return false;
Signature · 1 line
2.55.0
Eduard Zingerman
maintainerPATCH v4 5/7 bpf: infer zext_dst based on static register liveness analysis
In reply to Eduard ZingermanAs reported in the thread [1], the verifier's 32-bit operations zero extension logic is broken. This logic is responsible for correct semantics of 32-bit operations on s390 architecture. According to BPF semantics, operation w1 += 1 is supposed to zero extend the upper half of the register r1. On s390 the JIT relies on the verifier emitting explicit zero extension before such operations.
The verifier attempts to minimize the amount of zero extensions inserted by tracking whether upper halves of the 64-bit registers are ever used. Previously such tracking worked as follows:
- bpf_reg_state->subreg_def field was set by do_check_insn() for each operation defining lower but not the upper halves of the register.
- Whenever an operation reading the whole register was verified, the verifier checked register's subreg_def and set bpf_insn_aux_data->zext_dst flag as true via a call to mark_insn_zext() function.
- After the verification was complete, a special pass bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations with bpf_insn_aux_data->zext_dst set as true by adding explicit zero extension.
Note that the logic above relies on bpf_reg_state->subreg_def, which is a property of a current verifier state.
Before the commit [2] two additional steps happened:
- The verifier tracked upper and lower register halves' liveness as flags REG_LIVE_READ{32,64} in bpf_reg_state->live.
- The function propagate_liveness() called mark_insn_zext() in order to transfer the knowledge about which registers have their upper halves alive (and thus might require zero extension).
The commit [2] removed the two steps described above, hence making possible a situation like below:
- The register's upper half is set and is used on some verification path P1 and the register happens not to be marked as precise.
- The checkpoint C is created while processing some instruction between register initialization and usage.
- On some other verification path P2 the register's upper half is not initialized and that path ends hitting the checkpoint C.
- In such a case the register's initialization on path P2 would lack zext_dst mark, making it possible for the program to inject an arbitrary value in the register's upper half.
This commit replaces subreg_def based logic with computing zext_dst statically, as a part of the bpf_compute_live_registers() analysis:
- The analysis now tracks usage of upper and lower halves of the registers separately.
- If some instruction defines a 32-bit subregister, but not the whole register, *and* the upper half of the register is alive after that instruction, the instruction is marked as zext_dst.
There is one notable drop in precision: whenever a BPF subprogram is called, all 64 bits of parameter registers are presumed to be used. The assumption is that such a drop in precision would not inflict a noticeable performance penalty.
[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness")
--- include/linux/bpf_verifier.h | 7 +-- kernel/bpf/fixups.c | 27 +++++++---- kernel/bpf/liveness.c | 14 ++++++ kernel/bpf/verifier.c | 112 +++---------------------------------------- 4 files changed, 38 insertions(+), 122 deletions(-)
include/linux/bpf_verifier.h +1 -6
@@ -162,11 +162,6 @@ struct bpf_reg_state {* pointing to bpf_func_state. */ u32 frameno; -/* Tracks subreg definition. The stored value is the insn_idx of the -* writing insn. This is safe because subreg_def is used before any insn -* patching which only happens after main verification finished. -*/ -s32 subreg_def;/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */ bool precise; };
@@ -1637,7 +1632,6 @@ struct bpf_kfunc_desc_tab { };/* Functions exported from verifier.c, used by fixups.c */ -bool bpf_is_reg64 (struct bpf_insn *insn,u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t);void bpf_clear_insn_aux_data (struct bpf_verifier_env *env,int start,int len);void bpf_mark_subprog_exc_cb (struct bpf_verifier_env *env,int subprog);bool bpf_allow_tail_call_in_subprogs (struct bpf_verifier_env *env);
@@ -1661,5 +1655,6 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);int bpf_jit_subprogs (struct bpf_verifier_env *env);int bpf_fixup_call_args (struct bpf_verifier_env *env);int bpf_do_misc_fixups (struct bpf_verifier_env *env); +int bpf_insn_def32 (struct bpf_prog *prog,struct bpf_insn *insn);#endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/fixups.c +17 -10
@@ -68,8 +68,8 @@ static int insn_def_regno(const struct bpf_insn *insn)* code only. It returns TRUE if the source or destination register operates * on 64-bit, otherwise return FALSE. */ -bool bpf_is_reg64 (struct bpf_insn *insn, -u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t) +static bool bpf_is_reg64 (struct bpf_prog *prog,struct bpf_insn *insn, +u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t) {u8 code, class, op;
@@ -103,6 +103,10 @@ bool bpf_is_reg64(struct bpf_insn *insn,if (class ==BPF_ALU64 && op ==BPF_END && (insn->imm ==16 || insn->imm ==32 ))return false; +/* address space casts converted to BPF_ALU, see bpf_do_misc_fixups() */ +if (is_addr_space_cast32 (prog, insn)) +return false; +if (class ==BPF_ALU64 || class ==BPF_JMP || (class ==BPF_ALU && op ==BPF_END && insn->imm ==64 ))return true;
@@ -154,15 +158,18 @@ bool bpf_is_reg64(struct bpf_insn *insn,return true; } -/* Return TRUE if INSN has defined any 32-bit value explicitly. */ -static bool insn_has_def32 (struct bpf_insn *insn) +/* +* Return the 32-bit subregister defined by INSN, or -1 if INSN does not +* explicitly define a 32-bit value. +*/ +int bpf_insn_def32 (struct bpf_prog *prog,struct bpf_insn *insn) {int dst_reg =insn_def_regno (insn); -if (dst_reg ==-1 ) -return false; +if (dst_reg <0 ||bpf_is_reg64 (prog, insn, dst_reg,NULL ,DST_OP )) +return -1 ; -return !bpf_is_reg64 (insn, dst_reg,NULL ,DST_OP ); +return dst_reg; }static int kfunc_desc_cmp_by_imm_off (const void *a,const void *b)
@@ -279,7 +286,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,* (cnt == 1) is taken or not. There is no guarantee INSN at OFF is the * original insn at old prog. */ - data[off].zext_dst =insn_has_def32 (insn + off + cnt -1 ); + data[off].zext_dst =bpf_insn_def32 (new_prog, insn + off + cnt -1 ) >=0 ;if (cnt ==1 )return ;
@@ -291,7 +298,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,for (i = off; i < off + cnt -1 ; i++) {/* Expand insni[off]'s seen count to the patched range. */ data[i].seen = old_seen; - data[i].zext_dst =insn_has_def32 (insn + i); + data[i].zext_dst =bpf_insn_def32 (new_prog, insn + i) >=0 ; }/*
@@ -730,7 +737,7 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env,* BPF_STX + SRC_OP, so it is safe to pass NULL * here. */ -if (bpf_is_reg64 (&insn, load_reg,NULL ,DST_OP )) { +if (bpf_is_reg64 (env->prog, &insn, load_reg,NULL ,DST_OP )) {if (class ==BPF_LD &&BPF_MODE (code) ==BPF_IMM ) i++;
kernel/bpf/liveness.c +14 -0
@@ -2232,6 +2232,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)struct bpf_insn *insns = env->prog->insnsi;struct insn_live_regs *state;int insn_cnt = env->prog->len; +u64 pos, insn_pos;int err =0 , i, j;bool changed;
@@ -2291,9 +2292,18 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env) }for (i =0 ; i < insn_cnt; ++i) { +int def32 =bpf_insn_def32 (env->prog, &insns[i]); +u32 out = state[i].out;u32 in = state[i].in; insn_aux[i].live_regs_before =mask_lo (in) |mask_hi (in); +/* +* On architectures where 32-bit operations do not reset upper halves +* of the registers, the verifier needs to zero extend a destination +* register if an instruction defines a 32-bit subregister and the +* upper half of that register is alive after the instruction. +*/ + insn_aux[i].zext_dst = def32 >=0 && (mask_hi (out) &BIT (def32)); }if (env->log.level &BPF_LOG_LEVEL2 ) {
@@ -2310,7 +2320,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)else verbose (env,"." );verbose (env," " ); + pos = env->log.end_pos;bpf_verbose_insn (env, &insns[i]); + insn_pos = env->log.end_pos; +if (insn_aux[i].zext_dst) +verbose (env,"%*c; zext" ,bpf_vlog_alignment (insn_pos - pos),' ' );verbose (env,"\n" );if (bpf_is_ldimm64 (&insns[i])) i++;
kernel/bpf/verifier.c +6 -106
302 lines not on this page · raw message
Signature · 1 line
2.55.0
Eduard Zingerman
maintainerPATCH v4 6/7 bpf: simplify the bpf_is_reg64()
In reply to Eduard ZingermanAfter the previous commit bpf_is_reg64() is only used in a context where destination register's property is queried, and only for instructions for which insn_def_regno() >= 0.
Hence, simplify the function by:
- removing unused parameters;
- removing code paths considering BPF_JMP{,32} instructions;
- streamlining the condition expressions.
--- kernel/bpf/fixups.c | 110 +++++++++++++--------------------------------------- 1 file changed, 27 insertions(+), 83 deletions(-)
kernel/bpf/fixups.c +27 -83
@@ -64,95 +64,43 @@ static int insn_def_regno(const struct bpf_insn *insn) } } -/* This function is supposed to be used by the zero extension optimization -* code only. It returns TRUE if the source or destination register operates -* on 64-bit, otherwise return FALSE. +/* +* For use only in combination with insn_def_regno() >= 0. +* Returns TRUE if the destination register operates on 64-bit, +* otherwise return FALSE. */ -static bool bpf_is_reg64 (struct bpf_prog *prog,struct bpf_insn *insn, -u32 regno,struct bpf_reg_state *reg,enum bpf_reg_arg_type t) +static bool bpf_is_reg64 (struct bpf_prog *prog,struct bpf_insn *insn) { -u8 code, class, op; - - code = insn->code; - class =BPF_CLASS (code); - op =BPF_OP (code); -if (class ==BPF_JMP ) { -/* BPF_EXIT for "main" will reach here. Return TRUE -* conservatively. -*/ -if (op ==BPF_EXIT ) -return true; -if (op ==BPF_CALL ) { -/* BPF to BPF call will reach here because of marking -* caller saved clobber with DST_OP_NO_MARK for which we -* don't care the register def because they are anyway -* marked as NOT_INIT already. -*/ -if (insn->src_reg ==BPF_PSEUDO_CALL ) -return false; -/* Helper call will reach here because of arg type -* check, conservatively return TRUE. -*/ -if (t ==SRC_OP ) -return true; - -return false; - } - } +u8 class =BPF_CLASS (insn->code); +u8 mode =BPF_MODE (insn->code); +u8 size =BPF_SIZE (insn->code); +u8 op =BPF_OP (insn->code); +bool mode_mem; + +/* subregister endiness swap */ +if ((class ==BPF_ALU || class ==BPF_ALU64 ) && op ==BPF_END && insn->imm !=64 ) +return false; -if (class ==BPF_ALU64 && op ==BPF_END && (insn->imm ==16 || insn->imm ==32 )) +/* w0 += 1 */ +if (class ==BPF_ALU && op !=BPF_END )return false;/* address space casts converted to BPF_ALU, see bpf_do_misc_fixups() */ if (is_addr_space_cast32 (prog, insn))return false; -if (class ==BPF_ALU64 || class ==BPF_JMP || - (class ==BPF_ALU && op ==BPF_END && insn->imm ==64 )) -return true; - -if (class ==BPF_ALU || class ==BPF_JMP32 ) +/* non 64-bit, non signed extended loads */ + mode_mem = mode ==BPF_MEM || mode ==BPF_PROBE_MEM || mode ==BPF_PROBE_MEM32 ; +if (class ==BPF_LDX && mode_mem && size !=BPF_DW )return false; -if (class ==BPF_LDX ) { -if (t !=SRC_OP ) -return BPF_SIZE (code) ==BPF_DW ||BPF_MODE (code) ==BPF_MEMSX ; -/* LDX source must be ptr. */ -return true; - } - -if (class ==BPF_STX ) { -/* BPF_STX (including atomic variants) has one or more source -* operands, one of which is a ptr. Check whether the caller is -* asking about it. -*/ -if (t ==SRC_OP && reg->type !=SCALAR_VALUE ) -return true; -return BPF_SIZE (code) ==BPF_DW ; - } - -if (class ==BPF_LD ) { -u8 mode =BPF_MODE (code); - -/* LD_IMM64 */ -if (mode ==BPF_IMM ) -return true; - -/* Both LD_IND and LD_ABS return 32-bit data. */ -if (t !=SRC_OP ) -return false; - -/* Implicit ctx ptr. */ -if (regno ==BPF_REG_6 ) -return true; - -/* Explicit source could be any width. */ -return true; - } +/* atomics, see insn_def_regno() */ +if (class ==BPF_STX && size !=BPF_DW ) +return false; -if (class ==BPF_ST ) -/* The only source register for BPF_ST is a ptr. */ -return true; +/* both LD_IND and LD_ABS return 32-bit data. */ +if (class ==BPF_LD && (mode ==BPF_IND || mode ==BPF_ABS )) +return false;/* Conservatively return true at default. */ return true;
@@ -166,7 +114,7 @@ int bpf_insn_def32(struct bpf_prog *prog, struct bpf_insn *insn) {int dst_reg =insn_def_regno (insn); -if (dst_reg <0 ||bpf_is_reg64 (prog, insn, dst_reg,NULL ,DST_OP )) +if (dst_reg <0 ||bpf_is_reg64 (prog, insn))return -1 ;return dst_reg;
@@ -733,11 +681,7 @@ int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env,if (load_reg ==-1 )continue ; -/* NOTE: arg "reg" (the fourth one) is only used for -* BPF_STX + SRC_OP, so it is safe to pass NULL -* here. -*/ -if (bpf_is_reg64 (env->prog, &insn, load_reg,NULL ,DST_OP )) { +if (bpf_is_reg64 (env->prog, &insn)) {if (class ==BPF_LD &&BPF_MODE (code) ==BPF_IMM ) i++;
Signature · 1 line
2.55.0
Eduard Zingerman
maintainerPATCH v4 7/7 selftests/bpf: verify zext_dst annotations for various instructions
In reply to Eduard ZingermanIncludes the following test cases:
- a test showing that zero extension flags do not propagate through state pruning in the unpatched kernel.
- a 32-bit subregister consumed by MOV32 and ALU32 operations (never zext'ed);
- a 64-bit MOV (never zext'ed);
- a narrow (32-bit) BPF_LDX load whose result is read as 64-bit;
- 32-bit atomic fetch_add and cmpxchg whose result is read as 64-bit;
- a CFG case where a 32-bit definition's upper half is used only on one of two branches;
- no zext for dead registers;
- LD_ABS defines only lower 32 bits, hence needs zext when the result is used as 64-bits;
- helper, kfunc and subprogram parameters are considered to use full 64 bits;
- a 32-bit subregister consumed by JMP32 (X/K) operations;
- a 32-bit subregister consumed by JMP (X/K) operations;
- a 64-bit register consumed by both JMP and JMP32 operations (never zext'ed);
- ALU64 and address space cast operations on arena pointers;
- memory loads using BPF_PROBE_MEM instructions.
--- tools/testing/selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/verifier_zext.c | 392 ++++++++++++++++++++++ 2 files changed, 394 insertions(+)
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -126,6 +126,7 @@#include "verifier_jit_inline.skel.h" #include "irq.skel.h" #include "verifier_ctx_ptr_param.skel.h" +#include "verifier_zext.skel.h" #define MAX_ENTRIES 11
@@ -281,6 +282,7 @@ void test_irq(void) { RUN(irq); }void test_verifier_mtu (void ) {RUN (verifier_mtu); }void test_verifier_jit_inline (void ) {RUN (verifier_jit_inline); }void test_verifier_ctx_ptr_param (void ) {RUN (verifier_ctx_ptr_param); } +void test_verifier_zext (void ) {RUN_TESTS (verifier_zext); }static int init_test_val_map (struct bpf_object *obj,char *map_name) {
tools/testing/selftests/bpf/progs/verifier_zext.c new file +392 -0
392 lines not on this page · raw message
Signature · 1 line
2.55.0
Re: [PATCH bpf-next v4 7/7] selftests/bpf: verify zext_dst annotations for various instructions
Hello,
Thank you for preparing the fix. I tested the v4 series against my reproducer on s390x.
Tested revisions
I first built and booted the exact final commit of the v4 series applied by the patchwork bot:
8b365b3c68b474a1053ec0755dacdc751578afb0
Linux 7.2.0-rc6-g8b365b3c68b4
I then fetched, built, and booted the current master directly from the official bpf-next repository on 2026-08-17:
repository: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git
commit: 8eb18920646415f6b4f100b34bd456c4fe002196
Linux: 7.2.0-rc6-g8eb189206464
I verified that the v4 final commit is an ancestor of this master revision, which contains 106 subsequent commits. Both tested source trees were clean.
Test conditions
Both kernels were tested in s390x QEMU guests with Debian trixie userspace. The reproducer ran as uid 1000 with every capability set, including the bounding set, cleared:
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000000000000000
CapAmb: 0000000000000000
The relevant runtime and build conditions were:
kernel.unprivileged_bpf_disabled = 0
net.core.bpf_jit_enable = 1
CONFIG_BPF_JIT_ALWAYS_ON=y
The BPF program was accepted, attached as a socket filter, and JIT-executed in every run. The result was therefore not caused by verifier rejection or interpreter fallback.
Results
The results were identical on the v4 final commit and the later bpf-next master revision.
I ran my reproducer ten times on each kernel. All runs stopped before a kernel address could be disclosed or an out-of-bounds read/write primitive could be established. None reached the task, credential, or privilege-escalation stages.
I also ran a focused regression test using the same verifier/JIT mismatch trigger to distinguish the attempted out-of-bounds accesses from valid map value accesses. On each kernel, all 20 runs confirmed that both the read and write remained within the map value.
Finally, I ran the merged zext_lost_across_checkpoint regression program with BPF_F_TEST_STATE_FREQ and BPF_F_TEST_RND_HI32. On each kernel, BPF_PROG_TEST_RUN completed all 20 repetitions with:
PASS: zext_lost_across_checkpoint retval=0 repeat=20
The kernel taint value remained zero after testing. I found no BTF error, BUG, WARNING, Oops, user process fault, kernel panic, timeout, or signal termination.
These results confirm that the merged v4 series prevents the s390x verifier/JIT zero-extension mismatch exercised by my reproducer. The same behavior is preserved in the tested current bpf-next master revision.
For the v4 series, my test result is:
Thank you for your time.
Thanks,
Min-gyu Kim
2026년 8월 8일 (토) 오전 6:00, Eduard Zingerman <[email protected]>님이 작성:
Quoted 450 lines
Includes the following test cases:
- a test showing that zero extension flags do not propagate through state pruning in the unpatched kernel.
- a 32-bit subregister consumed by MOV32 and ALU32 operations (never zext'ed);
- a 64-bit MOV (never zext'ed);
- a narrow (32-bit)BPF_LDXload whose result is read as 64-bit;
- 32-bit atomicfetch_addand cmpxchg whose result is read as 64-bit;
- a CFG case where a 32-bit definition's upper half is used only on one of two branches;
- no zext for dead registers;
-LD_ABSdefines only lower 32 bits, hence needs zext when the result is used as 64-bits;
- helper, kfunc and subprogram parameters are considered to use full 64 bits;
- a 32-bit subregister consumed by JMP32 (X/K) operations;
- a 32-bit subregister consumed by JMP (X/K) operations;
- a 64-bit register consumed by both JMP and JMP32 operations (never zext'ed);
- ALU64 and address space cast operations on arena pointers;
- memory loads usingBPF_PROBE_MEMinstructions.Signed-off-by Eduard Zingerman <[email protected]>--- tools/testing/selftests/bpf/prog_tests/verifier.c | 2 + tools/testing/selftests/bpf/progs/verifier_zext.c | 392 ++++++++++++++++++++++ 2 files changed, 394 insertions(+)tools/testing/selftests/bpf/prog_tests/verifier.c +2
-0@@ -126,6 +126,7 @@#include "verifier_jit_inline.skel.h" #include "irq.skel.h" #include "verifier_ctx_ptr_param.skel.h" +#include "verifier_zext.skel.h" #define MAX_ENTRIES 11@@ -281,6 +282,7 @@ void test_irq(void) { RUN(irq); }void test_verifier_mtu (void ) {RUN (verifier_mtu); }void test_verifier_jit_inline (void ) {RUN (verifier_jit_inline); }void test_verifier_ctx_ptr_param (void ) {RUN (verifier_ctx_ptr_param); } +void test_verifier_zext (void ) {RUN_TESTS (verifier_zext); }static int init_test_val_map (struct bpf_object *obj,char *map_name) {tools/testing/selftests/bpf/progs/verifier_zext.c new file +392
-1@@ -0,0 +1,392 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" +#include <bpf_arena_common.h> +#include <bpf/bpf_core_read.h> +#include "bpf_misc.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE | BPF_F_NO_USER_CONV); + __uint(max_entries, 1); +} arena SEC(".maps"); + +extern long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym; + +/* to retain debug info for BTF generation */ +void __kfunc_btf_root (void ) +{ +bpf_kfunc_call_test4 (0 ,0 ,0 ,0 ); +bpf_arena_alloc_pages (0 ,0 ,0 ,0 ,0 ); +bpf_rdonly_cast (0 ,0 ); +} + +SEC("socket") +__flag (BPF_F_TEST_STATE_FREQ) +__flag (BPF_F_TEST_RND_HI32 ) +__success__retval (0 ) +__naked voidzext_lost_across_checkpoint (void) +{ + asmvolatile (" \ +call %[bpf_ktime_get_ns]; \ +r8 = r0; \ +r6 = 0xdeadbeefcafebabe ll; /* inject some value for r6's upper half */ \ +if r8 != 0 goto 1f; /* fall-through cached first, branch pruned */ \ +r6 = 32; /* full 64-bit def */ \ +goto 2f; \ +1: w6 = 32; /* 32-bit def, zext mark lost */ \ +2: r0 = r6; /* buggy verifier believed upper 32 bits are 0 */ \ +/* and thus did not zero extended w6 = 32. */ \ +r0 >>= 32; \ +exit; \ +" : + : __imm(bpf_ktime_get_ns) + : __clobber_all); +} + +/* 32-bit ALU result read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("w1 = w0{{ +}}; zext") +__naked void zext_alu32_hi_used (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +w1 = w0; \ +r0 = r1; \ +exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* 32-bit ALU result read only as 32-bit -> no zext */ +SEC("socket") +__success __log_level(2) +__not_msg("; zext") +__naked void no_zext_alu32_hi_unused (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +w1 = w0; /* MOV */ \ +w2 = w1; \ +w2 += w1; /* ALU32, BPF_X */ \ +w2 += 1; /* ALU32, BPF_K */ \ +w2 = w2; /* keep w2 alive for previous instruction */ \ +r0 = 0; \ +exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* 64-bit definition is never zero extended */ +SEC("socket") +__success __log_level(2) +__not_msg("r1 = r0{{.*}}; zext") +__naked void no_zext_mov64 (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +r1 = r0; \ +r0 = r1; \ +exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* Narrow load result read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("r1 = *(u32 *)(r10 -8){{ +}}; zext") +__naked void zext_narrow_load_hi_used (void ) +{ + asmvolatile (" \ +r0 = 0; \ +*(u64 *)(r10 - 8) = r0; \ +r1 = *(u32 *)(r10 - 8); \ +r0 = r1; \ +exit; \ +" ::: __clobber_all); +} + +/* 32-bit atomic fetch result read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("r1 = atomic_fetch_add((u32 *)(r10 -8), r1){{ +}}; zext") +__naked void zext_atomic_fetch32_hi_used (void ) +{ + asmvolatile (" \ +r1 = 0; \ +*(u64 *)(r10 - 8) = r1; \ +w1 = 1; \ +.8byte %[fetch_add32]; \ +r0 = r1; \ +exit; \ +" : + : __imm_insn(fetch_add32, + BPF_ATOMIC_OP(BPF_W, BPF_ADD | BPF_FETCH, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +/* 32-bit atomic cmpxchg result (r0) read as 64-bit -> zext */ +SEC("socket") +__success __log_level(2) +__msg("r0 = atomic_cmpxchg((u32 *)(r10 -8), r0, r1){{ +}}; zext") +__naked void zext_cmpxchg32_hi_used (void ) +{ + asmvolatile (" \ +r1 = 0; \ +*(u64 *)(r10 - 8) = r1; \ +w0 = 0; \ +w1 = 1; \ +.8byte %[cmpxchg32]; \ +r2 = r0; \ +r0 = r2; \ +exit; \ +" : + : __imm_insn(cmpxchg32, + BPF_ATOMIC_OP(BPF_W, BPF_CMPXCHG, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +/* 32-bit def before a branch, upper half used on one branch -> zext */ +SEC("socket") +__success __log_level(2) +__msg("w6 = 32{{ +}}; zext") +__naked void zext_cfg_hi_used_one_branch (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +w6 = 32; \ +if r0 == 0 goto 1f; \ +r0 = r6; \ +exit; \ +1: r0 = 0; \ +exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* r1's upper half is dead, so 'w1 = 1' must NOT be marked for zero extension. */ +SEC("socket") +__success __log_level(2) +__not_msg("w1 = 1{{.*}}; zext") +__naked void no_zext_other_reg_hi_used (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +r6 = r0; \ +r6 <<= 32; \ +w1 = 1; \ +r0 = r6; \ +exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* LD_ABS defines r0; when r0 is read as 64-bit it must be zero extended */ +SEC("socket") +__success __log_level(2) +__msg("r0 = *(u8 *)skb[0]{{.*}}; zext") +__naked void zext_ld_abs_hi_used (void ) +{ + asmvolatile (" \ +r6 = r1; \ +r0 = *(u8 *)skb[0]; \ +r7 = r0; \ +r0 = r7; \ +exit; \ +" ::: __clobber_all); +} + +/* Helper parameters are read as 64-bit (call_use_mask() fallback) */ +SEC("socket") +__success __log_level(2) +__msg("w2 = 1{{ +}}; zext") +__naked void helper_param_read_as_64bit (void ) +{ + asmvolatile (" \ +r1 = r10; \ +r1 += -8; \ +w2 = 1; \ +call %[bpf_trace_printk]; \ +r0 = 0; \ +exit; \ +" : + : __imm(bpf_trace_printk) + : __clobber_all); +} + +static __used __naked int subprog_reads_arg_as_64bit (void ) +{ + asmvolatile (" \ +r0 = r1; \ +exit; \ +" ::: __clobber_all); +} + +/* subprogram parameters are conservatively read as 64-bit */ +SEC("socket") +__success __log_level(2) +__msg("w1 = w0{{ +}}; zext") +__naked void subprog_param_read_as_64bit (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +w1 = w0; \ +call subprog_reads_arg_as_64bit; \ +r0 = 0; \ +exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +/* kfunc parameters are zero extended */ +SEC("tc") +__success __log_level(2) +__msg("w1 = 1{{ +}}; zext") +__msg("w2 = 1{{ +}}; zext") +__msg("w3 = 1{{ +}}; zext") +__msg("w4 = 1{{ +}}; zext") +__naked void kfunc_param_read_per_btf (void ) +{ + asmvolatile (" \ +w1 = 1; \ +w2 = 1; \ +w3 = 1; \ +w4 = 1; \ +call bpf_kfunc_call_test4; \ +r0 = 0; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__success __log_level(2) +__not_msg("; zext") +__naked void alu32_and_32bit_conditional (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +w1 = w0; \ +if w1 > 42 goto 1f; /* BPF_K */ \ +w2 = 28; \ +if w2 > w1 goto 1f; /* BPF_X */ \ +r0 = 0; \ +1: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__success __log_level(2) +__msg("w1 = w0{{ +}}; zext") +__naked void alu32_and_64bit_conditional (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +w1 = w0; \ +if r1 > 42 goto 1f; /* BPF_K */ \ +r2 = 28; \ +if r2 > r1 goto 1f; /* BPF_X */ \ +r0 = 0; \ +1: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__success __log_level(2) +__not_msg("; zext") +__naked void alu64_and_conditionals (void ) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +r1 = r0; \ +if w1 > 42 goto 1f; /* BPF_K */ \ +if r1 > 42 goto 1f; /* BPF_K */ \ +r2 = 28; \ +if w2 > w1 goto 1f; /* BPF_X */ \ +if r2 > r1 goto 1f; /* BPF_X */ \ +r0 = 0; \ +1: exit; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +#ifdef __BPF_FEATURE_ADDR_SPACE_CAST + +SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") +__arch_s390x +__xlated("7: w1 = w0") +__xlated("8: w1 = w1") +__xlated("9: w1 += 8") +__xlated("10: w1 = w1") +__xlated("11: w2 = w1") +__xlated("12: w2 = w2") +__xlated("13: *(u64 *)(r1 +0) = r2") +__naked void arena_ptr (void ) +{ + asmvolatile (" \ +r1 = %[arena] ll; \ +r2 = 0; \ +r3 = 1; \ +r4 = 0; \ +r5 = 0; \ +call %[bpf_arena_alloc_pages]; \ +r1 = addr_space_cast(r0, 0, 1); /* needs zext */ \ +r1 += 8; /* needs zext */ \ +r2 = addr_space_cast(r1, 1, 0); /* needs zext because of BPF_F_NO_USER_CONV */ \ +*(u64 *)(r1 +0) = r2; \ +r0 = 0; \ +exit; \ +" : + : __imm(bpf_arena_alloc_pages), +__imm_addr (arena) + : __clobber_all); +} + +#endif + +/* Check if probe mem loads keep their zero extension. */ +SEC("socket") +__success __log_level(2) +__arch_s390x +__xlated("3: r1 = *(u64 *)(r0 +0)") +__xlated("4: r2 = *(u32 *)(r0 +0)") +__xlated("5: w2 = w2") +__xlated("6: r3 = *(u16 *)(r0 +0)") +__xlated("7: w3 = w3") +__xlated("8: r4 = *(u8 *)(r0 +0)") +__xlated("9: w4 = w4") +__naked void probe_mem (void ) +{ + asmvolatile (" \ +r1 = 0; \ +r2 = 0; \ +call %[bpf_rdonly_cast]; \ +r1 = *(u64 *)(r0 + 0); /* BPF_PROBE_MEM */ \ +r2 = *(u32 *)(r0 + 0); /* BPF_PROBE_MEM */ \ +r3 = *(u16 *)(r0 + 0); /* BPF_PROBE_MEM */ \ +r4 = *(u8 *)(r0 + 0); /* BPF_PROBE_MEM */ \ +r0 = r1; /* make the registers used */ \ +r0 += r2; \ +r0 += r3; \ +r0 += r4; \ +1: exit; \ +" : + : __imm(bpf_rdonly_cast) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL"; --2.55.0
Eduard Zingerman
maintainerRe: [PATCH bpf-next v4 7/7] selftests/bpf: verify zext_dst annotations for various instructions
On Mon, 2026-08-17 at 20:11 +0900, Min-gyu Kim wrote:
...
Quoted 9 lines
These results confirm that the merged v4 series prevents the s390x verifier/JIT zero-extension mismatch exercised by my reproducer. The same behavior is preserved in the tested current bpf-next master revision.
For the v4 series, my test result is:
Tested-by Min-gyu Kim <[email protected]>Thank you for your time.
Great, thank you for testing.
...
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Quoted 8 lines · On Fri, 7 Aug 2026 13:59:29 -0700 you wrote:
Min-gyu Kim reported a bug in 32-bit operations zero extension handling [1]. The same issue was independently identified by STAR Labs SG. The bug was introduced by the commit [2]. The tl;dr of the bug is that the verifier does not carry zero extension marks across pruning points. The detailed mechanism is described in patch #3.
[...]
Here is the summary with links:
- [bpf-next,v4,1/7] bpf: do not print a newline after disassembly in bpf_verbose_insn()
https://git.kernel.org/bpf/bpf-next/c/483a1bb0b6cf - [bpf-next,v4,2/7] bpf: extract is_addr_space_cast32() utility function https://git.kernel.org/bpf/bpf-next/c/d977dca7d073 - [bpf-next,v4,3/7] bpf: move bpf_is_reg64() to fixups.c https://git.kernel.org/bpf/bpf-next/c/05b71078f305 - [bpf-next,v4,4/7] bpf: track upper 32-bit register halves' liveness in compute_live_registers() https://git.kernel.org/bpf/bpf-next/c/ef1ddbfcfaef - [bpf-next,v4,5/7] bpf: infer zext_dst based on static register liveness analysis https://git.kernel.org/bpf/bpf-next/c/7ce090afbf72 - [bpf-next,v4,6/7] bpf: simplify the bpf_is_reg64() https://git.kernel.org/bpf/bpf-next/c/be4f8d6f2ff7 - [bpf-next,v4,7/7] selftests/bpf: verify zext_dst annotations for various instructions https://git.kernel.org/bpf/bpf-next/c/8b365b3c68b4
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html