Skip to contentSkip to conversation

bpf: infer zext_dst based on static register liveness analysis

PATCH Superseded Eduard Zingerman 3 participants 11 messages Aug 2 – Aug 3

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eduard Zingerman

author maintainer
#0 lore raw

v3 cover bpf: infer zext_dst based on static register liveness analysis

Min-gyu Kim reported a bug in 32-bit operations zero extension handling [1]. The bug was introduced by the commit [2]. The tl;dr of the bug is that the verifier does not carry zero extension marks across pruning points. The detailed mechanism is described in patch #3.

Fix this by reworking zero extension logic to avoid main-path based subreg_def tracking, and instead extend the live registers analysis to track upper register halves' liveness.

As noted in the commit message for patch #3:

There is one notable drop in precision: whenever a BPF subprogram is called, all 64 bits of parameter registers are presumed to be used. The assumption is that such a drop in precision would not inflict noticeable performance penalty.

Ilya, could you please help with testing this conjecture? Min-gyu, could you please test the proposed fix against your reproducer?

Side note:
Patch #1 is a small refactoring for disasm.c, as patch #3 adds a new location where bpf_verbose_insn() is called and there as well I have to wrangle with the newline added by the disasm code.

[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness")

Changelog:
v1 -> v2:
- fixed BPF_JMP32 handling, these no longer for zero extension (sashiko)
- removed dead code in print_insn_for_graph() (bot+bpf-ci)
- reworked bpf_is_reg64() to drop dead code (sashiko, bot+bpf-ci)

v2 -> v3:
- fix for proper s390x calling convention all call parameters are now zero extended if necessary (sashiko)
- fix for PTR_TO_ARENA alu operations to set zext_dst when needs_zext is set, in order for such operations to be zero extended (sashiko)
- updated bpf_is_reg64() to handle BPR_PROBE_MEM{,32} loads in order to avoid a false positive from sashiko.

v1: https://lore.kernel.org/bpf/[email protected]/T/
v2: https://lore.kernel.org/bpf/[email protected]/T/

---
Eduard Zingerman (5):
bpf: do not print a newline after disassembly in bpf_verbose_insn()
bpf: track upper 32-bit register halves' liveness in compute_live_registers()
bpf: infer zext_dst based on static register liveness analysis
bpf: simplify the bpf_is_reg64()
selftests/bpf: verify zext_dst annotations for various instructions

 include/linux/bpf_verifier.h                      |   7 +-
 kernel/bpf/backtrack.c                            |   1 +
 kernel/bpf/disasm.c                               |  68 ++--
 kernel/bpf/fixups.c                               |  66 +++-
 kernel/bpf/liveness.c                             | 109 ++++--
 kernel/bpf/verifier.c                             | 205 +-----------
 tools/bpf/bpftool/xlated_dumper.c                 |  19 +-
 tools/testing/selftests/bpf/disasm_helpers.c      |   3 +-
 tools/testing/selftests/bpf/prog_tests/verifier.c |   2 +
 tools/testing/selftests/bpf/progs/verifier_zext.c | 388 ++++++++++++++++++++++
 10 files changed, 568 insertions(+), 300 deletions(-)

---
base-commit: fdec474c65fd35d5a6e1497ed50a9f98c07192f0
change-id: 20260731-static-zext-938cd128273c

author maintainer
#1 lore raw

PATCH v3 1/5 bpf: do not print a newline after disassembly in bpf_verbose_insn()

At the moment there are more callsites that want bpf_verbose_insn() to not print a newline after the instruction, than callsites that want a newline. Drop '\n' from disasm.c. Non-functional change.

The changes in bpftool are verified by writing a bpf program using a variety of instructions and comparing prog dump xlated output in the following modes: plain, opcodes, visual, visual opcodes. The output before and after the changes is identical.

---
 kernel/bpf/backtrack.c                       |  1 +
 kernel/bpf/disasm.c                          | 68 ++++++++++++++--------------
 kernel/bpf/liveness.c                        |  6 +--
 kernel/bpf/verifier.c                        |  1 +
 tools/bpf/bpftool/xlated_dumper.c            | 19 ++------
 tools/testing/selftests/bpf/disasm_helpers.c |  3 +-
 6 files changed, 43 insertions(+), 55 deletions(-)
kernel/bpf/backtrack.c +1 -0
index 2e4ae0ef0860..7c4599e65ece 100644
@@ -285,6 +285,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 		verbose(env, "stack=%s before ", env->tmp_str_buf);
 		verbose(env, "%d: ", idx);
 		bpf_verbose_insn(env, insn);
+		verbose(env, "\n");
 	}
 
 	/* If there is a history record that some registers gained range at this insn,
kernel/bpf/disasm.c +34 -34

219 lines not on this page · raw message

kernel/bpf/liveness.c +2 -4
index 0aadfbae0acc..ff1e68cc4bd1 100644
@@ -497,7 +497,6 @@ static void print_instance(struct bpf_verifier_env *env, struct func_instance *i
 		pos = env->log.end_pos;
 		verbose(env, "%3d: ", insn_idx);
 		bpf_verbose_insn(env, &insns[insn_idx]);
-		bpf_vlog_reset(&env->log, env->log.end_pos - 1); /* remove \n */
 		insn_pos = env->log.end_pos;
 		verbose(env, "%*c;", bpf_vlog_alignment(insn_pos - pos), ' ');
 		pos = env->log.end_pos;
@@ -1043,7 +1042,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
 		if (!printed) {
 			verbose(env, "%3d: ", idx);
 			bpf_verbose_insn(env, insn);
-			bpf_vlog_reset(&env->log, env->log.end_pos - 1);
 			printed = true;
 		}
 		verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]);
@@ -1058,7 +1056,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
 		if (!printed) {
 			verbose(env, "%3d: ", idx);
 			bpf_verbose_insn(env, insn);
-			bpf_vlog_reset(&env->log, env->log.end_pos - 1);
 			printed = true;
 		}
 		verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]);
@@ -1070,7 +1067,6 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i
 		if (!printed) {
 			verbose(env, "%3d: ", idx);
 			bpf_verbose_insn(env, insn);
-			bpf_vlog_reset(&env->log, env->log.end_pos - 1);
 			printed = true;
 		}
 		verbose(env, "\tfp%+d: ", -(i + 1) * 8); verbose_arg_track(env, &at_stack_in[i]);
@@ -1545,6 +1541,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env,
 
 		verbose(env, "%3d: ", idx);
 		bpf_verbose_insn(env, &insns[idx]);
+		verbose(env, "\n");
 
 		/* Collect what needs printing */
 		if (is_ldx_stx_call &&
@@ -2285,6 +2282,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 					verbose(env, ".");
 			verbose(env, " ");
 			bpf_verbose_insn(env, &insns[i]);
+			verbose(env, "\n");
 			if (bpf_is_ldimm64(&insns[i]))
 				i++;
 		}
kernel/bpf/verifier.c +1 -0
index e6f35f4e715b..6123e7e4a320 100644
@@ -17475,6 +17475,7 @@ static int do_check(struct bpf_verifier_env *env)
 			env->prev_log_pos = env->log.end_pos;
 			verbose(env, "%d: ", env->insn_idx);
 			bpf_verbose_insn(env, insn);
+			verbose(env, "\n");
 			env->prev_insn_print_pos = env->log.end_pos - env->prev_log_pos;
 			env->prev_log_pos = env->log.end_pos;
 		}
tools/bpf/bpftool/xlated_dumper.c +4 -15
index 5e7cb8b36fef..5579173a61e3 100644
@@ -107,14 +107,7 @@ print_insn_for_graph(void *private_data, const char *fmt, ...)
 
 	p = buf;
 	while (*p != '\0') {
-		if (*p == '\n') {
-			memmove(p + 3, p, strlen(buf) + 1 - (p - buf));
-			/* Align each instruction dump row left. */
-			*p++ = '\\';
-			*p++ = 'l';
-			/* Output multiline concatenation. */
-			*p++ = '\\';
-		} else if (*p == '<' || *p == '>' || *p == '|' || *p == '&') {
+		if (*p == '<' || *p == '>' || *p == '|' || *p == '&') {
 			memmove(p + 1, p, strlen(buf) + 1 - (p - buf));
 			/* Escape special character. */
 			*p++ = '\\';
@@ -129,16 +122,10 @@ print_insn_for_graph(void *private_data, const char *fmt, ...)
 static void __printf(2, 3)
 print_insn_json(void *private_data, const char *fmt, ...)
 {
-	unsigned int l = strlen(fmt);
-	char chomped_fmt[l];
 	va_list args;
 
 	va_start(args, fmt);
-	if (l > 0) {
-		strncpy(chomped_fmt, fmt, l - 1);
-		chomped_fmt[l - 1] = '\0';
-	}
-	jsonw_vprintf_enquote(json_wtr, chomped_fmt, args);
+	jsonw_vprintf_enquote(json_wtr, fmt, args);
 	va_end(args);
 }
 
@@ -351,6 +338,7 @@ void dump_xlated_plain(struct dump_data *dd, void *buf, unsigned int len,
 
 		printf("%4u: ", i);
 		print_bpf_insn(&cbs, insn + i, true);
+		printf("\n");
 
 		if (opcodes) {
 			printf("       ");
@@ -417,6 +405,7 @@ void dump_xlated_for_graph(struct dump_data *dd, void *buf_start, void *buf_end,
 
 		printf("%u: ", insn_off);
 		print_bpf_insn(&cbs, cur, true);
+		printf("\\l\\\n");
 
 		if (opcodes) {
 			printf("\\ \\ \\ \\ ");
tools/testing/selftests/bpf/disasm_helpers.c +1 -2
index f529f1c8c171..30221352568d 100644
@@ -55,10 +55,9 @@ struct bpf_insn *disasm_insn(struct bpf_insn *insn, char *buf, size_t buf_sz)
 	 * for each instruction (FF stands for instruction `code` byte).
 	 * Remove the prefix inplace, and also simplify call instructions.
 	 * E.g.: "(85) call foo#10" -> "call foo".
-	 * Also remove newline in the end (the 'max(strlen(buf) - 1, 0)' thing).
 	 */
 	pfx_end = buf + 5;
-	sfx_start = buf + max((int)strlen(buf) - 1, 0);
+	sfx_start = buf + (int)strlen(buf);
 	if (strncmp(pfx_end, "call ", 5) == 0 && (tmp = strrchr(buf, '#')))
 		sfx_start = tmp;
 	len = sfx_start - pfx_end;
Signature · 1 line
2.53.0
maintainer Reviewed-by on v3 1/5
#2 lore raw

Re: [PATCH bpf-next v3 1/5] bpf: do not print a newline after disassembly in bpf_verbose_insn()

Quoted 11 lines · Eduard Zingerman

At the moment there are more callsites that want bpf_verbose_insn() to not print a newline after the instruction, than callsites that want a newline. Drop '\n' from disasm.c. Non-functional change.

The changes in bpftool are verified by writing a bpf program using a variety of instructions and comparing prog dump xlated output in the following modes: plain, opcodes, visual, visual opcodes. The output before and after the changes is identical.

Looks good, and looking at the diff, the change makes sense. Thanks!

author maintainer
#3 lore raw

PATCH v3 2/5 bpf: track upper 32-bit register halves' liveness in compute_live_registers()

In reply to Eduard Zingerman

Extend compute_live_registers() to track upper and lower register halves' liveness separately. This is mostly straightforward:
- use/def masks are extended to track 2 bits per register;
- compute_insn_live_regs() is updated to properly track these 2 bits according to the instruction semantics.

---
 kernel/bpf/liveness.c | 89 ++++++++++++++++++++++++++++++++++-----------------
 kernel/bpf/verifier.c |  1 -
 2 files changed, 59 insertions(+), 31 deletions(-)
kernel/bpf/liveness.c +59 -30
index ff1e68cc4bd1..edfc2480b0f0 100644
@@ -2047,29 +2047,38 @@ int bpf_compute_subprog_arg_access(struct bpf_verifier_env *env)
 
 /* Each field is a register bitmask */
 struct insn_live_regs {
-	u16 use;	/* registers read by instruction */
-	u16 def;	/* registers written by instruction */
-	u16 in;		/* registers that may be alive before instruction */
-	u16 out;	/* registers that may be alive after instruction */
+	u32 use;	/* registers read by instruction */
+	u32 def;	/* registers written by instruction */
+	u32 in;		/* registers that may be alive before instruction */
+	u32 out;	/* registers that may be alive after instruction */
 };
 
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
+static inline u32 reg32_mask(u32 n) { return BIT(n); }
+static inline u32 reg64_mask(u32 n) { return BIT(n) | BIT(n + 16); }
+static inline u32 mask_widen(u32 m) { return m | (m << 16); }
+static inline u16 mask_lo(u32 m) { return (u16)m; }
+static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
+
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
 				   struct insn_live_regs *info)
 {
 	struct bpf_call_summary cs;
-	u8 class = BPF_CLASS(insn->code);
-	u8 code = BPF_OP(insn->code);
-	u8 mode = BPF_MODE(insn->code);
-	u16 src = BIT(insn->src_reg);
-	u16 dst = BIT(insn->dst_reg);
-	u16 r0  = BIT(0);
-	u16 def = 0;
-	u16 use = 0xffff;
+	const u8 class = BPF_CLASS(insn->code);
+	const u8 code = BPF_OP(insn->code);
+	const u8 mode = BPF_MODE(insn->code);
+	const u8 size = BPF_SIZE(insn->code);
+	const u32 src = reg64_mask(insn->src_reg);
+	const u32 dst = reg64_mask(insn->dst_reg);
+	const u32 src32 = mask_lo(src);
+	const u32 dst32 = mask_lo(dst);
+	const u32 r0  = reg64_mask(0);
+	u32 def = 0;
+	u32 use = U32_MAX;
 
 	switch (class) {
 	case BPF_LD:
@@ -2080,8 +2089,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				use = 0;
 			}
 			break;
-		case BPF_LD | BPF_ABS:
-		case BPF_LD | BPF_IND:
+		case BPF_ABS:
+		case BPF_IND:
 			/* stick with defaults */
 			break;
 		}
@@ -2089,7 +2098,15 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 	case BPF_LDX:
 		switch (mode) {
 		case BPF_MEM:
+			/* a narrow load still redefines the whole register */
+			def = dst;
+			use = src;
+			break;
 		case BPF_MEMSX:
+			/*
+			 * sign extension defines the whole register;
+			 * src holds a pointer, hence is used as 64-bit.
+			 */
 			def = dst;
 			use = src;
 			break;
@@ -2107,12 +2124,19 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 		switch (mode) {
 		case BPF_MEM:
 			def = 0;
-			use = dst | src;
+			use = dst | (size == BPF_DW ? src : src32);
 			break;
-		case BPF_ATOMIC:
+		case BPF_ATOMIC: {
+			/*
+			 * dst holds a pointer and is always used as 64-bit;
+			 * the value operand and r0 are read as 32-bit for BPF_W atomics.
+			 */
+			u32 srcv = size == BPF_DW ? src : src32;
+			u32 r0v  = size == BPF_DW ? r0 : mask_lo(r0);
+
 			switch (insn->imm) {
 			case BPF_CMPXCHG:
-				use = r0 | dst | src;
+				use = r0v | dst | srcv;
 				def = r0;
 				break;
 			case BPF_LOAD_ACQ:
@@ -2121,10 +2145,10 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				break;
 			case BPF_STORE_REL:
 				def = 0;
-				use = dst | src;
+				use = dst | srcv;
 				break;
 			default:
-				use = dst | src;
+				use = dst | srcv;
 				if (insn->imm & BPF_FETCH)
 					def = src;
 				else
@@ -2132,6 +2156,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			}
 			break;
 		}
+		}
 		break;
 	case BPF_ALU:
 	case BPF_ALU64:
@@ -2145,14 +2170,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			if (BPF_SRC(insn->code) == BPF_K)
 				use = 0;
 			else
-				use = src;
+				use = class == BPF_ALU64 ? src : src32;
 			break;
 		default:
 			def = dst;
 			if (BPF_SRC(insn->code) == BPF_K)
-				use = dst;
+				use = class == BPF_ALU64 ? dst : dst32;
 			else
-				use = dst | src;
+				use = class == BPF_ALU64 ? (dst | src) : (dst32 | src32);
 		}
 		break;
 	case BPF_JMP:
@@ -2178,13 +2203,14 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			use = def & ~BIT(BPF_REG_0);
 			if (bpf_get_call_summary(env, insn, &cs))
 				use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1);
+			def = mask_widen(def);
+			use = mask_widen(use);
 			break;
 		default:
 			def = 0;
-			if (BPF_SRC(insn->code) == BPF_K)
-				use = dst;
-			else
-				use = dst | src;
+			use = class == BPF_JMP ? dst : dst32;
+			if (BPF_SRC(insn->code) == BPF_X)
+				use |= class == BPF_JMP ? src : src32;
 		}
 		break;
 	}
@@ -2249,8 +2275,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 			int insn_idx = env->cfg.insn_postorder[i];
 			struct insn_live_regs *live = &state[insn_idx];
 			struct bpf_iarray *succ;
-			u16 new_out = 0;
-			u16 new_in = 0;
+			u32 new_out = 0;
+			u32 new_in = 0;
 
 			succ = bpf_insn_successors(env, insn_idx);
 			for (int s = 0; s < succ->cnt; ++s)
@@ -2264,8 +2290,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		}
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		insn_aux[i].live_regs_before = state[i].in;
+	for (i = 0; i < insn_cnt; ++i) {
+		u32 in = state[i].in;
+
+		insn_aux[i].live_regs_before = mask_lo(in) | mask_hi(in);
+	}
 
 	if (env->log.level & BPF_LOG_LEVEL2) {
 		verbose(env, "Live regs before insn:\n");
kernel/bpf/verifier.c +0 -1
index 6123e7e4a320..896bc5aaaaaf 100644
@@ -16730,7 +16730,6 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 	int i;
 
 	if (bpf_helper_call(call)) {
-
 		if (bpf_get_helper_proto(env, call->imm, &fn) < 0)
 			/* error would be reported later */
 			return false;
Signature · 1 line
2.53.0
author maintainer
#4 lore raw

PATCH v3 3/5 bpf: infer zext_dst based on static register liveness analysis

In reply to Eduard Zingerman

As reported in the thread [1], the verifier's 32-bit operations zero extension logic is broken. This logic is responsible for correct semantics of 32-bit operations on s390 architecture. According to BPF semantics, operation w1 += 1 is supposed to zero extend the upper half of the register r1. On s390 the JIT relies on the verifier emitting explicit zero extension before such operations.

The verifier attempts to minimize the amount of zero extensions inserted by tracking whether upper halves of the 64-bit registers are ever used. Previously such tracking worked as follows:
- bpf_reg_state->subreg_def field was set by do_check_insn() for each operation defining lower but not the upper halves of the register.
- Whenever an operation reading the whole register was verified, the verifier checked register's subreg_def and set bpf_insn_aux_data->zext_dst flag as true via a call to mark_insn_zext() function.
- After the verification was complete, a special pass bpf_opt_subreg_zext_lo32_rnd_hi32() extended 32-bit operations with bpf_insn_aux_data->zext_dst set as true by adding explicit zero extension.

Note that the logic above relies on bpf_reg_state->subreg_def, which is a property of a current verifier state.
Before the commit [2] two additional steps happened:
- The verifier tracked upper and lower register halves' liveness as flags REG_LIVE_READ{32,64} in bpf_reg_state->live.
- The function propagate_liveness() called mark_insn_zext() in order to transfer the knowledge about which registers have their upper halves alive (and thus might require zero extension).

The commit [2] removed the two steps described above, hence making possible a situation like below:
- The register's upper half is set and is used on some verification path P1 and the register happens not to be marked as precise.
- The checkpoint C is created while processing some instruction between register initialization and usage.
- On some other verification path P2 the register's upper half is not initialized and that path ends hitting the checkpoint C.
- In such a case the register's initialization on path P2 would lack zext_dst mark, making it possible for the program to inject an arbitrary value in the register's upper half.

This commit replaces subreg_def based logic with computing zext_dst statically, as a part of the bpf_compute_live_registers() analysis:
- The analysis now tracks usage of upper and lower halves of the registers separately.
- If some instruction defines a 32-bit subregister, but not the whole register, *and* the upper half of the register is alive after that instruction, the instruction is marked as zext_dst.

There is one notable drop in precision: whenever a BPF subprogram is called, all 64 bits of parameter registers are presumed to be used. The assumption is that such a drop in precision would not inflict a noticeable performance penalty.

[1] https://lore.kernel.org/bpf/CAGKGUv=sOuqQtA1Ub-5JXfA4FPosJFYKAQE4B79cK+P1erxqtg@mail.gmail.com/
[2] commit 107e16979905 ("bpf: disable and remove registers chain based liveness")

---
 include/linux/bpf_verifier.h |   6 +--
 kernel/bpf/fixups.c          |  17 ++++---
 kernel/bpf/liveness.c        |  14 ++++++
 kernel/bpf/verifier.c        | 113 +++----------------------------------------
 4 files changed, 32 insertions(+), 118 deletions(-)
include/linux/bpf_verifier.h +1 -5
index 682c2cd3b844..956243547446 100644
@@ -162,11 +162,6 @@ struct bpf_reg_state {
 	 * pointing to bpf_func_state.
 	 */
 	u32 frameno;
-	/* Tracks subreg definition. The stored value is the insn_idx of the
-	 * writing insn. This is safe because subreg_def is used before any insn
-	 * patching which only happens after main verification finished.
-	 */
-	s32 subreg_def;
 	/* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
 	bool precise;
 };
@@ -1655,5 +1650,6 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
 int bpf_jit_subprogs(struct bpf_verifier_env *env);
 int bpf_fixup_call_args(struct bpf_verifier_env *env);
 int bpf_do_misc_fixups(struct bpf_verifier_env *env);
+int bpf_insn_def32(struct bpf_insn *insn);
 
 #endif /* _LINUX_BPF_VERIFIER_H */
kernel/bpf/fixups.c +10 -7
index a0bddada7964..6125598d16a8 100644
@@ -44,15 +44,18 @@ static int insn_def_regno(const struct bpf_insn *insn)
 	}
 }
 
-/* Return TRUE if INSN has defined any 32-bit value explicitly. */
-static bool insn_has_def32(struct bpf_insn *insn)
+/*
+ * Return the 32-bit subregister defined by INSN, or -1 if INSN does not
+ * explicitly define a 32-bit value.
+ */
+int bpf_insn_def32(struct bpf_insn *insn)
 {
 	int dst_reg = insn_def_regno(insn);
 
-	if (dst_reg == -1)
-		return false;
+	if (dst_reg < 0 || bpf_is_reg64(insn, dst_reg, NULL, DST_OP))
+		return -1;
 
-	return !bpf_is_reg64(insn, dst_reg, NULL, DST_OP);
+	return dst_reg;
 }
 
 static int kfunc_desc_cmp_by_imm_off(const void *a, const void *b)
@@ -169,7 +172,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 	 * (cnt == 1) is taken or not. There is no guarantee INSN at OFF is the
 	 * original insn at old prog.
 	 */
-	data[off].zext_dst = insn_has_def32(insn + off + cnt - 1);
+	data[off].zext_dst = bpf_insn_def32(insn + off + cnt - 1) >= 0;
 
 	if (cnt == 1)
 		return;
@@ -181,7 +184,7 @@ static void adjust_insn_aux_data(struct bpf_verifier_env *env,
 	for (i = off; i < off + cnt - 1; i++) {
 		/* Expand insni[off]'s seen count to the patched range. */
 		data[i].seen = old_seen;
-		data[i].zext_dst = insn_has_def32(insn + i);
+		data[i].zext_dst = bpf_insn_def32(insn + i) >= 0;
 	}
 
 	/*
kernel/bpf/liveness.c +14 -0
index edfc2480b0f0..78351e4a6f25 100644
@@ -2232,6 +2232,7 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct bpf_insn *insns = env->prog->insnsi;
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
+	u64 pos, insn_pos;
 	int err = 0, i, j;
 	bool changed;
 
@@ -2291,9 +2292,18 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	}
 
 	for (i = 0; i < insn_cnt; ++i) {
+		int def32 = bpf_insn_def32(&insns[i]);
+		u32 out = state[i].out;
 		u32 in = state[i].in;
 
 		insn_aux[i].live_regs_before = mask_lo(in) | mask_hi(in);
+		/*
+		 * On architectures where 32-bit operations do not reset upper halves
+		 * of the registers, the verifier needs to zero extend a destination
+		 * register if an instruction defines a 32-bit subregister and the
+		 * upper half of that register is alive after the instruction.
+		 */
+		insn_aux[i].zext_dst = def32 >= 0 && (mask_hi(out) & BIT(def32));
 	}
 
 	if (env->log.level & BPF_LOG_LEVEL2) {
@@ -2310,7 +2320,11 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 				else
 					verbose(env, ".");
 			verbose(env, " ");
+			pos = env->log.end_pos;
 			bpf_verbose_insn(env, &insns[i]);
+			insn_pos = env->log.end_pos;
+			if (insn_aux[i].zext_dst)
+				verbose(env, "%*c; zext", bpf_vlog_alignment(insn_pos - pos), ' ');
 			verbose(env, "\n");
 			if (bpf_is_ldimm64(&insns[i]))
 				i++;
kernel/bpf/verifier.c +7 -106

309 lines not on this page · raw message

Signature · 1 line
2.53.0
#5 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Missing zero-extension for BPF_ADDR_SPACE_CAST allows out-of-bounds arena access on…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Missing zero-extension for BPF_ADDR_SPACE_CAST allows out-of-bounds arena access on architectures needing explicit zero-extension. --

--- Patch [3]: [PATCH bpf-next v3 3/5] bpf: infer zext_dst based on static register liveness analysis --- commit 740202f3328f4855ad10128ef188400086445fa9
Author: Eduard Zingerman <[email protected]>

bpf: infer zext_dst based on static register liveness analysis

This patch replaces dynamic subreg_def tracking with a static computation of zext_dst during register liveness analysis. The verifier now tracks the usage of upper and lower register halves separately to infer whether a 32-bit subregister definition requires explicit zero extension.

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
index 896bc5aaaaaff..e51cf5238f60a 100644

[ ... ]

kernel/bpf/verifier.c static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 15033

Quoted 10 lines
kernel/bpf/verifier.c +1 -4
@@ -15026,18 +14941,14 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
 				if (insn->imm) {
 					/* off == BPF_ADDR_SPACE_CAST */
 					mark_reg_unknown(env, regs, insn->dst_reg);
-					if (insn->imm == 1) { /* cast from as(1) to as(0) */
+					if (insn->imm == 1) /* cast from as(1) to as(0) */
 						dst_reg->type = PTR_TO_ARENA;
-						/* PTR_TO_ARENA is 32-bit */
-						dst_reg->subreg_def = env->insn_idx + 1;
-					}

[Severity: Critical]
Does this missing zero-extension for BPF_ADDR_SPACE_CAST allow out-of-bounds arena access on architectures needing explicit zero-extension?

The patch removed the dynamic dst_reg->subreg_def = env->insn_idx + 1 assignment here. Since a BPF_ADDR_SPACE_CAST instruction is encoded as BPF_ALU64 | BPF_MOV | BPF_X, the new bpf_insn_def32() function returns -1, which evaluates zext_dst to false in the static liveness analysis.

Because check_alu_op() handles the cast but does not manually set aux->zext_dst = true (unlike how it was handled in adjust_reg_min_max_vals()), the subsequent zero-extension insertion pass is skipped.

On architectures like s390x, wouldn't this leave uninitialized garbage in the upper 32 bits, allowing arbitrary kernel memory read/writes when the JIT adds this register to the 64-bit arena base?

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
 				} else if (insn->off == 0) {
 					/* case: R1 = R2
 					 * copy register state to dest reg
 					 */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author maintainer
#6 lore raw

On Sun, 2026-08-02 at 21:07 +0000, [email protected] wrote:

...

Quoted 6 lines
Quoted 6 lines
kernel/bpf/verifier.c +1 -1
@@ -15026,18 +14941,14 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
  				if (insn->imm) {
  					/* off == BPF_ADDR_SPACE_CAST */
  					mark_reg_unknown(env, regs, insn->dst_reg);
-					if (insn->imm == 1) { /* cast from as(1) to as(0) */
+					if (insn->imm == 1) /* cast from as(1) to as(0) */
Quoted 8 lines

kernel/bpf/verifier.c static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 15033

Quoted 4 lines
kernel/bpf/verifier.c +0 -3
  						dst_reg->type = PTR_TO_ARENA;
-						/* PTR_TO_ARENA is 32-bit */
-						dst_reg->subreg_def = env->insn_idx + 1;
-					}

[Severity: Critical]
Does this missing zero-extension for BPF_ADDR_SPACE_CAST allow out-of-bounds arena access on architectures needing explicit zero-extension?

Nope, in combination with changes to bpf_is_reg64() in the next patch it does not. bpf_is_reg64() handles BPF_ADDR_SPACE_CAST after that change. The commit log is not very clear about that, though.

...

author maintainer
#7 lore raw

PATCH v3 4/5 bpf: simplify the bpf_is_reg64()

In reply to Eduard Zingerman

After the previous commit bpf_is_reg64() is only used in a context where destination register's property is queried, and only for instructions for which insn_def_regno() >= 0.
Hence, simplify the function by:
- removing unused parameters;
- removing code paths considering BPF_JMP{,32} instructions;
- streamlining the condition expressions.

---
 include/linux/bpf_verifier.h |  1 -
 kernel/bpf/fixups.c          | 51 ++++++++++++++++++++++---
 kernel/bpf/verifier.c        | 90 --------------------------------------------
 3 files changed, 45 insertions(+), 97 deletions(-)
include/linux/bpf_verifier.h +0 -1
index 956243547446..39b0db2a4f17 100644
@@ -1626,7 +1626,6 @@ struct bpf_kfunc_desc_tab {
 };
 
 /* Functions exported from verifier.c, used by fixups.c */
-bool bpf_is_reg64(struct bpf_insn *insn, u32 regno, struct bpf_reg_state *reg, enum bpf_reg_arg_type t);
 void bpf_clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len);
 void bpf_mark_subprog_exc_cb(struct bpf_verifier_env *env, int subprog);
 bool bpf_allow_tail_call_in_subprogs(struct bpf_verifier_env *env);
kernel/bpf/fixups.c +45 -6

69 lines not on this page · raw message

kernel/bpf/verifier.c +0 -90

96 lines not on this page · raw message

Signature · 1 line
2.53.0
#8 lore raw

Re: [PATCH bpf-next v3 4/5] bpf: simplify the bpf_is_reg64()

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] bpf_is_reg64() fails to identify BPF_ADDR_SPACE_CAST operations with imm == 1U << 16 as 32-bit…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] bpf_is_reg64() fails to identify BPF_ADDR_SPACE_CAST operations with imm == 1U << 16 as 32-bit when the arena has BPF_F_NO_USER_CONV set, leading to missing zero-extension and possible sandbox escape.
- [Critical] Unconditional overwrite of zext_dst destroys manual overrides for PTR_TO_ARENA arithmetic, leading to missing zero-extensions and arbitrary kernel memory access. --

--- Patch [4]: [PATCH bpf-next v3 4/5] bpf: simplify the bpf_is_reg64() --- commit 4ece654ab9bc1fb0f5d2f2cc247cae7c94f2cb89
Author: Eduard Zingerman <[email protected]>

bpf: simplify the bpf_is_reg64()

This commit simplifies bpf_is_reg64() by removing unused parameters, removing BPF_JMP instructions handling, and streamlining conditions. It also moves the function to kernel/bpf/fixups.c.

Quoted 4 lines
kernel/bpf/fixups.c +0 -0
index 6125598d16a84..7b3510d46b377 100644

[ ... ]

Quoted 19 lines
kernel/bpf/fixups.c +15 -0
@@ -44,6 +44,49 @@ static int insn_def_regno(const struct bpf_insn *insn)
 	}
 }
 
+/*
+ * For use only in combination with insn_def_regno() >= 0.
+ * Returns TRUE if the destination register operates on 64-bit,
+ * otherwise return FALSE.
+ */
+static bool bpf_is_reg64(struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code);
+	u8 mode = BPF_MODE(insn->code);
+	u8 size = BPF_SIZE(insn->code);
+	u8 op = BPF_OP(insn->code);
+	bool mode_mem;
+
+	/* subregister endiness swap */
+	if ((class == BPF_ALU || class == BPF_ALU64) && op == BPF_END && insn->imm != 64)

kernel/bpf/fixups.c static int insn_def_regno(const struct bpf_insn *insn) · line 71

Quoted 10 lines
kernel/bpf/fixups.c +10 -0
+		return false;
+
+	/* w0 += 1 */
+	if (class == BPF_ALU && op != BPF_END)
+		return false;
+
+	/* cast from as(1) to as(0), PTR_TO_ARENA is 32-bit */
+	if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) &&
+	    insn->off == BPF_ADDR_SPACE_CAST && insn->imm == 1)
+		return false;

[Severity: Critical]
Does this logic miss identifying BPF_ADDR_SPACE_CAST operations with imm == 1U << 16 as 32-bit?

If the arena has BPF_F_NO_USER_CONV set, bpf_do_misc_fixups() rewrites all BPF_ADDR_SPACE_CAST instructions (including imm == 1U << 16) into a 32-bit MOV to explicitly clear the upper 32 bits.

However, since this falls through and evaluates to 64-bit, the static liveness analysis evaluates zext_dst to false. When bpf_opt_subreg_zext_lo32_rnd_hi32() runs, it skips inserting a BPF_ZEXT_REG zero-extension.

Could this leave the upper 32 bits un-zeroed on architectures requiring explicit zero-extension for 32-bit operations (e.g., s390x) or that sign-extend them (e.g., riscv64), leading to out-of-bounds memory accesses?

Quoted 18 lines
kernel/bpf/fixups.c +18 -0
+
+	/* non 64-bit, non signed extended loads */
+	mode_mem = mode == BPF_MEM || mode == BPF_PROBE_MEM || mode == BPF_PROBE_MEM32;
+	if (class == BPF_LDX && mode_mem && size != BPF_DW)
+		return false;
+
+	/* atomics, see insn_def_regno() */
+	if (class == BPF_STX && size != BPF_DW)
+		return false;
+
+	/* both LD_IND and LD_ABS return 32-bit data. */
+	if (class == BPF_LD && (mode == BPF_IND || mode == BPF_ABS))
+		return false;
+
+	/* Conservatively return true at default. */
+	return true;
+}
+

kernel/bpf/fixups.c static int insn_def_regno(const struct bpf_insn *insn)

Quoted 10 lines
kernel/bpf/fixups.c +10 -0
+/*
+ * Return the 32-bit subregister defined by INSN, or -1 if INSN does not
+ * explicitly define a 32-bit value.
+ */
+int bpf_insn_def32(struct bpf_insn *insn)
+{
+	int dst_reg = insn_def_regno(insn);
+
+	if (dst_reg < 0 || bpf_is_reg64(insn))
+		return -1;

[Severity: Critical]
Would this unconditional behavior inadvertently overwrite manual overrides for PTR_TO_ARENA arithmetic?

During verifier simulation, adjust_reg_min_max_vals() manually forces aux->zext_dst = true on arena arithmetic to ensure zero-extension is applied by the JIT.

However, the static liveness pass in bpf_compute_live_registers() runs afterwards and does an unconditional assignment based on def32. Because bpf_is_reg64() evaluates 64-bit ALU operations as 64-bit, def32 here becomes -1.

Could this cause the liveness pass to blindly clear the manually set zext_dst flag to false? If bpf_do_misc_fixups() then rewrites the instruction to a 32-bit operation, it might miss the necessary zero extensions, potentially allowing an escape from the arena boundaries.

Quoted 3 lines
kernel/bpf/fixups.c +3 -0
+
+	return dst_reg;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author maintainer
#9 lore raw

Re: [PATCH bpf-next v3 4/5] bpf: simplify the bpf_is_reg64()

Quoted 56 lines · On Sun, 2026-08-02 at 21:26 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] bpf_is_reg64() fails to identify BPF_ADDR_SPACE_CAST operations with imm == 1U << 16 as 32-bit when the arena has BPF_F_NO_USER_CONV set, leading to missing zero-extension and possible sandbox escape.
- [Critical] Unconditional overwrite of zext_dst destroys manual overrides for PTR_TO_ARENA arithmetic, leading to missing zero-extensions and arbitrary kernel memory access. --

--- Patch [4]: [PATCH bpf-next v3 4/5] bpf: simplify the bpf_is_reg64() --- commit 4ece654ab9bc1fb0f5d2f2cc247cae7c94f2cb89
Author: Eduard Zingerman <[email protected]>

bpf: simplify the bpf_is_reg64()

This commit simplifies bpf_is_reg64() by removing unused parameters, removing BPF_JMP instructions handling, and streamlining conditions. It also moves the function to kernel/bpf/fixups.c.

Quoted 4 lines
kernel/bpf/fixups.c +0 -0
index 6125598d16a84..7b3510d46b377 100644

[ ... ]

Quoted 19 lines
kernel/bpf/fixups.c +0 -0
@@ -44,6 +44,49 @@ static int insn_def_regno(const struct bpf_insn *insn)
  	}
  }

kernel/bpf/fixups.c +15 -0
+/*
+ * For use only in combination with insn_def_regno() >= 0.
+ * Returns TRUE if the destination register operates on 64-bit,
+ * otherwise return FALSE.
+ */
+static bool bpf_is_reg64(struct bpf_insn *insn)
+{
+	u8 class = BPF_CLASS(insn->code);
+	u8 mode = BPF_MODE(insn->code);
+	u8 size = BPF_SIZE(insn->code);
+	u8 op = BPF_OP(insn->code);
+	bool mode_mem;
+
+	/* subregister endiness swap */
+	if ((class == BPF_ALU || class == BPF_ALU64) && op == BPF_END && insn->imm != 64)

kernel/bpf/fixups.c static int insn_def_regno(const struct bpf_insn *insn) · line 71

Quoted 10 lines
kernel/bpf/fixups.c +10 -0
+		return false;
+
+	/* w0 += 1 */
+	if (class == BPF_ALU && op != BPF_END)
+		return false;
+
+	/* cast from as(1) to as(0), PTR_TO_ARENA is 32-bit */
+	if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) &&
+	    insn->off == BPF_ADDR_SPACE_CAST && insn->imm == 1)
+		return false;

[Severity: Critical]
Does this logic miss identifying BPF_ADDR_SPACE_CAST operations with imm == 1U << 16 as 32-bit?

Quoted 3 lines

If the arena has BPF_F_NO_USER_CONV set, bpf_do_misc_fixups() rewrites all BPF_ADDR_SPACE_CAST instructions (including imm == 1U << 16) into a 32-bit MOV to explicitly clear the upper 32 bits.

I think this is a pre-existing bug.
I'll add a test.

...

Quoted 6 lines
Quoted 6 lines
kernel/bpf/fixups.c +6 -0
+/*
+ * Return the 32-bit subregister defined by INSN, or -1 if INSN does not
+ * explicitly define a 32-bit value.
+ */
+int bpf_insn_def32(struct bpf_insn *insn)
+{
Quoted 8 lines

kernel/bpf/fixups.c

Quoted 4 lines
kernel/bpf/fixups.c +4 -0
+	int dst_reg = insn_def_regno(insn);
+
+	if (dst_reg < 0 || bpf_is_reg64(insn))
+		return -1;

[Severity: Critical]
Would this unconditional behavior inadvertently overwrite manual overrides for PTR_TO_ARENA arithmetic?

No it won't.
convert_ctx_access() downgrades those operations to BPF_ALU. There is also a test to check this.

...

author maintainer
#10 lore raw

PATCH v3 5/5 selftests/bpf: verify zext_dst annotations for various instructions

In reply to Eduard Zingerman

Includes the following test cases:
- a test showing that zero extension flags do not propagate through state pruning in the unpatched kernel.
- a 32-bit subregister consumed by MOV32 and ALU32 operations (never zext'ed);
- a 64-bit MOV (never zext'ed);
- a narrow (32-bit) BPF_LDX load whose result is read as 64-bit;
- 32-bit atomic fetch_add and cmpxchg whose result is read as 64-bit;
- a CFG case where a 32-bit definition's upper half is used only on one of two branches;
- no zext for dead registers;
- LD_ABS defines only lower 32 bits, hence needs zext when the result is used as 64-bits;
- helper, kfunc and subprogram parameters are considered to use full 64 bits;
- a 32-bit subregister consumed by JMP32 (X/K) operations;
- a 32-bit subregister consumed by JMP (X/K) operations;
- a 64-bit register consumed by both JMP and JMP32 operations (never zext'ed);
- ALU64 operation on arena pointer;
- memory loads using BPF_PROBE_MEM instructions.

---
 tools/testing/selftests/bpf/prog_tests/verifier.c |   2 +
 tools/testing/selftests/bpf/progs/verifier_zext.c | 388 ++++++++++++++++++++++
 2 files changed, 390 insertions(+)
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index be97f6887f0e..fba562c81969 100644
@@ -124,6 +124,7 @@
 #include "verifier_jit_inline.skel.h"
 #include "irq.skel.h"
 #include "verifier_ctx_ptr_param.skel.h"
+#include "verifier_zext.skel.h"
 
 #define MAX_ENTRIES 11
 
@@ -277,6 +278,7 @@ void test_irq(void)			      { RUN(irq); }
 void test_verifier_mtu(void)		      { RUN(verifier_mtu); }
 void test_verifier_jit_inline(void)               { RUN(verifier_jit_inline); }
 void test_verifier_ctx_ptr_param(void)       { RUN(verifier_ctx_ptr_param); }
+void test_verifier_zext(void)                 { RUN_TESTS(verifier_zext); }
 
 static int init_test_val_map(struct bpf_object *obj, char *map_name)
 {
tools/testing/selftests/bpf/progs/verifier_zext.c new file +388 -0

388 lines not on this page · raw message

Signature · 1 line
2.53.0