Fix fastcall rewrite with indirect stack accesses
Subsystems (4)
Kumar Kartikeya Dwivedi
maintainercover Fix fastcall rewrite with indirect stack accesses
When a bpf_fastcall call is inlined, the verifier removes the spill/fill pairs around it and shrinks the stack frame to exclude their slots. Two kinds of stack access skip the check that must disable this rewrite, so the program can then access kernel stack outside its frame.
Patch 1 checks helper and kfunc stack buffers, which BPF CI reported after commit 5da4a9f26fca ("bpf: Preserve stack initialization for generic output buffers"). Patch 2 makes a callee's load from its caller's stack check the caller's frame. Patch 3 adds tests.
Kumar Kartikeya Dwivedi (3):
bpf: Check fastcall contract for helper and kfunc stack buffers
bpf: Check fastcall contract of the frame a stack read targets selftests/bpf: Test fastcall rewrite with indirect stack accesses
kernel/bpf/verifier.c | 10 +- .../bpf/progs/verifier_bpf_fastcall.c | 161 ++++++++++++++++++ 2 files changed, 169 insertions(+), 2 deletions(-)
base-commit: 91f8613d95ad8cd99d8baf094806d1ef98bc6380
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerPATCH 1/3 bpf: Check fastcall contract for helper and kfunc stack buffers
When the verifier inlines a bpf_fastcall helper or kfunc, it removes the spill/fill pairs that clang emitted around the call and lowers the subprogram's stack depth so that their slots are no longer part of the frame. This is only safe if nothing else accesses those slots or any slot below them. check_fastcall_stack_contract() enforces this: an access to that region from outside a fastcall pattern disables the rewrite for the subprogram.
Stack loads and stores call this check, but stack buffers passed to helpers and kfuncs are validated by check_stack_range_initialized(), which does not. If such a buffer is the only other access to the region, the rewrite is still applied, the JIT reserves a frame that does not contain the buffer, and the helper reads or writes live kernel stack below the program's frame. For example, without CAP_PERFMON:
*(u64 *)(r10 - 8) = r1; call bpf_get_smp_processor_id; r1 = *(u64 *)(r10 - 8); r2 = 0; r3 = r10; r3 += -64; r4 = 8; call bpf_skb_load_bytes;
is accepted with a stack depth of 0 instead of 64.
Some buffers got the check indirectly. For constant-sized outputs in raw mode, mark_raw_stack() marks the buffer initialized through ordinary byte stores, which call the check. Commit 5da4a9f26fca ("bpf: Preserve stack initialization for generic output buffers") limited raw mode to programs that may read uninitialized stack, so outputs of programs without CAP_PERFMON lost the check, as in the example above. Buffers that never used raw mode have lacked the check since fastcall support was added: helper inputs such as a map key whose only store is a fastcall spill, variable-sized outputs, and buffers at a variable stack offset.
Call check_fastcall_stack_contract() from check_stack_range_initialized() for every nonempty access, before the raw-mode return. Use the frame that owns the buffer, so that a callee passing a pointer into its caller's stack disables the caller's rewrite. Zero-sized accesses touch no stack and leave the rewrite enabled. Variable-offset stack reads now get the check through check_stack_range_initialized(), so drop the separate call in check_stack_read_var_off().
--- kernel/bpf/verifier.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/bpf/verifier.c +7 -1
@@ -4138,7 +4138,6 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg dst_regno);if (err)return err; -check_fastcall_stack_contract (env, ptr_state, env->insn_idx, min_off);return 0 ; }
@@ -7041,6 +7040,13 @@ static int check_stack_range_initialized( max_off =reg_smax (reg) + off; } +/* +* Helpers and kfuncs can access stack slots without going through the +* regular stack read/write paths, including when raw mode is disabled. +*/ +if (access_size) +check_fastcall_stack_contract (env, state, env->insn_idx, min_off); +/* Unprivileged outputs retain each byte's initialization state. */ if (raw_mode) { meta->arg_raw_mem.size[arg_slot] = access_size;
Signature · 1 line
2.53.0
Re: [PATCH bpf-next v1 1/3] bpf: Check fastcall contract for helper and kfunc stack buffers
kernel/bpf/verifier.c static int check_stack_range_initialized( · line 7048
Quoted 6 lines · Kumar Kartikeya Dwivedi
kernel/bpf/verifier.c +6
-0+/* +* Helpers and kfuncs can access stack slots without going through the +* regular stack read/write paths, including when raw mode is disabled. +*/ +if (access_size) +check_fastcall_stack_contract (env, state, env->insn_idx, min_off);
can we do it once in check_stack_access_within_bounds() ? Both check_mem_access() and check_stack_range_initialized() call it first, and it already has the frame that owns the slot and min_off. Then the other four check_fastcall_stack_contract() calls can go and patch 2 is not needed.
Kumar Kartikeya Dwivedi
maintainerPATCH 2/3 bpf: Check fastcall contract of the frame a stack read targets
In reply to Kumar Kartikeya Dwivedicheck_stack_read_fixed_off() applies the fastcall contract check to the current frame, even when the pointer targets a caller's stack. A callee can therefore read a caller's fastcall spill slot without disabling the caller's rewrite:
caller: r1 = 1; *(u64 *)(r10 - 8) = r1; call bpf_get_smp_processor_id; r1 = *(u64 *)(r10 - 8); r1 = r10; r1 += -8; call callee; callee: r0 = *(u64 *)(r1 + 0);
The caller's spill and fill are removed and its stack depth drops to 0, but the verifier still tracks the slot as holding the spilled constant. The callee's load then reads kernel stack outside the caller's frame, while the verifier assumes r0 is 1.
Check the contract of reg_state, the frame that owns the slot, as the stack write paths already do.
--- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/verifier.c +1 -1
@@ -3951,7 +3951,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, reg = ®_state->stack[spi].spilled_ptr;mark_stack_slot_scratched (env, spi); -check_fastcall_stack_contract (env, state, env->insn_idx, off); +check_fastcall_stack_contract (env, reg_state, env->insn_idx, off);/* * Refine the in-progress load record's origin to the source stack slot.
Signature · 1 line
2.53.0
Kumar Kartikeya Dwivedi
maintainerPATCH 3/3 selftests/bpf: Test fastcall rewrite with indirect stack accesses
In reply to Kumar Kartikeya DwivediAdd tests where a helper stack buffer, or a load through a pointer into another frame, overlaps the slots of a fastcall spill/fill pair. The rewrite must not be applied in these cases, so check that the spill and fill remain in the translated program and that the final stack depth still covers the accessed slots. Cover:
- a constant-sized uninitialized output without CAP_PERFMON;
- the same output in the caller's stack, passed to a helper by a callee;
- a helper input whose only initialization is the fastcall spill;
- a callee load from the caller's fastcall spill slot.
Also add a zero-sized buffer, for which the rewrite must still be applied.
The tests only load the programs and inspect the verifier log and the translated instructions. Do not run them: without the fixes, the verifier accepts programs that access memory outside their stack frame.
--- .../bpf/progs/verifier_bpf_fastcall.c | 161 ++++++++++++++++++ 1 file changed, 161 insertions(+)
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +161 -0
@@ -502,6 +502,167 @@ __naked void bad_helper_write(void) : __clobber_all); } +/* +* Load these programs without running them: removing the fastcall spills must +* not shrink the stack below an output buffer that has no explicit accesses. +*/ +SEC("tc") +__log_level(4) +__msg_unpriv("subprog 0 (helper_uninit_stack) main {{.*}} stack 64") +__xlated_unpriv("*(u64 *)(r10 -8) = r1") +__xlated_unpriv("...") +__xlated_unpriv("r1 = *(u64 *)(r10 -8)") +__caps_unpriv(CAP_BPF | CAP_NET_ADMIN) +__success_unpriv +__naked void helper_uninit_stack (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = r1;" +"call %[bpf_get_smp_processor_id];" +"r1 = *(u64 *)(r10 - 8);" +"r2 = 0;" +"r3 = r10;" +"r3 += -64;" +"r4 = 8;" +"call %[bpf_skb_load_bytes];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_smp_processor_id), +__imm (bpf_skb_load_bytes) + : __clobber_all); +} + +static __used __naked void helper_uninit_stack_callee (void ) +{ + asmvolatile ( +"r3 = r2;" +"r2 = 0;" +"r4 = 8;" +"call %[bpf_skb_load_bytes];" +"exit;" + : + : __imm(bpf_skb_load_bytes) + : __clobber_all); +} + +SEC("tc") +__log_level(4) +__msg_unpriv("subprog 0 (helper_uninit_stack_caller) main {{.*}} stack 64") +__xlated_unpriv("*(u64 *)(r10 -8) = r1") +__xlated_unpriv("...") +__xlated_unpriv("r1 = *(u64 *)(r10 -8)") +__caps_unpriv(CAP_BPF | CAP_NET_ADMIN) +__success_unpriv +__naked void helper_uninit_stack_caller (void ) +{ + asmvolatile ( +"*(u64 *)(r10 - 8) = r1;" +"call %[bpf_get_smp_processor_id];" +"r1 = *(u64 *)(r10 - 8);" +"r2 = r10;" +"r2 += -64;" +"call helper_uninit_stack_callee;" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, __u64); +} fastcall_map SEC(".maps"); + +SEC("tc") +__log_level(4) +__msg("subprog 0 (helper_reads_fastcall_spill) main {{.*}} stack 8") +__xlated("*(u64 *)(r10 -8) = r1") +__xlated("...") +__xlated("r1 = *(u64 *)(r10 -8)") +__success +__naked void helper_reads_fastcall_spill (void ) +{ + asmvolatile ( +"r1 = 0;" +"*(u64 *)(r10 - 8) = r1;" +"call %[bpf_get_smp_processor_id];" +"r1 = *(u64 *)(r10 - 8);" +"r1 = %[fastcall_map] ll;" +"r2 = r10;" +"r2 += -8;" +"call %[bpf_map_lookup_elem];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_smp_processor_id), +__imm (bpf_map_lookup_elem), +__imm_addr (fastcall_map) + : __clobber_all); +} + +static __used __naked void read_caller_stack_callee (void ) +{ + asmvolatile ( +"r0 = *(u64 *)(r1 + 0);" +"exit;" + ::: __clobber_all); +} + +SEC("raw_tp") +__log_level(4) +__msg("subprog 0 (callee_reads_fastcall_spill) main {{.*}} stack 8") +__xlated("*(u64 *)(r10 -8) = r1") +__xlated("...") +__xlated("r1 = *(u64 *)(r10 -8)") +__success +__naked void callee_reads_fastcall_spill (void ) +{ + asmvolatile ( +"r1 = 1;" +"*(u64 *)(r10 - 8) = r1;" +"call %[bpf_get_smp_processor_id];" +"r1 = *(u64 *)(r10 - 8);" +"r1 = r10;" +"r1 += -8;" +"call read_caller_stack_callee;" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +SEC("raw_tp") +__arch_x86_64 +__log_level(4) +__msg("subprog 0 (helper_zero_size_buffer) main {{.*}} stack 0") +__xlated("0: r1 = 1") +__xlated("1: r0 =") +__success +__naked void helper_zero_size_buffer (void ) +{ + asmvolatile ( +"r1 = 1;" +"*(u64 *)(r10 - 8) = r1;" +"call %[bpf_get_smp_processor_id];" +"r1 = *(u64 *)(r10 - 8);" +"r1 = r10;" +"r1 += -64;" +"r2 = 0;" +"r3 = 0;" +"call %[bpf_probe_read_kernel];" +"r0 = 0;" +"exit;" + : + : __imm(bpf_get_smp_processor_id), +__imm (bpf_probe_read_kernel) + : __clobber_all); +} + SEC("raw_tp") __arch_x86_64/* main, not patched */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH bpf-next v1 3/3] selftests/bpf: Test fastcall rewrite with indirect stack accesses
This isn't a bug, but would it read better to scope this comment to the first two tests and give each of the remaining cases its own one-liner, the way the jump-pattern tests further down in this…
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +0
-0
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c __naked void bad_helper_write(void) · line 511
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +7
-0@@ -502,6 +502,167 @@ __naked void bad_helper_write(void) : __clobber_all); } +/* +* Load these programs without running them: removing the fastcall spills must +* not shrink the stack below an output buffer that has no explicit accesses. +*/ +SEC("tc") +__log_level(4) +__msg_unpriv ("subprog 0 (helper_uninit_stack) main {{.*}} stack 64" )
This isn't a bug, but would it read better to scope this comment to the first two tests and give each of the remaining cases its own one-liner, the way the jump-pattern tests further down in this file do? As written the comment says the stack must not shrink, which is the opposite of what helper_zero_size_buffer asserts (it expects "stack 0").
The comment accurately describes helper_uninit_stack and helper_uninit_stack_caller, but helper_reads_fastcall_spill is a map key whose only initialization is the fastcall spill, callee_reads_fastcall_spill is a load from the caller's frame, and helper_zero_size_buffer checks that the rewrite is still applied and the stack does shrink to 0.
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c __naked void bad_helper_write(void) · line 517
Quoted 6 lines
tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +6
-0+__xlated_unpriv("*(u64 *)(r10 -8) = r1") +__xlated_unpriv("...") +__xlated_unpriv("r1 = *(u64 *)(r10 -8)") +__caps_unpriv(CAP_BPF | CAP_NET_ADMIN) +__success_unpriv +__naked void helper_uninit_stack (void )
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/35840155529