Don't trust r0 bounds after BPF to BPF calls with abnormal returns
Subsystems (4)
v2 cover Don't trust r0 bounds after BPF to BPF calls with abnormal returns
A BPF function can return before its exit instruction: LD_ABS, LD_IND, and tail_call() can all cause it to return prematurely.
When such a function is called by another BPF function, the verifier doesn't take this into account when calculating the bounds of r0 in the caller after the callee returns.
---
Changes in v2:
- Handle LD_ABS and LD_IND, not just tail_call()
- Split tests out
- Use inline asm for tests
---
Arthur Fabre (2):
bpf: Don't trust r0 bounds after BPF to BPF calls with abnormal
returns
selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal
return
kernel/bpf/verifier.c | 18 ++-- .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_abnormal_ret.c | 88 +++++++++++++++++++ 3 files changed, 102 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c
Signature · 1 line
2.34.1
PATCH v2 1/2 bpf: Don't trust r0 bounds after BPF to BPF calls with abnormal returns
When making BPF to BPF calls, the verifier propagates register bounds info for r0 from the callee to the caller.
For example loading:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
static __attribute__((noinline)) int callee(struct xdp_md *ctx)
{
int ret;
asm volatile("%0 = 23" : "=r"(ret));
return ret;
}
static SEC("xdp") int caller(struct xdp_md *ctx)
{
int res = callee(ctx);
if (res == 23) {
return XDP_PASS;
}
return XDP_DROP;
}The verifier logs:
func#0 @0
func#1 @6
0: R1=ctx() R10=fp0
; int res = callee(ctx); @ test.c:15
0: (85) call pc+5
caller:
R10=fp0
callee:
frame1: R1=ctx() R10=fp0
6: frame1: R1=ctx() R10=fp0
; asm volatile("%0 = 23" : "=r"(ret)); @ test.c:9
6: (b7) r0 = 23 ; frame1: R0_w=23
; return ret; @ test.c:10
7: (95) exit
returning from callee:
frame1: R0_w=23 R1=ctx() R10=fp0
to caller at 1:
R0_w=23 R10=fp0
from 7 to 1: R0_w=23 R10=fp0
; int res = callee(ctx); @ test.c:15
1: (bc) w1 = w0 ; R0_w=23 R1_w=23
2: (b4) w0 = 2 ; R0_w=2
; @ test.c:0
3: (16) if w1 == 0x17 goto pc+1
3: R1_w=23
; } @ test.c:20
5: (95) exit
processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0And correctly tracks R0_w=23 from the callee through to the caller. This lets it completely prune the res != 23 branch, skipping over instruction 4.
But this isn't sound if the callee can return "abnormally" before an exit instruction:
- If LD_ABS or LD_IND try to access data beyond the end of the packet, the callee returns 0 directly.
- If a tail_call succeeds, the return value of the tail called program will be returned directly.
We can't know what the bounds of r0 will be.
The verifier still incorrectly tracks the bounds of r0 in these cases. Loading:
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
struct {
__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
__uint(max_entries, 1);
__uint(key_size, sizeof(__u32));
__uint(value_size, sizeof(__u32));
} tail_call_map SEC(".maps");
static __attribute__((noinline)) int callee(struct xdp_md *ctx)
{
bpf_tail_call(ctx, &tail_call_map, 0);
int ret;
asm volatile("%0 = 23" : "=r"(ret));
return ret;
}
static SEC("xdp") int caller(struct xdp_md *ctx)
{
int res = callee(ctx);
if (res == 23) {
return XDP_PASS;
}
return XDP_DROP;
}The verifier logs:
func#0 @0
func#1 @6
0: R1=ctx() R10=fp0
; int res = callee(ctx); @ test.c:24
0: (85) call pc+5
caller:
R10=fp0
callee:
frame1: R1=ctx() R10=fp0
6: frame1: R1=ctx() R10=fp0
; bpf_tail_call(ctx, &tail_call_map, 0); @ test.c:15
6: (18) r2 = 0xffff8a9c82a75800 ; frame1: R2_w=map_ptr(map=tail_call_map,ks=4,vs=4)
8: (b4) w3 = 0 ; frame1: R3_w=0
9: (85) call bpf_tail_call#12
10: frame1:
; asm volatile("%0 = 23" : "=r"(ret)); @ test.c:18
10: (b7) r0 = 23 ; frame1: R0_w=23
; return ret; @ test.c:19
11: (95) exit
returning from callee:
frame1: R0_w=23 R10=fp0
to caller at 1:
R0_w=23 R10=fp0
from 11 to 1: R0_w=23 R10=fp0
; int res = callee(ctx); @ test.c:24
1: (bc) w1 = w0 ; R0_w=23 R1_w=23
2: (b4) w0 = 2 ; R0=2
; @ test.c:0
3: (16) if w1 == 0x17 goto pc+1
3: R1=23
; } @ test.c:29
5: (95) exit
processed 10 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1It still prunes the res != 23 branch, skipping over instruction 4. But the tail called program can return any value.
Aside from pruning incorrect branches, this can also be used to read and write arbitrary memory by using r0 as a index.
--- kernel/bpf/verifier.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/bpf/verifier.c +12 -6
@@ -10314,6 +10314,11 @@ static bool retval_range_within(struct bpf_retval_range range, const struct bpf_return range.minval <= reg->smin_value && reg->smax_value <= range.maxval; } +static bool has_abnormal_return (struct bpf_subprog_info *subprog_info) +{ +return subprog_info->has_ld_abs || subprog_info->has_tail_call; +} +static int prepare_func_exit (struct bpf_verifier_env *env,int *insn_idx) {struct bpf_verifier_state *state = env->cur_state, *prev_st;
@@ -10359,6 +10364,10 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) *insn_idx, callee->callsite);return -EFAULT ; } + }else if (has_abnormal_return ( + &env->subprog_info[state->frame[state->curframe]->subprogno])) { +/* callee can return before exit instruction, r0 could hold anything */ +__mark_reg_unknown (env, &caller->regs[BPF_REG_0 ]); }else {/* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0 ] = *r0;
@@ -16881,17 +16890,14 @@ static int check_cfg(struct bpf_verifier_env *env)return ret; } +static int check_abnormal_return (struct bpf_verifier_env *env) {int i;for (i =1 ; i < env->subprog_cnt; i++) { -if (env->subprog_info[i].has_ld_abs) { -verbose (env,"LD_ABS is not allowed in subprogs without BTF\n" ); -return -EINVAL ; - } -if (env->subprog_info[i].has_tail_call) { -verbose (env,"tail_call is not allowed in subprogs without BTF\n" ); +if (has_abnormal_return (&env->subprog_info[i])) { +verbose (env,"LD_ABS/tail_call is not allowed in subprogs without BTF\n" );return -EINVAL ; } }
Signature · 1 line
2.34.1
Re: [PATCH bpf v2 1/2] bpf: Don't trust r0 bounds after BPF to BPF calls with abnormal returns
Quoted 138 lines · Arthur Fabre
When making BPF to BPF calls, the verifier propagates register bounds info for r0 from the callee to the caller.
For example loading:
#include <linux/bpf.h> #include <bpf/bpf_helpers.h> static __attribute__((noinline)) int callee(struct xdp_md *ctx) { int ret; asm volatile("%0 = 23" : "=r"(ret)); return ret; } static SEC("xdp") int caller(struct xdp_md *ctx) { int res = callee(ctx); if (res == 23) { return XDP_PASS; } return XDP_DROP; }The verifier logs:
func#0 @0 func#1 @6 0: R1=ctx() R10=fp0 ; int res = callee(ctx); @ test.c:15 0: (85) call pc+5 caller: R10=fp0 callee: frame1: R1=ctx() R10=fp0 6: frame1: R1=ctx() R10=fp0 ; asm volatile("%0 = 23" : "=r"(ret)); @ test.c:9 6: (b7) r0 = 23 ; frame1: R0_w=23 ; return ret; @ test.c:10 7: (95) exit returning from callee: frame1: R0_w=23 R1=ctx() R10=fp0 to caller at 1: R0_w=23 R10=fp0 from 7 to 1: R0_w=23 R10=fp0 ; int res = callee(ctx); @ test.c:15 1: (bc) w1 = w0 ; R0_w=23 R1_w=23 2: (b4) w0 = 2 ; R0_w=2 ; @ test.c:0 3: (16) if w1 == 0x17 goto pc+1 3: R1_w=23 ; } @ test.c:20 5: (95) exit processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0And correctly tracks R0_w=23 from the callee through to the caller. This lets it completely prune the res != 23 branch, skipping over instruction 4.
But this isn't sound if the callee can return "abnormally" before an exit instruction:
- IfLD_ABSorLD_INDtry to access data beyond the end of the packet, the callee returns 0 directly.
- If atail_callsucceeds, the return value of the tail called program will be returned directly.
We can't know what the bounds of r0 will be.The verifier still incorrectly tracks the bounds of r0 in these cases. Loading:
#include <linux/bpf.h> #include <bpf/bpf_helpers.h> struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY); __uint(max_entries, 1); __uint(key_size, sizeof(__u32)); __uint(value_size, sizeof(__u32)); } tail_call_map SEC(".maps"); static __attribute__((noinline)) int callee(struct xdp_md *ctx) { bpf_tail_call(ctx, &tail_call_map, 0); int ret; asm volatile("%0 = 23" : "=r"(ret)); return ret; } static SEC("xdp") int caller(struct xdp_md *ctx) { int res = callee(ctx); if (res == 23) { return XDP_PASS; } return XDP_DROP; }The verifier logs:
func#0 @0 func#1 @6 0: R1=ctx() R10=fp0 ; int res = callee(ctx); @ test.c:24 0: (85) call pc+5 caller: R10=fp0 callee: frame1: R1=ctx() R10=fp0 6: frame1: R1=ctx() R10=fp0 ; bpf_tail_call(ctx, &tail_call_map, 0); @ test.c:15 6: (18) r2 = 0xffff8a9c82a75800 ; frame1: R2_w=map_ptr(map=tail_call_map,ks=4,vs=4) 8: (b4) w3 = 0 ; frame1: R3_w=0 9: (85) call bpf_tail_call#12 10: frame1: ; asm volatile("%0 = 23" : "=r"(ret)); @ test.c:18 10: (b7) r0 = 23 ; frame1: R0_w=23 ; return ret; @ test.c:19 11: (95) exit returning from callee: frame1: R0_w=23 R10=fp0 to caller at 1: R0_w=23 R10=fp0 from 11 to 1: R0_w=23 R10=fp0 ; int res = callee(ctx); @ test.c:24 1: (bc) w1 = w0 ; R0_w=23 R1_w=23 2: (b4) w0 = 2 ; R0=2 ; @ test.c:0 3: (16) if w1 == 0x17 goto pc+1 3: R1=23 ; } @ test.c:29 5: (95) exit processed 10 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1It still prunes the res != 23 branch, skipping over instruction 4. But the tail called program can return any value.
Quoted 7 lines
Aside from pruning incorrect branches, this can also be used to read and write arbitrary memory by using r0 as a index.
Fixes e411901c0b77 ("bpf: allow for tailcalls in BPF subprograms for x64 JIT")Signed-off-by Arthur Fabre <[email protected]>---
[...]
kernel/bpf/verifier.c static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · line 10368
Quoted 6 lines
kernel/bpf/verifier.c +2
-0@@ -10359,6 +10364,10 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) *insn_idx, callee->callsite);return -EFAULT ; } + }else if (has_abnormal_return ( + &env->subprog_info[state->frame[state->curframe]->subprogno])) {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nit: this is 'callee'
kernel/bpf/verifier.c static int check_cfg(struct bpf_verifier_env *env) · line 16893
Quoted 10 lines
kernel/bpf/verifier.c +3
-0+/* callee can return before exit instruction, r0 could hold anything */ +__mark_reg_unknown (env, &caller->regs[BPF_REG_0 ]); }else {/* return to the caller whatever r0 had in the callee */ caller->regs[BPF_REG_0 ] = *r0;@@ -16881,17 +16890,14 @@ static int check_cfg(struct bpf_verifier_env *env)return ret; } +
Nit: this empty line is not needed.
[...]
PATCH v2 2/2 selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
In reply to Arthur FabreTest the bounds of r0 aren't known by the verifier in all three cases where a callee can abnormally return.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../bpf/progs/verifier_abnormal_ret.c | 88 +++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -3,6 +3,7 @@#include <test_progs.h> #include "cap_helpers.h" +#include "verifier_abnormal_ret.skel.h" #include "verifier_and.skel.h" #include "verifier_arena.skel.h" #include "verifier_arena_large.skel.h"
@@ -133,6 +134,7 @@ static void run_tests_aux(const char *skel_name,#define RUN (skel) run_tests_aux(#skel, skel##__elf_bytes, NULL) +void test_verifier_abnormal_ret (void ) {RUN (verifier_abnormal_ret); }void test_verifier_and (void ) {RUN (verifier_and); }void test_verifier_arena (void ) {RUN (verifier_arena); }void test_verifier_arena_large (void ) {RUN (verifier_arena_large); }
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +88 -0
88 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
Quoted 5 lines · Arthur Fabre
Test the bounds of r0 aren't known by the verifier in all three cases where a callee can abnormally return.
Signed-off-by Arthur Fabre <[email protected]>---
[...]
Quoted 5 lines
+++
b/tools/testing/selftests/bpf/progs/verifier_abnormal_ret.ctools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +3
-0@@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h>
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c line 13
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+#include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#define TEST (NAME ,CALLEE ) \ + SEC("socket") \ + __description("abnormal_ret: " #NAME) \ + __failure __msg("math between ctx pointer and register with unbounded min value") \ + __naked void check_abnormal_ret_##NAME(void) \ + { \
Nit: this one and 'callee_tail_call' could be plain C.
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+ asm volatile(" \ + r6 = r1; \ + call " #CALLEE "; \ + r6 += r0; \ + r0 = 0; \ + exit; \ + " : \ + : \ + : __clobber_all); \ + }
[...]
Quoted 15 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +15
-0+static __naked __noinline __used +int callee_tail_call(void) +{ + asmvolatile (" \ +r2 = %[map_prog] ll; \ +r3 = 0; \ +call %[bpf_tail_call]; \ +r0 = 0; \ +exit; \ +" : + : __imm(bpf_tail_call), __imm_addr(map_prog) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
Re: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
Quoted 1 line · Eduard Zingerman
On Fri, 2024-12-13 at 22:27 +0100, Arthur Fabre wrote:
[...]
Quoted 31 lines
Quoted 5 lines
+++
b/tools/testing/selftests/bpf/progs/verifier_abnormal_ret.ctools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +3
-0@@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c line 13
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+#include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#define TEST (NAME ,CALLEE ) \ + SEC("socket") \ + __description("abnormal_ret: " #NAME) \ + __failure __msg("math between ctx pointer and register with unbounded min value") \ + __naked void check_abnormal_ret_##NAME(void) \ + { \Nit: this one and '
callee_tail_call' could be plain C.Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+ asm volatile(" \ + r6 = r1; \ + call " #CALLEE "; \ + r6 += r0; \ + r0 = 0; \ + exit; \ + " : \ + : \ + : __clobber_all); \ + }[...]
Quoted 15 lines
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +5
-0+static __naked __noinline __used +int callee_tail_call(void) +{ + asm volatile(" \ + r2 = %[map_prog] ll; \tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c line 88
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+ r3 =0 ; \ + call %[bpf_tail_call]; \ + r0 =0 ; \ + exit; \ +" : + : __imm(bpf_tail_call), __imm_addr(map_prog) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
Thanks for the review! Good point, I'll try to write them in C.
It might not be possible to do them both entirely: clang also doesn't know that bpf_tail_call() can return, so it assumes the callee() will return a constant r0. It sometimes optimizes branches / loads out because of this.
Re: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
Quoted 53 lines · Arthur Fabre
Quoted 1 line · Eduard Zingerman
On Fri, 2024-12-13 at 22:27 +0100, Arthur Fabre wrote:
[...]
Quoted 31 lines
Quoted 5 lines
+++
b/tools/testing/selftests/bpf/progs/verifier_abnormal_ret.ctools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +3
-0@@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c line 13
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+#include <bpf/bpf_helpers.h> +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#define TEST (NAME ,CALLEE ) \ + SEC("socket") \ + __description("abnormal_ret: " #NAME) \ + __failure __msg("math between ctx pointer and register with unbounded min value") \ + __naked void check_abnormal_ret_##NAME(void) \ + { \Nit: this one and '
callee_tail_call' could be plain C.Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+ asm volatile(" \ + r6 = r1; \ + call " #CALLEE "; \ + r6 += r0; \ + r0 = 0; \ + exit; \ + " : \ + : \ + : __clobber_all); \ + }[...]
Quoted 15 lines
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +5
-0+static __naked __noinline __used +int callee_tail_call(void) +{ + asm volatile(" \ + r2 = %[map_prog] ll; \tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c line 88
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_abnormal_ret.c new file +10
-0+ r3 =0 ; \ + call %[bpf_tail_call]; \ + r0 =0 ; \ + exit; \ +" : + : __imm(bpf_tail_call), __imm_addr(map_prog) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";Thanks for the review! Good point, I'll try to write them in C.
Quoted 4 lines
It might not be possible to do them both entirely: clang also doesn't know that
bpf_tail_call()can return, so it assumes thecallee()will return a constant r0. It sometimes optimizes branches / loads out because of this.
I wonder whether we should tell llvm that it's similar to longjmp() with __attribute__((noreturn)) or some other attribute.
Eduard Zingerman
maintainerRe: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
On Mon, 2024-12-16 at 10:05 -0800, Alexei Starovoitov wrote:
[...]
Quoted 9 lines
Quoted 6 lines
Thanks for the review! Good point, I'll try to write them in C.
It might not be possible to do them both entirely: clang also doesn't know that
bpf_tail_call()can return, so it assumes thecallee()will return a constant r0. It sometimes optimizes branches / loads out because of this.I wonder whether we should tell llvm that it's similar to
longjmp()with __attribute__((noreturn)) or some other attribute.
GCC documents it as follows [1]:
Quoted 5 lines
The noreturn keyword tells the compiler to assume that fatal cannot reaturn. It can then optimize without regard to what would happen if fatal ever did return. This makes slightly better code. More importantly, it helps avoid spurious warnings of uninitialized variables.
But the bpf_tail_call could return if MAX_TAIL_CALL_CNT limit is exceeded, or programs map index is out of bounds.
Re: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
Quoted 23 lines · Eduard Zingerman
On Mon, 2024-12-16 at 10:05 -0800, Alexei Starovoitov wrote:
[...]
Quoted 9 lines
Quoted 6 lines
Thanks for the review! Good point, I'll try to write them in C.
It might not be possible to do them both entirely: clang also doesn't know that
bpf_tail_call()can return, so it assumes thecallee()will return a constant r0. It sometimes optimizes branches / loads out because of this.I wonder whether we should tell llvm that it's similar to
longjmp()with __attribute__((noreturn)) or some other attribute.GCC documents it as follows [1]:
Quoted 5 lines
The noreturn keyword tells the compiler to assume that fatal cannot reaturn. It can then optimize without regard to what would happen if fatal ever did return. This makes slightly better code. More importantly, it helps avoid spurious warnings of uninitialized variables.
Quoted 4 lines
But the
bpf_tail_callcould return ifMAX_TAIL_CALL_CNTlimit is exceeded, or programs map index is out of bounds.
Yeah. noreturn is too heavy.attr(returns_twice) is another option, but it will probably pessimize the code too much as well.
Re: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
Quoted 24 lines · Alexei Starovoitov
Quoted 23 lines · Eduard Zingerman
On Mon, 2024-12-16 at 10:05 -0800, Alexei Starovoitov wrote:
[...]
Quoted 9 lines
Quoted 6 lines
Thanks for the review! Good point, I'll try to write them in C.
It might not be possible to do them both entirely: clang also doesn't know that
bpf_tail_call()can return, so it assumes thecallee()will return a constant r0. It sometimes optimizes branches / loads out because of this.I wonder whether we should tell llvm that it's similar to
longjmp()with __attribute__((noreturn)) or some other attribute.GCC documents it as follows [1]:
Quoted 5 lines
The noreturn keyword tells the compiler to assume that fatal cannot reaturn. It can then optimize without regard to what would happen if fatal ever did return. This makes slightly better code. More importantly, it helps avoid spurious warnings of uninitialized variables.
Quoted 8 lines
Quoted 4 lines
But the
bpf_tail_callcould return ifMAX_TAIL_CALL_CNTlimit is exceeded, or programs map index is out of bounds.Yeah. noreturn is too heavy.
attr(returns_twice)is another option, but it will probably pessimize the code too much as well.
We could provide a macro like:
#define BPF_TAIL_CALL(ctx, map, index) do { \
bpf_tail_call(ctx, map, index); \
int maybe_return; \
asm volatile("%0 = 0" : "=r"(maybe_return)); \
if (maybe_return) \
return maybe_return; \
} while(0)It correctly tricks clang into thinking it can return early, but the verifier removes the dead branch so there's no runtime cost.
But users would have to switch to it, I don't think we can replace the definition in bpf_helper_defs.h in a backwards compatible way.
Eduard Zingerman
maintainerRe: [PATCH bpf v2 2/2] selftests/bpf: Test r0 bounds after BPF to BPF call with abnormal return
In reply to Arthur FabreOn Mon, 2024-12-16 at 18:39 +0100, Arthur Fabre wrote:
[...]
Quoted 4 lines
It might not be possible to do them both entirely: clang also doesn't know that
bpf_tail_call()can return, so it assumes thecallee()will return a constant r0. It sometimes optimizes branches / loads out because of this.
Hm, haven't thought about this.
You would need assembly to hide the r0 indeed.
Quoted 6 lines · Arthur Fabre
A BPF function can return before its exit instruction:
LD_ABS,LD_IND, andtail_call()can all cause it to return prematurely.When such a function is called by another BPF function, the verifier doesn't take this into account when calculating the bounds of r0 in the caller after the callee returns.
I've just realized r0 isn't he only problem: a caller can pass a reference to its stack to a callee, and the verifier also tracks the value of this.
If the callee writes to the caller's stack after the abnormal return (tail_call, ld_abs), the verifier will also incorrectly assume the write always happens.
I think we need to treat these abnormal returns as a branch that can exit. That way the verifier will explore both possibilities, and the combined result will really reflect what can happen.
I'll try that out for a v3.
Quoted 13 lines · Arthur Fabre
Quoted 6 lines · Arthur Fabre
A BPF function can return before its exit instruction:
LD_ABS,LD_IND, andtail_call()can all cause it to return prematurely.When such a function is called by another BPF function, the verifier doesn't take this into account when calculating the bounds of r0 in the caller after the callee returns.
I've just realized r0 isn't he only problem: a caller can pass a reference to its stack to a callee, and the verifier also tracks the value of this.
Quoted 8 lines
If the callee writes to the caller's stack after the abnormal return (
tail_call,ld_abs), the verifier will also incorrectly assume the write always happens.I think we need to treat these abnormal returns as a branch that can exit. That way the verifier will explore both possibilities, and the combined result will really reflect what can happen.
I'll try that out for a v3.
Good idea. Makes sense to me.