s390/bpf: Write back tail call counter
Subsystems (5)
Ilya Leoshkevich
maintainerv2 cover s390/bpf: Write back tail call counter
v1: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2: Write back tail call counter only for BPF_PSEUDO_CALL
Hi,
This series fixes the tailcall_bpf2bpf_hierarchy test failures on s390.
It takes a simpler approach than x86_64 and aarch64: instead of introducing a pointer to the tail call counter, it copies the updated value back from the callee's frame to the caller's frame. This needs to be done in two locations: after BPF_PSEUDO_CALL and after BPF_TRAMP_F_CALL_ORIG.
Patch 1 is a cleanup, patches 2 and 3 are the actual fixes, patch 4 improves the hierarchy tests in order to catch issues with accidentally clobbering the counter.
Best regards,
Ilya
Ilya Leoshkevich (4):
s390/bpf: Do not write tail call counter into helper and kfunc frames s390/bpf: Write back tail call counter for BPF_PSEUDO_CALL s390/bpf: Write back tail call counter for BPF_TRAMP_F_CALL_ORIG selftests/bpf: Clobber a lot of registers in
tailcall_bpf2bpf_hierarchy tests
arch/s390/net/bpf_jit_comp.c | 42 ++++++++++++++----- .../selftests/bpf/progs/bpf_test_utils.h | 18 ++++++++ .../bpf/progs/tailcall_bpf2bpf_hierarchy1.c | 3 ++ .../bpf/progs/tailcall_bpf2bpf_hierarchy2.c | 3 ++ .../bpf/progs/tailcall_bpf2bpf_hierarchy3.c | 3 ++ .../progs/tailcall_bpf2bpf_hierarchy_fentry.c | 3 ++ 6 files changed, 61 insertions(+), 11 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/bpf_test_utils.h
Signature · 1 line
2.50.1
Ilya Leoshkevich
maintainerPATCH v2 1/4 s390/bpf: Do not write tail call counter into helper and kfunc frames
Only BPF functions make use of the tail call counter; helpers and kfuncs ignore and most likely also clobber it. Writing it into these functions' frames is pointless and misleading, so do not do it.
--- arch/s390/net/bpf_jit_comp.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
arch/s390/net/bpf_jit_comp.c +12 -4
@@ -1790,6 +1790,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,REG_SET_SEEN (BPF_REG_5 ); jit->seen |=SEEN_FUNC ; +/* * Copy the tail call counter to where the callee expects it. *
@@ -1800,10 +1801,17 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,* Note 2: We assume that the verifier does not let us call the * main program, which clears the tail call counter on entry. */ -/* mvc tail_call_cnt(4,%r15),frame_off+tail_call_cnt(%r15) */ -_EMIT6 (0xd203f000 | offsetof(struct prog_frame , tail_call_cnt), -0xf000 | (jit->frame_off + - offsetof(struct prog_frame , tail_call_cnt))); + +if (insn->src_reg ==BPF_PSEUDO_CALL ) +/* +* mvc tail_call_cnt(4,%r15), +* frame_off+tail_call_cnt(%r15) +*/ +_EMIT6 (0xd203f000 | offsetof(struct prog_frame , + tail_call_cnt), +0xf000 | (jit->frame_off + + offsetof(struct prog_frame , + tail_call_cnt)));/* Sign-extend the kfunc arguments. */ if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL ) {
Signature · 1 line
2.50.1
Ilya Leoshkevich
maintainerPATCH v2 2/4 s390/bpf: Write back tail call counter for BPF_PSEUDO_CALL
In reply to Ilya LeoshkevichThe tailcall_bpf2bpf_hierarchy_1 test hangs on s390. Its call graph is as follows:
entry()
subprog_tail()
bpf_tail_call_static(0) -> entry + tail_call_start
subprog_tail()
bpf_tail_call_static(0) -> entry + tail_call_startentry() copies its tail call counter to the subprog_tail()'s frame, which then increments it. However, the incremented result is discarded, leading to an astronomically large number of tail calls.
Fix by writing the incremented counter back to the entry()'s frame.
--- arch/s390/net/bpf_jit_comp.c | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-)
arch/s390/net/bpf_jit_comp.c +16 -7
@@ -1793,13 +1793,6 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,/* * Copy the tail call counter to where the callee expects it. -* -* Note 1: The callee can increment the tail call counter, but -* we do not load it back, since the x86 JIT does not do this -* either. -* -* Note 2: We assume that the verifier does not let us call the -* main program, which clears the tail call counter on entry. */ if (insn->src_reg ==BPF_PSEUDO_CALL )
@@ -1833,6 +1826,22 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,call_r1 (jit);/* lgr %b0,%r2: load return value into %b0 */ EMIT4 (0xb9040000 ,BPF_REG_0 ,REG_2 ); + +/* +* Copy the potentially updated tail call counter back. +*/ + +if (insn->src_reg ==BPF_PSEUDO_CALL ) +/* +* mvc frame_off+tail_call_cnt(%r15), +* tail_call_cnt(4,%r15) +*/ +_EMIT6 (0xd203f000 | (jit->frame_off + + offsetof(struct prog_frame , + tail_call_cnt)), +0xf000 | offsetof(struct prog_frame , + tail_call_cnt)); +break ; }case BPF_JMP |BPF_TAIL_CALL : {
Signature · 1 line
2.50.1
Ilya Leoshkevich
maintainerPATCH v2 3/4 s390/bpf: Write back tail call counter for BPF_TRAMP_F_CALL_ORIG
In reply to Ilya LeoshkevichThe tailcall_bpf2bpf_hierarchy_fentry test hangs on s390. Its call graph is as follows:
entry()
subprog_tail()
trampoline()
fentry()
the rest of subprog_tail() # via BPF_TRAMP_F_CALL_ORIG return to entry()
The problem is that the rest of subprog_tail() increments the tail call counter, but the trampoline discards the incremented value. This results in an astronomically large number of tail calls.
Fix by making the trampoline write the incremented tail call counter back.
--- arch/s390/net/bpf_jit_comp.c | 3 +++ 1 file changed, 3 insertions(+)
arch/s390/net/bpf_jit_comp.c +3 -0
@@ -2839,6 +2839,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,/* stg %r2,retval_off(%r15) */ EMIT6_DISP_LH (0xe3000000 ,0x0024 ,REG_2 ,REG_0 ,REG_15 , tjit->retval_off); +/* mvc tccnt_off(%r15),tail_call_cnt(4,%r15) */ +_EMIT6 (0xd203f000 | tjit->tccnt_off, +0xf000 | offsetof(struct prog_frame , tail_call_cnt)); im->ip_after_call = jit->prg_buf + jit->prg;
Signature · 1 line
2.50.1
Ilya Leoshkevich
maintainerPATCH v2 4/4 selftests/bpf: Clobber a lot of registers in tailcall_bpf2bpf_hierarchy tests
In reply to Ilya LeoshkevichClobbering a lot of registers and stack slots helps exposing tail call counter overwrite bugs in JITs.
--- .../selftests/bpf/progs/bpf_test_utils.h | 18 ++++++++++++++++++ .../bpf/progs/tailcall_bpf2bpf_hierarchy1.c | 3 +++ .../bpf/progs/tailcall_bpf2bpf_hierarchy2.c | 3 +++ .../bpf/progs/tailcall_bpf2bpf_hierarchy3.c | 3 +++ .../progs/tailcall_bpf2bpf_hierarchy_fentry.c | 3 +++ 5 files changed, 30 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/bpf_test_utils.h
tools/testing/selftests/bpf/progs/bpf_test_utils.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __BPF_TEST_UTILS_H__ +#define __BPF_TEST_UTILS_H__ + +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +/* Clobber as many native registers and stack slots as possible. */ +static __always_inline void clobber_regs_stack(void) +{ +char tmp_str[] ="123456789" ; +unsigned long tmp; + +bpf_strtoul (tmp_str,sizeof (tmp_str),0 , &tmp); +__sink (tmp); +} + +#endif
tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy1.c +3 -0
@@ -2,6 +2,7 @@#include <linux/bpf.h> #include <bpf/bpf_helpers.h> #include "bpf_legacy.h" +#include "bpf_test_utils.h" struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
@@ -24,6 +25,8 @@ int entry(struct __sk_buff *skb) {int ret =1 ; +clobber_regs_stack (); + count++;subprog_tail (skb);subprog_tail (skb);
tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy2.c +3 -0
@@ -2,6 +2,7 @@#include <linux/bpf.h> #include <bpf/bpf_helpers.h> #include "bpf_misc.h" +#include "bpf_test_utils.h" int classifier_0 (struct __sk_buff *skb);int classifier_1 (struct __sk_buff *skb);
@@ -60,6 +61,8 @@ int tailcall_bpf2bpf_hierarchy_2(struct __sk_buff *skb) {int ret =0 ; +clobber_regs_stack (); +subprog_tail0 (skb);subprog_tail1 (skb);
tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy3.c +3 -0
@@ -2,6 +2,7 @@#include <linux/bpf.h> #include <bpf/bpf_helpers.h> #include "bpf_misc.h" +#include "bpf_test_utils.h" int classifier_0 (struct __sk_buff *skb);
@@ -53,6 +54,8 @@ int tailcall_bpf2bpf_hierarchy_3(struct __sk_buff *skb) {int ret =0 ; +clobber_regs_stack (); +bpf_tail_call_static (skb, &jmp_table0,0 );__sink (ret);
tools/testing/selftests/bpf/progs/tailcall_bpf2bpf_hierarchy_fentry.c +3 -0
@@ -4,6 +4,7 @@#include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> +#include "bpf_test_utils.h" struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
@@ -24,6 +25,8 @@ int subprog_tail(void *ctx) SEC("fentry/dummy")int BPF_PROG (fentry ,struct sk_buff *skb) { +clobber_regs_stack (); + count++;subprog_tail (ctx);subprog_tail (ctx);
Signature · 1 line
2.50.1
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Quoted 8 lines · On Wed, 13 Aug 2025 14:06:27 +0200 you wrote:
v1: https://lore.kernel.org/bpf/[email protected]/
v1 -> v2: Write back tail call counter only forBPF_PSEUDO_CALLHi,
This series fixes the
tailcall_bpf2bpf_hierarchytest failures on s390.[...]
Here is the summary with links:
- [bpf-next,v2,1/4] s390/bpf: Do not write tail call counter into helper and kfunc frames
https://git.kernel.org/bpf/bpf-next/c/eada40e057fc - [bpf-next,v2,2/4] s390/bpf: Write back tail call counter for BPF_PSEUDO_CALL https://git.kernel.org/bpf/bpf-next/c/c861a6b14713 - [bpf-next,v2,3/4] s390/bpf: Write back tail call counter for BPF_TRAMP_F_CALL_ORIG https://git.kernel.org/bpf/bpf-next/c/bc3905a71f02 - [bpf-next,v2,4/4] selftests/bpf: Clobber a lot of registers in tailcall_bpf2bpf_hierarchy tests https://git.kernel.org/bpf/bpf-next/c/12741630350c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html