Skip to content

bpf: Support stack arguments for BPF functions and kfuncs

Yonghong Song bpf 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest18 patches 28 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/18bpf: Support stack arguments for bpf functionsinclude/linux/bpf.h include/linux/bpf_verifier.h kernel/bpf/btf.c +3 more+282 -1211 replies
2/18bpf: Add precision marking and backtracking for stack argument slotsinclude/linux/bpf_verifier.h kernel/bpf/backtrack.c kernel/bpf/verifier.c+98 -63 replies
3/18bpf: Refactor record_call_access() to extract per-arg logickernel/bpf/liveness.c+41 -282 replies
4/18bpf: Extend liveness analysis to track stack argument slotsinclude/linux/bpf_verifier.h kernel/bpf/const_fold.c kernel/bpf/liveness.c+135 -133 replies
5/18bpf: Reject stack arguments in non-JITed programskernel/bpf/core.c kernel/bpf/fixups.c+7 -11 reply
6/18bpf: Prepare architecture JIT support for stack argumentsinclude/linux/filter.h kernel/bpf/btf.c kernel/bpf/core.c +1 more+18 -12 replies
7/18bpf: Enable r11 based insnskernel/bpf/verifier.c+25 -41 reply
8/18bpf: Support stack arguments for kfunc callskernel/bpf/verifier.c+56 -171 reply
9/18bpf: Reject stack arguments if tail call reachablekernel/bpf/verifier.c+15 -10 replies
10/18bpf,x86: Implement JIT support for stack argumentsarch/x86/net/bpf_jit_comp.c+148 -61 reply
11/18selftests/bpf: Add tests for BPF function stack argumentstools/testing/selftests/bpf/prog_tests/stack_arg.c tools/testing/selftests/bpf/progs/stack_arg.c tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +2 more+638 -10 replies
12/18selftests/bpf: Add tests for stack argument validationtools/testing/selftests/bpf/prog_tests/stack_arg_fail.c tools/testing/selftests/bpf/progs/stack_arg_fail.c tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1 more+139 -00 replies
13/18selftests/bpf: Add verifier tests for stack argument validationtools/testing/selftests/bpf/prog_tests/verifier.c tools/testing/selftests/bpf/progs/verifier_stack_arg.c+458 -01 reply
14/18selftests/bpf: Add BTF fixup for __naked subprog parameter namestools/testing/selftests/bpf/progs/bpf_misc.h tools/testing/selftests/bpf/test_loader.c+136 -10 replies
15/18selftests/bpf: Add precision backtracking test for stack argumentstools/testing/selftests/bpf/prog_tests/stack_arg_precision.c tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c tools/testing/selftests/bpf/progs/stack_arg_precision.c+154 -00 replies
16/18bpf, arm64: Map BPF_REG_0 to x8 instead of x7arch/arm64/net/bpf_jit_comp.c arch/arm64/net/bpf_timed_may_goto.S tools/testing/selftests/bpf/progs/verifier_jit_inline.c +2 more+15 -150 replies
17/18bpf, arm64: Add JIT support for stack argumentsarch/arm64/net/bpf_jit_comp.c+83 -42 replies
18/18selftests/bpf: Enable stack argument tests for arm64tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c tools/testing/selftests/bpf/progs/stack_arg.c tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +2 more+10 -50 replies

Reviews trailers per patch

7 of 18 patches have a Reviewed-by or Acked-by.

PatchPuranjay MohanEduard Zingerman
1/18
2/18A-b
3/18A-b
4/18
5/18A-b A-b
6/18A-b A-b
7/18A-b
8/18
9/18
10/18A-b
11/18A-b
12/18
13/18
14/18
15/18
16/18
17/18
18/18

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH bpf-next 00/18] bpf: Support stack arguments for BPF functions and kfuncs

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct although it may have some inconvenience. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (BPF_REG_PARAMS), to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The current maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

All kfunc/bpf-function arguments are caller saved, including stack arguments. For register arguments (r1-r5), the verifier already marks them as clobbered after each call. For stack arguments, the verifier invalidates all outgoing stack arg slots immediately after a call, requiring the compiler to re-store them before any subsequent call. This follows the native calling convention where all function parameters are caller saved.

The x86_64 JIT translates r11-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the callee-saved registers. This makes implementation easier as the r10 can be reused for stack argument access. At both BPF-to-BPF and kfunc calls, outgoing args are pushed onto the expected calling convention locations directly. The incoming parameters can directly get the value from caller.

To support kfunc stack arguments, before doing any stack arguments, existing codes are refactored/modified to use bpf_reg_state as much as possible instead of using regno, and to pass a non-negative argno, encoded to support both registers and stack arguments, as a single variable.

Global subprogs and freplace progs with >5 args are not yet supported. Only x86_64 and arm64 are supported for now. Same selftests are tested by both x86_64 and arm64. Please see each individual patch for details.

[1] https://github.com/llvm/llvm-project/pull/189060

Note:
- The patch set is on top of the following commit:
fd30cf86e1194 Merge branch 'bpf-prepare-to-support-stack-arguments'
- This patch set requires latest llvm23 compiler. It is possible that a build

    failure may appear:
      /home/yhs/work/bpf-next/scripts/mod/modpost.c:59:13: error: variable 'extra_warn' set but not used [-Werror,-Wunused-but-set-global]
             59 | static bool extra_warn;
                |             ^
          1 error generated.
    In this case, the following hack can workaround the build issue:
      --- a/Makefile
      +++ b/Makefile
      @@ -467,7 +467,7 @@ KERNELDOC       = $(srctree)/tools/docs/kernel-doc
       export KERNELDOC

       KBUILD_USERHOSTCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \
      -                        -O2 -fomit-frame-pointer -std=gnu11
      +                        -O2 -fomit-frame-pointer -std=gnu11 -Wno-unused-but-set-global
       KBUILD_USERCFLAGS  := $(KBUILD_USERHOSTCFLAGS) $(USERCFLAGS)
       KBUILD_USERLDFLAGS := $(USERLDFLAGS)

Puranjay Mohan (3):
bpf, arm64: Map BPF_REG_0 to x8 instead of x7 bpf, arm64: Add JIT support for stack arguments selftests/bpf: Enable stack argument tests for arm64

Yonghong Song (15):
bpf: Support stack arguments for bpf functions
bpf: Add precision marking and backtracking for stack argument slots
bpf: Refactor record_call_access() to extract per-arg logic
bpf: Extend liveness analysis to track stack argument slots
bpf: Reject stack arguments in non-JITed programs
bpf: Prepare architecture JIT support for stack arguments
bpf: Enable r11 based insns
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments if tail call reachable bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add tests for stack argument validation selftests/bpf: Add verifier tests for stack argument validation selftests/bpf: Add BTF fixup for __naked subprog parameter names selftests/bpf: Add precision backtracking test for stack arguments

 arch/arm64/net/bpf_jit_comp.c                 |  91 +++-
 arch/arm64/net/bpf_timed_may_goto.S           |   8 +-
 arch/x86/net/bpf_jit_comp.c                   | 154 +++++-
 include/linux/bpf.h                           |   2 +
 include/linux/bpf_verifier.h                  |  42 +-
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  61 ++-
 kernel/bpf/btf.c                              |  20 +-
 kernel/bpf/const_fold.c                       |  13 +-
 kernel/bpf/core.c                             |   7 +-
 kernel/bpf/fixups.c                           |  28 +-
 kernel/bpf/liveness.c                         | 200 ++++++--
 kernel/bpf/states.c                           |  31 ++
 kernel/bpf/verifier.c                         | 347 +++++++++++--
 .../selftests/bpf/prog_tests/stack_arg.c      | 139 ++++++
 .../selftests/bpf/prog_tests/stack_arg_fail.c |  10 +
 .../bpf/prog_tests/stack_arg_precision.c      |  10 +
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 tools/testing/selftests/bpf/progs/bpf_misc.h  |   1 +
 .../bpf/progs/btf__stack_arg_precision.c      |  24 +
 tools/testing/selftests/bpf/progs/stack_arg.c | 253 ++++++++++
 .../selftests/bpf/progs/stack_arg_fail.c      | 114 +++++
 .../selftests/bpf/progs/stack_arg_kfunc.c     | 164 +++++++
 .../selftests/bpf/progs/stack_arg_precision.c | 122 +++++
 .../selftests/bpf/progs/verifier_jit_inline.c |   2 +-
 .../selftests/bpf/progs/verifier_ldsx.c       |   6 +-
 .../bpf/progs/verifier_private_stack.c        |  10 +-
 .../selftests/bpf/progs/verifier_stack_arg.c  | 457 ++++++++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  72 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  26 +
 tools/testing/selftests/bpf/test_loader.c     | 136 +++++-
 31 files changed, 2448 insertions(+), 105 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_precision.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0