Skip to content

bpf: Support stack arguments for BPF functions and kfuncs

Yonghong Song bpf sashiko 9 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 posting 2/225 patches 13 repliesApplied 25/25
  2. v3 posting 2/224 patches 18 repliesSuperseded
  3. v2 posting 2/223 patches 24 repliesSuperseded
Earlier versions · 6
VersionPostedPatchesRepliesStatus
v6 Latest17 patches25 repliesIn review
v516 patches27 repliesSuperseded
v4 posting 1/218 patches24 repliesSuperseded
v3 posting 1/211 patches10 repliesSuperseded
v2 posting 1/211 patches1 replySuperseded
v110 patches20 repliesSuperseded

Changes in v6 from the cover letter

- v5: https://lore.kernel.org/bpf/[email protected]/
- Do stack arguments invalidation after bpf function or kfunc all. This is to following native achitecture calling convention.
- Fix some r11 related issues in const_fold, liveness and insn checking.
- Fix a few places for precision marking for stack arguments. All these cases have const value and mark them as precise.
- Unconditionally copy some incoming/outgoing stat.
- Fix a missing tailcall case with main prog having tailcall and there is no other subprog's.
- Add and fix tests as suggested in v5.
- Reorder some patches, e.g., preparing stack arguments in bpf functions, disabling JIT, before allowing proper r11 usage.

Patches in v6 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/17bpf: Remove unused parameter from check_map_kptr_access()kernel/bpf/verifier.c+2 -20 replies
2/17bpf: Refactor to avoid redundant calculation of bpf_reg_statekernel/bpf/verifier.c+93 -1200 replies
3/17bpf: Refactor to handle memory and size togetherkernel/bpf/verifier.c+30 -292 replies
4/17bpf: Prepare verifier logs for upcoming kfunc stack argumentsinclude/linux/bpf_verifier.h kernel/bpf/verifier.c tools/testing/selftests/bpf/prog_tests/bpf_nf.c +27 more+474 -3748 replies
5/17bpf: Introduce bpf register BPF_REG_PARAMSinclude/linux/filter.h kernel/bpf/core.c tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c +3 more+62 -611 reply
6/17bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of argumentsinclude/linux/bpf.h+5 -00 replies
7/17bpf: Support stack arguments for bpf functionsinclude/linux/bpf.h include/linux/bpf_verifier.h kernel/bpf/btf.c +3 more+390 -123 replies
8/17bpf: Reject stack arguments in non-JITed programskernel/bpf/core.c kernel/bpf/fixups.c+7 -11 reply
9/17bpf: Track r11 registers in const_fold and livenesskernel/bpf/const_fold.c kernel/bpf/liveness.c+24 -40 replies
10/17bpf: Prepare architecture JIT support for stack argumentsinclude/linux/filter.h kernel/bpf/btf.c kernel/bpf/core.c +1 more+18 -10 replies
11/17bpf: Enable r11 based insnskernel/bpf/verifier.c+15 -40 replies
12/17bpf: Support stack arguments for kfunc callskernel/bpf/verifier.c+103 -241 reply
13/17bpf: Reject stack arguments if tail call reachablekernel/bpf/verifier.c+15 -11 reply
14/17bpf,x86: Implement JIT support for stack argumentsarch/x86/net/bpf_jit_comp.c+148 -61 reply
15/17selftests/bpf: Add tests for BPF function stack argumentstools/testing/selftests/bpf/prog_tests/stack_arg.c tools/testing/selftests/bpf/progs/stack_arg.c tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +2 more+642 -11 reply
16/17selftests/bpf: Add tests for stack argument validationtools/testing/selftests/bpf/prog_tests/stack_arg_fail.c tools/testing/selftests/bpf/progs/stack_arg_fail.c tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1 more+149 -00 replies
17/17selftests/bpf: Add verifier tests for stack argument validationtools/testing/selftests/bpf/prog_tests/verifier.c tools/testing/selftests/bpf/progs/verifier_stack_arg.c+435 -01 reply

Reviews trailers per patch

None of the 17 patches has a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchPuranjay Mohan
SeriesA-b
1/17
2/17
3/17
4/17
5/17
6/17
7/17
8/17
9/17
10/17
11/17
12/17
13/17
14/17
15/17
16/17
17/17

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v6 thread

Cover letter [PATCH bpf-next v6 00/17] bpf: Support stack arguments for BPF functions and kfuncs

Currently, bpf function calls and kfunc's are limited by 5 reg-level parameters. For function calls with more than 5 parameters, developers can use always inlining or pass a struct pointer after packing more parameters in that struct although it may have some inconvenience. But there is no workaround for kfunc if more than 5 parameters is needed.

This patch set lifts the 5-argument limit by introducing stack-based argument passing for BPF functions and kfunc's, coordinated with compiler support in LLVM [1]. The compiler emits loads/stores through a new bpf register r11 (BPF_REG_PARAMS) to pass arguments beyond the 5th, keeping the stack arg area separate from the r10-based program stack. The current maximum number of arguments is capped at MAX_BPF_FUNC_ARGS (12), which is sufficient for the vast majority of use cases.

All kfunc/bpf-function arguments are caller saved, including stack arguments. For register arguments (r1-r5), the verifier already marks them as clobbered after each call. For stack arguments, the verifier invalidates all outgoing stack arg slots immediately after a call, requiring the compiler to re-store them before any subsequent call. This follows the native calling convention where all function parameters are caller saved.

The x86_64 JIT translates r11-relative accesses to RBP-relative native instructions. Each function's stack allocation is extended by 'max_outgoing' bytes to hold the outgoing arg area below the callee-saved registers. This makes implementation easier as the r10 can be reused for stack argument access. At both BPF-to-BPF and kfunc calls, outgoing args are pushed onto the expected calling convention locations directly. The incoming parameters can directly get the value from caller.

To support kfunc stack arguments, before doing any stack arguments, existing codes are refactored/modified to use bpf_reg_state as much as possible instead of using regno, and to pass a non-negative argno, encoded to support both registers and stack arguments, as a single variable.

Global subprogs with >5 args are not yet supported. Only x86_64 is supported for now.

For the rest of patches, patches 1-4 make changes to make it easy for future stack arguments for kfuncs. Patches 5-8 supports bpf-to-bpf stack arguments. Patch 9 rejects interpreter for stack arguments. Patch 10 rejects subprogs if tailcall reachable. Patch 11 adds stack argument support for kfuncs. Patch 12 enables stack arguments for x86_64 and Patch 13 implements the x86_64 JIT. Patches 14-16 are some test cases.

[1] https://github.com/llvm/llvm-project/pull/189060

Note:
- The patch set is on top of the following commit:
eb0d6d97c27c Merge tag 'bpf-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf
- This patch set requires latest llvm23 compiler. It is possible that a build

    failure may appear:
      /home/yhs/work/bpf-next/scripts/mod/modpost.c:59:13: error: variable 'extra_warn' set but not used [-Werror,-Wunused-but-set-global]
             59 | static bool extra_warn;
                |             ^
          1 error generated.
    In this case, the following hack can workaround the build issue:
      --- a/Makefile
      +++ b/Makefile
      @@ -467,7 +467,7 @@ KERNELDOC       = $(srctree)/tools/docs/kernel-doc
       export KERNELDOC

       KBUILD_USERHOSTCFLAGS := -Wall -Wmissing-prototypes -Wstrict-prototypes \
      -                        -O2 -fomit-frame-pointer -std=gnu11
      +                        -O2 -fomit-frame-pointer -std=gnu11 -Wno-unused-but-set-global
       KBUILD_USERCFLAGS  := $(KBUILD_USERHOSTCFLAGS) $(USERCFLAGS)
       KBUILD_USERLDFLAGS := $(USERLDFLAGS)

Changelogs:
v5 -> v6:
- v5: https://lore.kernel.org/bpf/[email protected]/
- Do stack arguments invalidation after bpf function or kfunc all. This is to following native achitecture calling convention.
- Fix some r11 related issues in const_fold, liveness and insn checking.
- Fix a few places for precision marking for stack arguments. All these cases have const value and mark them as precise.
- Unconditionally copy some incoming/outgoing stat.
- Fix a missing tailcall case with main prog having tailcall and there is no other subprog's.
- Add and fix tests as suggested in v5.
- Reorder some patches, e.g., preparing stack arguments in bpf functions, disabling JIT, before allowing proper r11 usage.
v4 -> v5:
- v4: https://lore.kernel.org/bpf/[email protected]/
- Use r11 instead of r12, llvm also updated with r11.
- Change int type 'reg_or_arg' to u32 'argno' where 'argno' encodes to support both bpf registers and stack arguments.
- Track per-state bitmask 'out_stack_arg_mask' for r11 based stores, so at any particular call, it knows what stores are available. This is important since stores may be in different basic block.
- Previously after each call, all store slots are invalidated. This patches disabled such invalidation.
- Ensure r11 reg only appearing in allowed insns. Also avoid r11 for reg tracking purpose.
- Make stack_arg_regs more similar to regular reg's (struct bpf_reg_state *)..
- Reorder r11 based stores from 'arg6:off:-24, arg7:off:-16, arg8:off:-8" to "arg6:off:-8, arg7:off:-16, arg8:off:-24".
- Add a few more tests, including e.g., two callee's with different number of stack arguments, shared r11-stores in different branches, etc.

v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Refactor/Modify codes to make it easier for later kfunc stack argument support
- Invalidate outgoing slots immediately after the call to prevent reuse
- Fix interaction between stack argument PTR_TO_STACK and dead slot poisoning
- Reject stack arguments if tail call reachable
- Disable private stack if stack argument is used
- Allocate outgoing stack argument region after callee saved registers, and this simplifies the JITed code a lot.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Fix selftest stack_arg_gap_at_minus8().
- Fix a few 'UTF-8' issues.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Add stack_arg_safe() to do pruning for stack arguments.
- Fix an issue with KF_ARG_PTR_TO_MEM_SIZE. Since a faked register is used, added verification log to indicate the start and end of such faked register usage.
- For x86_64 JIT, copying incoming parameter values directly from caller's stack.
- Add test cases with stack arguments e.g. mem, mem+size, dynptr, iter, etc.

Yonghong Song (17):
bpf: Remove unused parameter from check_map_kptr_access()
bpf: Refactor to avoid redundant calculation of bpf_reg_state
bpf: Refactor to handle memory and size together
bpf: Prepare verifier logs for upcoming kfunc stack arguments
bpf: Introduce bpf register BPF_REG_PARAMS
bpf: Reuse MAX_BPF_FUNC_ARGS for maximum number of arguments
bpf: Support stack arguments for bpf functions
bpf: Reject stack arguments in non-JITed programs
bpf: Track r11 registers in const_fold and liveness
bpf: Prepare architecture JIT support for stack arguments
bpf: Enable r11 based insns
bpf: Support stack arguments for kfunc calls
bpf: Reject stack arguments if tail call reachable bpf,x86: Implement JIT support for stack arguments selftests/bpf: Add tests for BPF function stack arguments selftests/bpf: Add tests for stack argument validation selftests/bpf: Add verifier tests for stack argument validation

 arch/x86/net/bpf_jit_comp.c                   |  154 +-
 include/linux/bpf.h                           |    7 +
 include/linux/bpf_verifier.h                  |   29 +-
 include/linux/filter.h                        |    6 +-
 kernel/bpf/btf.c                              |   20 +-
 kernel/bpf/const_fold.c                       |   14 +-
 kernel/bpf/core.c                             |   11 +-
 kernel/bpf/fixups.c                           |   28 +-
 kernel/bpf/liveness.c                         |   14 +-
 kernel/bpf/states.c                           |   41 +
 kernel/bpf/verifier.c                         | 1233 ++++++++++++-----
 .../testing/selftests/bpf/prog_tests/bpf_nf.c |   22 +-
 .../selftests/bpf/prog_tests/cb_refs.c        |    2 +-
 .../selftests/bpf/prog_tests/ctx_rewrite.c    |   14 +-
 .../selftests/bpf/prog_tests/kfunc_call.c     |    2 +-
 .../selftests/bpf/prog_tests/linked_list.c    |    4 +-
 .../selftests/bpf/prog_tests/stack_arg.c      |  139 ++
 .../selftests/bpf/prog_tests/stack_arg_fail.c |   10 +
 .../selftests/bpf/prog_tests/verifier.c       |    2 +
 .../selftests/bpf/progs/cgrp_kfunc_failure.c  |   14 +-
 .../selftests/bpf/progs/cpumask_failure.c     |   10 +-
 .../testing/selftests/bpf/progs/dynptr_fail.c |   22 +-
 .../selftests/bpf/progs/file_reader_fail.c    |    4 +-
 tools/testing/selftests/bpf/progs/irq.c       |    4 +-
 tools/testing/selftests/bpf/progs/iters.c     |    6 +-
 .../selftests/bpf/progs/iters_state_safety.c  |   14 +-
 .../selftests/bpf/progs/iters_testmod.c       |    4 +-
 .../selftests/bpf/progs/iters_testmod_seq.c   |    4 +-
 .../selftests/bpf/progs/map_kptr_fail.c       |    2 +-
 .../selftests/bpf/progs/percpu_alloc_fail.c   |    4 +-
 .../testing/selftests/bpf/progs/rbtree_fail.c |    6 +-
 .../bpf/progs/refcounted_kptr_fail.c          |    2 +-
 tools/testing/selftests/bpf/progs/stack_arg.c |  254 ++++
 .../selftests/bpf/progs/stack_arg_fail.c      |  124 ++
 .../selftests/bpf/progs/stack_arg_kfunc.c     |  164 +++
 .../testing/selftests/bpf/progs/stream_fail.c |    2 +-
 .../selftests/bpf/progs/task_kfunc_failure.c  |   18 +-
 .../selftests/bpf/progs/task_work_fail.c      |    6 +-
 .../selftests/bpf/progs/test_bpf_nf_fail.c    |    8 +-
 .../bpf/progs/test_kfunc_dynptr_param.c       |    2 +-
 .../bpf/progs/test_kfunc_param_nullable.c     |    2 +-
 .../selftests/bpf/progs/verifier_bits_iter.c  |    4 +-
 .../bpf/progs/verifier_bpf_fastcall.c         |   24 +-
 .../selftests/bpf/progs/verifier_may_goto_1.c |   12 +-
 .../bpf/progs/verifier_ref_tracking.c         |    6 +-
 .../selftests/bpf/progs/verifier_sdiv.c       |   64 +-
 .../selftests/bpf/progs/verifier_stack_arg.c  |  433 ++++++
 .../selftests/bpf/progs/verifier_vfs_reject.c |    8 +-
 .../testing/selftests/bpf/progs/wq_failures.c |    2 +-
 .../selftests/bpf/test_kmods/bpf_testmod.c    |   73 +
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |   26 +
 tools/testing/selftests/bpf/verifier/calls.c  |   14 +-
 52 files changed, 2533 insertions(+), 561 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/stack_arg_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_stack_arg.c
Signature · 1 line
2.52.0