Skip to content

bpf: Support aggregate return values up to 16 bytes

Yonghong Song bpf sashiko-reviews 7 versions – Applied

Status from this message: This series was applied to bpf/bpf-next.git (master)

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v7 Latest10 patches 11 repliesApplied 10/10
  2. v610 patches 15 repliesSuperseded
  3. v511 patches 21 repliesSuperseded
Earlier versions · 4
VersionPostedPatchesRepliesStatus
v413 patches32 repliesSuperseded
v313 patches5 repliesSuperseded
v213 patches5 repliesSuperseded
v112 patches35 repliesSuperseded

Changes in v7 from the cover letter

- v6: https://lore.kernel.org/bpf/[email protected]/
- Simplify type checking in function bpf_compute_subprog_ret_regs().
- Add __load_if_JITed() for a few negative tests.
- Adjust a few comments and error message.

Patches in v7 All 10 applied

#Subject and filesSizeRepliesApplied
1/10bpf: Factor check_global_ret_scalar_reg() out of the global return checkkernel/bpf/verifier.c+20 -120 repliesApplied
2/10bpf: Add helpers to describe the R0:R2 return register pairinclude/linux/bpf_verifier.h kernel/bpf/verifier.c+62 -113 repliesApplied
3/10bpf: Wire up JIT support for 16-byte kfunc returnsarch/arm64/net/bpf_jit_comp.c arch/riscv/net/bpf_jit_comp64.c arch/x86/net/bpf_jit_comp.c +3 more+48 -71 replyApplied
4/10bpf: Handle R2 as a return register in precision backtrackingkernel/bpf/backtrack.c+30 -161 replyApplied
5/10bpf: Account R2 of register-pair returns in live register analysiskernel/bpf/liveness.c+14 -62 repliesApplied
6/10bpf: Add verifier support for 16-byte returns in R0:R2kernel/bpf/btf.c kernel/bpf/verifier.c+67 -71 replyApplied
7/10bpf: Enable aggregate return types up to 16 bytesinclude/linux/bpf_verifier.h kernel/bpf/btf.c kernel/bpf/verifier.c +1 more+30 -152 repliesApplied
8/10selftests/bpf: Add C tests for 16-byte returns in R0:R2tools/testing/selftests/bpf/prog_tests/verifier.c tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +1 more+207 -01 replyApplied
9/10selftests/bpf: Add inline-asm and subprog tests for R0:R2 returnstools/testing/selftests/bpf/prog_tests/aggregate_ret.c tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c tools/testing/selftests/bpf/progs/aggregate_ret_func.c +6 more+511 -00 repliesApplied
10/10Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2Documentation/bpf/kfuncs.rst+65 -00 repliesApplied

Reviews trailers per patch

8 of 10 patches have a Reviewed-by or Acked-by.

PatchEduard Zingerman
1/10A-b
2/10
3/10A-b
4/10A-b
5/10A-b
6/10A-b
7/10A-b
8/10A-b
9/10A-b
10/10

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v7 thread

Cover letter [PATCH bpf-next v7 00/10] bpf: Support aggregate return values up to 16 bytes

LLVM 23 can return an __int128, or a struct/union larger than 8 bytes and no larger than 16 bytes, in the BPF R0:R2 register pair [1][2]. Before that the BPF backend could not return such values at all: a by-value aggregate return was rejected at compile time with "aggregate returns are not supported", and an __int128 return failed in the backend with "unable to allocate function return #1".

This series teaches the kernel the same convention, so that BPF programs and kfuncs can return these values. The first 8 bytes of the value come back in R0 and the second 8 bytes in R2. It applies to kfunc returns and to BPF-to-BPF subprogram returns, both global and static. The main program is unchanged: its return value is the program's exit code, so a return larger than 8 bytes is still rejected at BPF_EXIT.

Patches 1-2 are preparation: patch 1 factors out the per-register check used by the global return path, and patch 2 adds the shared helpers that answer "does this subprogram return a register pair", so that the patches which follow can be ordered independently. Patch 3 wires up the JIT side. Patches 4-5 teach precision backtracking and live register analysis about R2 as a second return register, ahead of the patch that starts modeling it. Patch 6 adds the verifier support proper. Patch 7 relaxes btf_distill_func_proto() and btf_validate_return_type(), which is what makes the whole thing reachable. Patches 8-9 add selftests and patch 10 documents the convention.

Constraints worth calling out:

- A by-value struct or union returned by a kfunc or by a global subprogram must be composed only of scalars. The verifier models the returned register bits as an unknown scalar, so a pointer member would be laundered into one and escape provenance and reference tracking. A static subprogram is verified inline and is not restricted this way.

- Returning the pair from a kfunc needs the JIT to place the second half into R2, which is architecture-specific work. Architectures opt in through bpf_jit_supports_kfunc_ret_reg_pair(); x86_64, arm64 and riscv64 do so here, and elsewhere bpf_add_kfunc_call() rejects such a kfunc with -EOPNOTSUPP. A register-pair return from a BPF subprogram needs no such capability.

- The pair is modelled only when the JIT is enabled, since the interpreter propagates R0 alone out of a subprogram. Modelling it sets jit_required, so a JIT fallback to the interpreter becomes a load failure rather than a silent divergence from what was verified. With the JIT off, the pair is not modelled and a caller reading R2 fails verification.

- The compiler side requires LLVM 23 or newer. The selftests written in C sit behind a __clang_major__ guard; an older compiler builds a dummy test instead, whose description says why nothing was exercised. The inline-asm tests run everywhere, apart from the few whose callee prototype returns a struct or union by value, which are guarded the same way.

[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876

Changelog:
v6 -> v7:
- v6: https://lore.kernel.org/bpf/[email protected]/
- Simplify type checking in function bpf_compute_subprog_ret_regs().
- Add __load_if_JITed() for a few negative tests.
- Adjust a few comments and error message.
v5 -> v6:
- v5: https://lore.kernel.org/bpf/[email protected]/
- Returning R0 only if jit is not enabled. If jit is enabled, main prog will not force jit even if main prog may return R0:R2.
- Simplify precision backtracking.
- Check returning R0:R2 in btf_check_func_type_match().
- Remove some redundant tests.
v4 -> v5:
- v4: https://lore.kernel.org/bpf/[email protected]/
- Removed R0:R2 restriction on callback functions and its related tests.
- Simplify the code for backtracking.
- Reduce comments and verify R0/R2 together.
- Use RUN_TESTS for selftests.
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Fix a few kernel comment format.
- Guard more kfunc's with x86_64/arm64 only to avoid s390x failure.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Add additional guard with __SIZEOF_INT128__ for 32bit kernel.
- Guard little endian to avoid arm64 big endian for selftests.
- Fix test failures for gcc15.
- Rebase to avoid conflict with latest master branch.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Split the R0:R2 helpers out of the verifier patch into their own preparation patch, and reordered the series so the core verifier patch comes after the infrastructure it depends on.
- New patch rejecting callbacks that return more than 8 bytes, both helper/kfunc callbacks and exception callbacks, with selftests.
- New patch rejecting a register-pair return once btf_check_subprog_call() has marked the subprogram's BTF unreliable, rather than silently mistracking R2.
- Folded "bpf: Force JIT for programs using the R0:R2 register pair" into the verifier patch.
- Dropped "bpf: Reject >8 byte return values on return-reading trampoline paths" and its selftests; that went in separately as commit c48796aa6c39.
- Described the register mapping as the first and second 8 bytes rather than the low and high 64 bits, which is only correct on little-endian, and reworded "16-byte" to "up to 16 bytes" where the range 9..16 was meant.

Yonghong Song (10):
bpf: Factor check_global_ret_scalar_reg() out of the global return
check
bpf: Add helpers to describe the R0:R2 return register pair
bpf: Wire up JIT support for 16-byte kfunc returns
bpf: Handle R2 as a return register in precision backtracking
bpf: Account R2 of register-pair returns in live register analysis
bpf: Add verifier support for 16-byte returns in R0:R2
bpf: Enable aggregate return types up to 16 bytes selftests/bpf: Add C tests for 16-byte returns in R0:R2 selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2

 Documentation/bpf/kfuncs.rst                  |  65 +++++
 arch/arm64/net/bpf_jit_comp.c                 |   5 +
 arch/riscv/net/bpf_jit_comp64.c               |   5 +
 arch/x86/net/bpf_jit_comp.c                   |  27 +-
 include/linux/bpf_verifier.h                  |   9 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  46 ++--
 kernel/bpf/btf.c                              |  29 ++-
 kernel/bpf/core.c                             |   5 +
 kernel/bpf/liveness.c                         |  20 +-
 kernel/bpf/verifier.c                         | 192 +++++++++++---
 .../selftests/bpf/prog_tests/aggregate_ret.c  |  11 +
 .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |  16 ++
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/aggregate_ret_func.c  | 237 ++++++++++++++++++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c | 121 +++++++++
 .../bpf/progs/aggregate_ret_target.c          |  29 +++
 .../bpf/progs/compute_live_registers.c        |  28 +++
 .../selftests/bpf/progs/exceptions_fail.c     |   2 +-
 .../selftests/bpf/progs/freplace_ret_pair.c   |  12 +
 .../bpf/progs/verifier_aggregate_ret.c        | 178 +++++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  55 ++++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  29 +++
 23 files changed, 1052 insertions(+), 72 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c
 create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
Signature · 1 line
2.53.0-Meta