bpf: Support aggregate return values up to 16 bytes
Status from this message: This series was applied to bpf/bpf-next.git (master)
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v7 Latest10 patches 11 repliesApplied 10/10
- v610 patches 15 repliesSuperseded
- v511 patches 21 repliesSuperseded
Earlier versions · 4
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v4 | 13 patches | 32 replies | Superseded | |
| v3 | 13 patches | 5 replies | Superseded | |
| v2 | 13 patches | 5 replies | Superseded | |
| v1 | 12 patches | 35 replies | Superseded |
Changes in v7 from the cover letter
- v6: https://lore.kernel.org/bpf/[email protected]/
- Simplify type checking in function bpf_compute_subprog_ret_regs().
- Add __load_if_JITed() for a few negative tests.
- Adjust a few comments and error message.
Patches in v7 All 10 applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/10 | bpf: Factor check_global_ret_scalar_reg() out of the global return checkkernel/ | +20 | 0 replies | Applied |
| 2/10 | bpf: Add helpers to describe the R0:R2 return register pairinclude/ kernel/ | +62 | 3 replies | Applied |
| 3/10 | bpf: Wire up JIT support for 16-byte kfunc returnsarch/ arch/ arch/ +3 more | +48 | 1 reply | Applied |
| 4/10 | bpf: Handle R2 as a return register in precision backtrackingkernel/ | +30 | 1 reply | Applied |
| 5/10 | bpf: Account R2 of register-pair returns in live register analysiskernel/ | +14 | 2 replies | Applied |
| 6/10 | bpf: Add verifier support for 16-byte returns in R0:R2kernel/ kernel/ | +67 | 1 reply | Applied |
| 7/10 | bpf: Enable aggregate return types up to 16 bytesinclude/ kernel/ kernel/ +1 more | +30 | 2 replies | Applied |
| 8/10 | selftests/bpf: Add C tests for 16-byte returns in R0:R2tools/ tools/ tools/ +1 more | +207 | 1 reply | Applied |
| 9/10 | selftests/bpf: Add inline-asm and subprog tests for R0:R2 returnstools/ tools/ tools/ +6 more | +511 | 0 replies | Applied |
| 10/10 | Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2Documentation/ | +65 | 0 replies | Applied |
Reviews trailers per patch
8 of 10 patches have a Reviewed-by or Acked-by.
| Patch | Eduard Zingerman |
|---|---|
| 1/10 | A-b |
| 2/10 | |
| 3/10 | A-b |
| 4/10 | A-b |
| 5/10 | A-b |
| 6/10 | A-b |
| 7/10 | A-b |
| 8/10 | A-b |
| 9/10 | A-b |
| 10/10 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH bpf-next v7 00/10] bpf: Support aggregate return values up to 16 bytes
LLVM 23 can return an __int128, or a struct/union larger than 8 bytes and no larger than 16 bytes, in the BPF R0:R2 register pair [1][2]. Before that the BPF backend could not return such values at all: a by-value aggregate return was rejected at compile time with "aggregate returns are not supported", and an __int128 return failed in the backend with "unable to allocate function return #1".
This series teaches the kernel the same convention, so that BPF programs and kfuncs can return these values. The first 8 bytes of the value come back in R0 and the second 8 bytes in R2. It applies to kfunc returns and to BPF-to-BPF subprogram returns, both global and static. The main program is unchanged: its return value is the program's exit code, so a return larger than 8 bytes is still rejected at BPF_EXIT.
Patches 1-2 are preparation: patch 1 factors out the per-register check used by the global return path, and patch 2 adds the shared helpers that answer "does this subprogram return a register pair", so that the patches which follow can be ordered independently. Patch 3 wires up the JIT side. Patches 4-5 teach precision backtracking and live register analysis about R2 as a second return register, ahead of the patch that starts modeling it. Patch 6 adds the verifier support proper. Patch 7 relaxes btf_distill_func_proto() and btf_validate_return_type(), which is what makes the whole thing reachable. Patches 8-9 add selftests and patch 10 documents the convention.
Constraints worth calling out:
- A by-value struct or union returned by a kfunc or by a global subprogram must be composed only of scalars. The verifier models the returned register bits as an unknown scalar, so a pointer member would be laundered into one and escape provenance and reference tracking. A static subprogram is verified inline and is not restricted this way.
- Returning the pair from a kfunc needs the JIT to place the second half into R2, which is architecture-specific work. Architectures opt in through bpf_jit_supports_kfunc_ret_reg_pair(); x86_64, arm64 and riscv64 do so here, and elsewhere bpf_add_kfunc_call() rejects such a kfunc with -EOPNOTSUPP. A register-pair return from a BPF subprogram needs no such capability.
- The pair is modelled only when the JIT is enabled, since the interpreter propagates R0 alone out of a subprogram. Modelling it sets jit_required, so a JIT fallback to the interpreter becomes a load failure rather than a silent divergence from what was verified. With the JIT off, the pair is not modelled and a caller reading R2 fails verification.
- The compiler side requires LLVM 23 or newer. The selftests written in C sit behind a __clang_major__ guard; an older compiler builds a dummy test instead, whose description says why nothing was exercised. The inline-asm tests run everywhere, apart from the few whose callee prototype returns a struct or union by value, which are guarded the same way.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876
Changelog:
v6 -> v7:
- v6: https://lore.kernel.org/bpf/[email protected]/
- Simplify type checking in function bpf_compute_subprog_ret_regs().
- Add __load_if_JITed() for a few negative tests.
- Adjust a few comments and error message.
v5 -> v6:
- v5: https://lore.kernel.org/bpf/[email protected]/
- Returning R0 only if jit is not enabled. If jit is enabled, main prog will not force jit even if main prog may return R0:R2.
- Simplify precision backtracking.
- Check returning R0:R2 in btf_check_func_type_match().
- Remove some redundant tests.
v4 -> v5:
- v4: https://lore.kernel.org/bpf/[email protected]/
- Removed R0:R2 restriction on callback functions and its related tests.
- Simplify the code for backtracking.
- Reduce comments and verify R0/R2 together.
- Use RUN_TESTS for selftests.
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Fix a few kernel comment format.
- Guard more kfunc's with x86_64/arm64 only to avoid s390x failure.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Add additional guard with __SIZEOF_INT128__ for 32bit kernel.
- Guard little endian to avoid arm64 big endian for selftests.
- Fix test failures for gcc15.
- Rebase to avoid conflict with latest master branch.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Split the R0:R2 helpers out of the verifier patch into their own preparation patch, and reordered the series so the core verifier patch comes after the infrastructure it depends on.
- New patch rejecting callbacks that return more than 8 bytes, both helper/kfunc callbacks and exception callbacks, with selftests.
- New patch rejecting a register-pair return once btf_check_subprog_call() has marked the subprogram's BTF unreliable, rather than silently mistracking R2.
- Folded "bpf: Force JIT for programs using the R0:R2 register pair" into the verifier patch.
- Dropped "bpf: Reject >8 byte return values on return-reading trampoline paths" and its selftests; that went in separately as commit c48796aa6c39.
- Described the register mapping as the first and second 8 bytes rather than the low and high 64 bits, which is only correct on little-endian, and reworded "16-byte" to "up to 16 bytes" where the range 9..16 was meant.
Yonghong Song (10):
bpf: Factor check_global_ret_scalar_reg() out of the global return
check
bpf: Add helpers to describe the R0:R2 return register pair
bpf: Wire up JIT support for 16-byte kfunc returns
bpf: Handle R2 as a return register in precision backtracking
bpf: Account R2 of register-pair returns in live register analysis
bpf: Add verifier support for 16-byte returns in R0:R2
bpf: Enable aggregate return types up to 16 bytes selftests/bpf: Add C tests for 16-byte returns in R0:R2 selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2
Documentation/bpf/kfuncs.rst | 65 +++++ arch/arm64/net/bpf_jit_comp.c | 5 + arch/riscv/net/bpf_jit_comp64.c | 5 + arch/x86/net/bpf_jit_comp.c | 27 +- include/linux/bpf_verifier.h | 9 + include/linux/filter.h | 1 + kernel/bpf/backtrack.c | 46 ++-- kernel/bpf/btf.c | 29 ++- kernel/bpf/core.c | 5 + kernel/bpf/liveness.c | 20 +- kernel/bpf/verifier.c | 192 +++++++++++--- .../selftests/bpf/prog_tests/aggregate_ret.c | 11 + .../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 16 ++ .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/aggregate_ret_func.c | 237 ++++++++++++++++++ .../selftests/bpf/progs/aggregate_ret_kfunc.c | 121 +++++++++ .../bpf/progs/aggregate_ret_target.c | 29 +++ .../bpf/progs/compute_live_registers.c | 28 +++ .../selftests/bpf/progs/exceptions_fail.c | 2 +- .../selftests/bpf/progs/freplace_ret_pair.c | 12 + .../bpf/progs/verifier_aggregate_ret.c | 178 +++++++++++++ .../selftests/bpf/test_kmods/bpf_testmod.c | 55 ++++ .../bpf/test_kmods/bpf_testmod_kfunc.h | 29 +++ 23 files changed, 1052 insertions(+), 72 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
Signature · 1 line
2.53.0-Meta