bpf: Support by-value struct and __int128 arguments
Status from this message: This series was applied to bpf/bpf-next.git (master)
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v4 Latest15 patches 2 repliesApplied 15/15
- v315 patches 8 repliesSuperseded
- v212 patches 9 repliesSuperseded
Earlier versions · 1
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v1 | 12 patches | 20 replies | Superseded |
Changes in v4 from the cover letter
- v3: https://lore.kernel.org/bpf/[email protected]/
- Rebase on top master branch. Mostly no change except the commit "Recognize by-value struct and __int128 kfunc arguments" which needs some adjustment.
- Commit "bpf, arm64: Place trampoline arguments by the arm64 calling convention" is new. The commit intends to fix the bug when any argument has 16-byte alignment requirement.
Patches in v4 All 15 applied
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/15 | bpf: Read a kfunc's __sz argument only when it is in a registerkernel/ | +3 | 1 reply | Applied |
| 2/15 | selftests/bpf: Add a test for an __int128 by-value argumenttools/ tools/ | +42 | 0 replies | Applied |
| 3/15 | bpf: Rename bpf_subprog_info::arg_cnt to arg_slot_cntinclude/ kernel/ kernel/ +1 more | +17 | 0 replies | Applied |
| 4/15 | bpf: Index global function arguments by argument slotkernel/ | +24 | 0 replies | Applied |
| 5/15 | bpf: Support by-value struct arguments up to 16 byteskernel/ kernel/ | +68 | 0 replies | Applied |
| 6/15 | bpf: Support __int128 as a by-value function argumentkernel/ tools/ | +4 | 0 replies | Applied |
| 7/15 | bpf: Rename bpf_call_summary::num_params to arg_slot_cntinclude/ kernel/ kernel/ | +10 | 0 replies | Applied |
| 8/15 | bpf: Recognize by-value struct and __int128 kfunc argumentskernel/ | +153 | 0 replies | Applied |
| 9/15 | bpf: Prepare kfunc arguments for the JIT from an ABI descriptioninclude/ include/ include/ +3 more | +231 | 0 replies | Applied |
| 10/15 | bpf, x86: Move kfunc arguments into the x86-64 calling conventionarch/ | +75 | 0 replies | Applied |
| 11/15 | bpf, arm64: Place trampoline arguments by the arm64 calling conventionarch/ | +54 | 1 reply | Applied |
| 12/15 | bpf, arm64: Move kfunc arguments into the arm64 calling conventionarch/ | +58 | 0 replies | Applied |
| 13/15 | selftests/bpf: Add C tests for by-value arguments up to 16 bytestools/ | +165 | 0 replies | Applied |
| 14/15 | selftests/bpf: Add inline-asm tests for by-value argumentstools/ tools/ | +197 | 0 replies | Applied |
| 15/15 | selftests/bpf: Add tests for by-value kfunc argumentstools/ tools/ tools/ +3 more | +270 | 0 replies | Applied |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.
Cover letter [PATCH bpf-next v4 00/15] bpf: Support by-value struct and __int128 arguments
A global function or a kfunc taking a struct or union by value is rejected today:
Arg#1 type STRUCT in tar() is not supported yet. Unrecognized R2 type STRUCT
and an __int128 argument is accepted but mis-counted: the compiler passes it in two registers while the verifier gives it one, so every argument after it is checked against the wrong register and the program is rejected for a register its source never names.
The compiler passes such a value in one argument register per eightbyte, so a value of at most 16 bytes arrives in one or two consecutive registers. This series teaches the verifier to describe that: sub->args[] gains one entry per argument slot rather than one per BTF parameter, and the kfunc argument walk counts slots the same way. A value may only be composed of scalars, as a pointer would reach the callee as an opaque scalar, losing the provenance and reference tracking that make it safe to use. A global function still has no stack arguments, so all of its slots have to fit in the argument registers.
For a kfunc the two calling conventions have to agree, and they do not always. Where the BPF convention splits a value between the last argument register and the stack, x86_64 moves the whole of it to the stack and gives the register to the argument that follows, while arm64 rounds the register number up to an even one for a 16-byte aligned value. For stack arguments, both x86_64 and arm64 will have 16-byte alignment for those 16-byte aligned parameters. Otherwise the alignment will be 8-byte. This is implemented in x86_64 and arm64 jit.
arm64 has some issues with trampolines due to certain 16-byte align requirement, similar to 16-byte align for kfunc arguments. calc_arg_aux(), save_args() and restore_args() all need changes to cope with 16-byte alignment.
Patch 1 fixed a pre-existing bug in bpf_kfunc_stack_access_bytes(). Patch 2 records the __int128 failure as it stands today, which patch 6 turns into a success. Patches 3 to 6 rename some fields, index the arguments of a global function by slot and accept a by-value struct and an __int128. Patches 8 and 9 have the same for a kfunc call. Patches 10 to 12 place the arguments per the kernel calling convention in the x86-64 and arm64 JITs. Patches 13 to 15 are the tests.
Tested on x86-64 and arm64 with test_progs. The by-value struct tests are built with clang, as GCC passes an aggregate by invisible reference, and some of them need __BPF_FEATURE_STACK_ARGUMENT.
Changelog:
v3 -> v4:
- v3: https://lore.kernel.org/bpf/[email protected]/
- Rebase on top master branch. Mostly no change except the commit "Recognize by-value struct and __int128 kfunc arguments" which needs some adjustment.
- Commit "bpf, arm64: Place trampoline arguments by the arm64 calling convention" is new. The commit intends to fix the bug when any argument has 16-byte alignment requirement.
v2 -> v3:
- v2: https://lore.kernel.org/bpf/[email protected]/
- Rename test name from verifier_int_arg to verifier_aggregate_arg.
- Rename from arg_cnt to arg_slot_cnt to make codes easy to understand.
- Fix an issue for by-value struct arguments up to 16 bytes where the bound check, e.g. number of slots is more than 5 or not, etc, is missing after all arguments are processed.
- Increase local slot size from MAX_BPF_FUNC_ARGS to 2 * MAX_BPF_FUNC_ARGS to avoid out-of-bound write.
- One codegen improvement for arm64 jit.
- Added a not-to-merge patch (patch 15) to workaround some tests. The pahole patch
https://lore.kernel.org/bpf/[email protected]/
is pending.
v1 -> v2:
- v1: https://lore.kernel.org/bpf/[email protected]/
- Fix a few issues due to potential out of bound access.
- Fix a few missed cases for kfunc with '> 8 byte' arguments.
- Add jit support for x86_64 and arm64. Previously certain functions are rejected, and now these functions can succeed.
Yonghong Song (15):
bpf: Read a kfunc's __sz argument only when it is in a register selftests/bpf: Add a test for an __int128 by-value argument
bpf: Rename bpf_subprog_info::arg_cnt to arg_slot_cnt
bpf: Index global function arguments by argument slot
bpf: Support by-value struct arguments up to 16 bytes
bpf: Support __int128 as a by-value function argument
bpf: Rename bpf_call_summary::num_params to arg_slot_cnt
bpf: Recognize by-value struct and __int128 kfunc arguments
bpf: Prepare kfunc arguments for the JIT from an ABI description bpf, x86: Move kfunc arguments into the x86-64 calling convention bpf, arm64: Place trampoline arguments by the arm64 calling convention bpf, arm64: Move kfunc arguments into the arm64 calling convention selftests/bpf: Add C tests for by-value arguments up to 16 bytes selftests/bpf: Add inline-asm tests for by-value arguments selftests/bpf: Add tests for by-value kfunc arguments
arch/arm64/net/bpf_jit_comp.c | 146 +++++++-- arch/x86/net/bpf_jit_comp.c | 78 ++++- include/linux/bpf.h | 15 + include/linux/bpf_verifier.h | 9 +- include/linux/filter.h | 32 ++ kernel/bpf/btf.c | 124 ++++++-- kernel/bpf/core.c | 94 ++++++ kernel/bpf/liveness.c | 10 +- kernel/bpf/verifier.c | 298 +++++++++++++++--- .../selftests/bpf/prog_tests/aggregate_arg.c | 11 + .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/aggregate_arg_func.c | 188 +++++++++++ .../selftests/bpf/progs/aggregate_arg_kfunc.c | 168 ++++++++++ .../testing/selftests/bpf/progs/arena_kfunc.c | 15 + .../selftests/bpf/progs/stack_arg_fail.c | 10 - .../bpf/progs/verifier_aggregate_arg.c | 200 ++++++++++++ .../bpf/progs/verifier_stack_arg_order.c | 4 +- .../selftests/bpf/test_kmods/bpf_testmod.c | 59 +++- .../bpf/test_kmods/bpf_testmod_kfunc.h | 27 ++ 19 files changed, 1348 insertions(+), 142 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_arg.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_func.c create mode 100644 tools/testing/selftests/bpf/progs/aggregate_arg_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_aggregate_arg.c
Signature · 1 line
2.53.0-Meta