bpf: tailcall: Eliminate max_entries and bpf_func access at runtime
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest4 patches 7 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/4 | bpf: tailcall: Introduce bpf_arch_tail_call_prologue_offsetinclude/ kernel/ | +27 | 1 reply | |
| 2/4 | bpf, x64: tailcall: Eliminate max_entries and bpf_func access at runtimearch/ kernel/ | +66 | 0 replies | |
| 3/4 | bpf, arm64: tailcall: Eliminate max_entries and bpf_func access at runtimearch/ | +51 | 0 replies | |
| 4/4 | bpf, lib/test_bpf: Fix broken tailcall testslib/ | +34 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [PATCH bpf-next 0/4] bpf: tailcall: Eliminate max_entries and bpf_func access at runtime
This patch series optimizes BPF tail calls on x86_64 and arm64 by eliminating runtime memory accesses for max_entries and 'prog->bpf_func' when the prog array map is known at verification time.
Currently, every tail call requires:
1. Loading max_entries from the prog array map
2. Dereferencing 'prog->bpf_func' to get the target address
This series introduces a mechanism to precompute and cache the tail call target addresses (bpf_func + prologue_offset) in the prog array itself:
array->ptrs[max_entries + index] = prog->bpf_func + prologue_offset
When a program is added to or removed from the prog array, the cached target is atomically updated via xchg().
The verifier now encodes additional information in the tail call instruction's imm field:
- bits 0-7: map index in used_maps[]
- bits 8-15: dynamic array flag (1 if map pointer is poisoned)
- bits 16-31: poke table index + 1 for direct tail calls
For static tail calls (map known at verification time):
- max_entries is embedded as an immediate in the comparison instruction
- The cached target from array->ptrs[max_entries + index] is used
directly, avoiding the 'prog->bpf_func' dereference
For dynamic tail calls (map pointer poisoned):
- Fall back to runtime lookup of max_entries and prog->bpf_func
This reduces cache misses and improves tail call performance for the common case where the prog array is statically known.
Leon Hwang (4):
bpf: tailcall: Introduce bpf_arch_tail_call_prologue_offset bpf, x64: tailcall: Eliminate max_entries and bpf_func access at
runtime
bpf, arm64: tailcall: Eliminate max_entries and bpf_func access at
runtime
bpf, lib/test_bpf: Fix broken tailcall tests
arch/arm64/net/bpf_jit_comp.c | 71 +++++++++++++++++++++++++---------- arch/x86/net/bpf_jit_comp.c | 51 ++++++++++++++++++------- include/linux/bpf.h | 1 + kernel/bpf/arraymap.c | 27 ++++++++++++- kernel/bpf/verifier.c | 30 ++++++++++++++- lib/test_bpf.c | 39 ++++++++++++++++--- 6 files changed, 178 insertions(+), 41 deletions(-)
--
2.52.0