Skip to content

RFC bpf: Introduce 64bit bitops kfuncs

Leon Hwang bpf 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest4 patches 2 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4bpf: Introduce 64bit bitops kfuncsinclude/linux/btf.h include/linux/filter.h kernel/bpf/core.c +2 more+142 -02 replies
2/4bpf, x86: Add 64bit bitops kfuncs support for x86_64arch/x86/net/bpf_jit_comp.c+153 -00 replies
3/4bpf, arm64: Add 64bit bitops kfuncs supportarch/arm64/net/bpf_jit_comp.c+143 -00 replies
4/4selftests/bpf: Add tests for 64bit bitops kfuncstools/testing/selftests/bpf/bpf_experimental.h tools/testing/selftests/bpf/prog_tests/bitops.c tools/testing/selftests/bpf/progs/bitops.c+264 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH bpf-next 0/4] bpf: Introduce 64bit bitops kfuncs

Introduce the following 64-bit bitops kfuncs for x86_64 and arm64:

* bpf_clz64(): Count leading zeros.
* bpf_ctz64(): Count trailing zeros.
* bpf_ffs64(): Find first set bit, 1-based index, returns 0 when input is 0.
* bpf_fls64(): Find last set bit, 1-based index.
* bpf_bitrev64(): Reverse bits.
* bpf_popcnt64(): Population count.
* bpf_rol64(): Rotate left.
* bpf_ror64(): Rotate right.

Especially,

* bpf_clz64(0) = 64
* bpf_ctz64(0) = 64
* bpf_ffs64(0) = 0
* bpf_fls64(0) = 0

bpf_ffs64() was previously discussed in "bpf: Add generic kfunc bpf_ffs64()" [1].

Background

In the earlier bpf_ffs64() discussion, the main concern with exposing such operations as generic kfuncs was ABI cost. A normal kfunc call follows the BPF calling convention, which forces the compiler/JIT to treat R1-R5 as call-clobbered, resulting in unnecessary spill/fill compared to a dedicated instruction.

This RFC keeps the user-facing API as kfuncs, but avoids the ABI cost in the fast path. The verifier rewrites supported bitops kfunc calls into a single internal ALU64 encoding (BPF_BITOPS with an immediate selector), and JIT backends emit native instructions directly. As a result, these kfuncs behave like ISA operations once loaded, rather than real helper calls.

To make this contract explicit, the kfuncs are marked with a new KF_MUST_INLINE flag: program load fails with -EOPNOTSUPP if the active JIT backend cannot inline a particular operation. This keeps the cost predictable and avoids silent slow fallbacks. A weak hook, bpf_jit_inlines_bitops(), allows each JIT backend to advertise support on a per-operation basis (and potentially based on CPU features).

Most operations are also tagged KF_FASTCALL to avoid clobbering unused argument registers. bpf_rol64() and bpf_ror64() are the exception on x86_64, where variable rotates require CL (BPF_REG_4).

Selftests output

On x86_64:

 #18/1    bitops/clz64:OK
 #18/2    bitops/ctz64:OK
 #18/3    bitops/ffs64:OK
 #18/4    bitops/fls64:OK
 #18/5    bitops/bitrev64:SKIP
 #18/6    bitops/popcnt64:OK
 #18/7    bitops/rol64:OK
 #18/8    bitops/ror64:OK
 #18      bitops:OK (SKIP: 1/8)

Summary: 1/7 PASSED, 1 SKIPPED, 0 FAILED

On arm64:

 #18/1    bitops/clz64:OK
 #18/2    bitops/ctz64:OK
 #18/3    bitops/ffs64:OK
 #18/4    bitops/fls64:OK
 #18/5    bitops/bitrev64:OK
 #18/6    bitops/popcnt64:SKIP
 #18/7    bitops/rol64:OK
 #18/8    bitops/ror64:OK
 #18      bitops:OK (SKIP: 1/8)

Summary: 1/7 PASSED, 1 SKIPPED, 0 FAILED

Open questions

1. Should these operations be exposed as a proper BPF ISA extension (new ALU64 ops) instead of a kfunc API plus verifier rewrite? This RFC takes the kfunc route to iterate without immediately committing to new uapi instruction semantics, while still ensuring instruction-like codegen.

2. For operations without a reasonable native implementation on some targets (e.g. bitrev64 on x86_64; popcnt64 on arm64 without touching SIMD registers), should we allow a true generic fallback by dropping KF_MUST_INLINE for those ops, or keep the "no-inline == reject" behavior for predictability?

Links:
[1] https://lore.kernel.org/bpf/[email protected]/

Leon Hwang (4):
bpf: Introduce 64bit bitops kfuncs
bpf, x86: Add 64bit bitops kfuncs support for x86_64 bpf, arm64: Add 64bit bitops kfuncs support selftests/bpf: Add tests for 64bit bitops kfuncs

 arch/arm64/net/bpf_jit_comp.c                 | 143 ++++++++++++++
 arch/x86/net/bpf_jit_comp.c                   | 153 ++++++++++++++
 include/linux/btf.h                           |   1 +
 include/linux/filter.h                        |  20 ++
 kernel/bpf/core.c                             |   6 +
 kernel/bpf/helpers.c                          |  50 +++++
 kernel/bpf/verifier.c                         |  65 ++++++
 .../testing/selftests/bpf/bpf_experimental.h  |   9 +
 .../testing/selftests/bpf/prog_tests/bitops.c | 186 ++++++++++++++++++
 tools/testing/selftests/bpf/progs/bitops.c    |  69 +++++++
 10 files changed, 702 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/bitops.c
 create mode 100644 tools/testing/selftests/bpf/progs/bitops.c

--
2.52.0