bpf: Inline the numeric open-coded iterator kfuncs
Subsystems (4)
v2 cover bpf: Inline the numeric open-coded iterator kfuncs
The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to tear it down. The verifier emits these as ordinary kfunc calls, so a bpf_for() loop pays function-call overhead on setup, teardown, and — most importantly — on every single iteration via bpf_iter_num_next().
All three kfuncs are tiny and only touch the 8-byte on-stack iterator state (struct bpf_iter_num_kern { int cur; int end; }). That makes them good candidates for inlining, the same way several other special kfuncs are already open-coded in bpf_fixup_kfunc_call(). This series replaces each of the three calls with an equivalent inline BPF instruction sequence:
- bpf_iter_num_new(): keeps the (s64)end - (s64)start overflow check (emitted with sign-extending moves) and returns the same -EINVAL / -E2BIG / 0 results as the kfunc.
- bpf_iter_num_next(): the hot path. Since cur and end are int, the
kfunc's (s64)(s->cur + 1) >= s->end test reduces to a signed 32-bit
comparison of (s->cur + 1) against s->end, so the inlined code uses a
32-bit compare with no sign extension. - bpf_iter_num_destroy(): a single 8-byte store zeroing the state.
The emitted instructions are plain BPF and remain valid for the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.
Patch 4 adds a bench_bpf_for benchmark, modeled on the existing bpf_loop benchmark, that runs a bpf_for() loop with an empty body so the per-iteration iterator cost can be measured directly.
Results (./bench -p 1 --nr_loops 1000 bpf-for):
+--------+----------------------+----------------------+--------------+ | arch | inlined | kfunc calls | result | +--------+----------------------+----------------------+--------------+ | x86-64 | 12884 M/s (0.078 ns) | 5812 M/s (0.172 ns) | ~2.2x faster | +--------+----------------------+----------------------+--------------+ | arm64 | 546.1 M/s (1.831 ns) | 545.8 M/s (1.832 ns) | neutral | +--------+----------------------+----------------------+--------------+
On x86-64, removing the per-iteration call to bpf_iter_num_next() roughly doubles throughput. On arm64 the numbers are unchanged: the loop is bound by the load/store dependency chain on the on-stack iterator state rather than by call overhead, so inlining neither helps nor hurts there. It still removes the calls.
Changelog:
v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Don't use sign-extending moves as some JITs don't support them (AI)
Puranjay Mohan (4):
bpf: Inline bpf_iter_num_new() kfunc
bpf: Inline bpf_iter_num_next() kfunc
bpf: Inline bpf_iter_num_destroy() kfunc
selftests/bpf: Add bpf_for() benchmark
kernel/bpf/verifier.c | 81 ++++++++++++++ tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 4 + .../selftests/bpf/benchs/bench_bpf_for.c | 105 ++++++++++++++++++ .../selftests/bpf/benchs/run_bench_bpf_for.sh | 15 +++ .../selftests/bpf/progs/bpf_for_bench.c | 32 ++++++ 6 files changed, 239 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c
base-commit: 1d91ea01185656ac3ee63c5f9f6f8bde3c746b3d
Signature · 1 line
2.53.0-Meta
PATCH v2 1/4 bpf: Inline bpf_iter_num_new() kfunc
The numeric iterator kfuncs bpf_iter_num_{new,next,destroy}() back the open-coded iterator macro bpf_for() and are emitted as regular kfunc calls by the verifier. bpf_iter_num_new() is small and only touches the on-stack iterator state, so the verifier can open-code it and avoid the call overhead of setting up an iterator.
Inline it in bpf_fixup_kfunc_call() by replacing the call with an equivalent instruction sequence. R1 holds the pointer to the on-stack bpf_iter_num, while R2 and R3 hold the start and end arguments. The arithmetic and the error paths mirror the kfunc exactly, so a program that inspects the return value keeps observing the same -EINVAL / -E2BIG / 0 results.
The kfunc guards against overflow with a (s64)end - (s64)start range check. The start > end case is rejected first, so start <= end at that point and the range end - start fits in a u32. The inlined code computes it with a 32-bit subtraction that zero-extends into a 64-bit register and compares the result against BPF_MAX_LOOPS. Sign-extending the operands with movsx instead would emit a cpuv4 instruction after verification; JITs that do not implement movsx (e.g. x86-32, mips32, sparc64) decode it as a plain move and would miscompile the check.
The emitted instructions are plain BPF and are handled by the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.
--- kernel/bpf/verifier.c | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
kernel/bpf/verifier.c +38 -0
@@ -19721,6 +19721,42 @@ static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux, *cnt =4 ; } +/* +* Inline bpf_iter_num_new(). R1 holds the pointer to the iterator, R2 and R3 +* hold the (int) start and end arguments. Keep in sync with the kfunc in +* kernel/bpf/bpf_iter.c. +*/ +static int inline_bpf_iter_num_new (struct bpf_insn *insn_buf) +{ +/* if (start > end) goto einval; */ + insn_buf[0 ] =BPF_JMP32_REG (BPF_JSGT ,BPF_REG_2 ,BPF_REG_3 ,8 ); +/* +* start <= end here, so the range end - start fits in a u32; compute it +* as a 32-bit subtraction that zero-extends into r0. +*/ + insn_buf[1 ] =BPF_MOV32_REG (BPF_REG_0 ,BPF_REG_3 ); + insn_buf[2 ] =BPF_ALU32_REG (BPF_SUB ,BPF_REG_0 ,BPF_REG_2 ); +/* if (r0 > BPF_MAX_LOOPS) goto e2big; */ + insn_buf[3 ] =BPF_JMP_IMM (BPF_JSGT ,BPF_REG_0 ,BPF_MAX_LOOPS ,8 ); +/* s->cur = start - 1; */ + insn_buf[4 ] =BPF_ALU32_IMM (BPF_ADD ,BPF_REG_2 ,-1 ); + insn_buf[5 ] =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_2 ,0 ); +/* s->end = end; */ + insn_buf[6 ] =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_3 ,4 ); +/* return 0; */ + insn_buf[7 ] =BPF_MOV64_IMM (BPF_REG_0 ,0 ); + insn_buf[8 ] =BPF_JMP_A (5 ); +/* einval: s->cur = s->end = 0; return -EINVAL; */ + insn_buf[9 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + insn_buf[10 ] =BPF_MOV64_IMM (BPF_REG_0 , -EINVAL ); + insn_buf[11 ] =BPF_JMP_A (2 ); +/* e2big: s->cur = s->end = 0; return -E2BIG; */ + insn_buf[12 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + insn_buf[13 ] =BPF_MOV64_IMM (BPF_REG_0 , -E2BIG ); + +return 14 ; +} +int bpf_fixup_kfunc_call (struct bpf_verifier_env *env,struct bpf_insn *insn,struct bpf_insn *insn_buf,int insn_idx,int *cnt) {
@@ -19850,6 +19886,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn_buf[4 ] =BPF_ALU64_REG (BPF_SUB ,BPF_REG_0 ,BPF_REG_1 ); insn_buf[5 ] =BPF_ALU64_IMM (BPF_NEG ,BPF_REG_0 ,0 ); *cnt =6 ; + }else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) { + *cnt =inline_bpf_iter_num_new (insn_buf); }if (env->insn_aux_data[insn_idx].arg_prog) {
Signature · 1 line
2.53.0-Meta
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 1/4] bpf: Inline bpf_iter_num_new() kfunc
On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:
...
Quoted 20 lines
kernel/bpf/verifier.c +20
-0+static int inline_bpf_iter_num_new (struct bpf_insn *insn_buf) +{ +/* if (start > end) goto einval; */ + insn_buf[0 ] =BPF_JMP32_REG (BPF_JSGT ,BPF_REG_2 ,BPF_REG_3 ,8 ); +/* +* start <= end here, so the range end - start fits in a u32; compute it +* as a 32-bit subtraction that zero-extends into r0. +*/ + insn_buf[1 ] =BPF_MOV32_REG (BPF_REG_0 ,BPF_REG_3 ); + insn_buf[2 ] =BPF_ALU32_REG (BPF_SUB ,BPF_REG_0 ,BPF_REG_2 ); +/* if (r0 > BPF_MAX_LOOPS) goto e2big; */ + insn_buf[3 ] =BPF_JMP_IMM (BPF_JSGT ,BPF_REG_0 ,BPF_MAX_LOOPS ,8 ); +/* s->cur = start - 1; */ + insn_buf[4 ] =BPF_ALU32_IMM (BPF_ADD ,BPF_REG_2 ,-1 ); + insn_buf[5 ] =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_2 ,0 ); +/* s->end = end; */ + insn_buf[6 ] =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_3 ,4 ); +/* return 0; */ + insn_buf[7 ] =BPF_MOV64_IMM (BPF_REG_0 ,0 ); + insn_buf[8 ] =BPF_JMP_A (5 );
kernel/bpf/verifier.c static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux, · line 19758
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* einval: s->cur = s->end = 0; return -EINVAL; */ + insn_buf[9 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + insn_buf[10 ] =BPF_MOV64_IMM (BPF_REG_0 , -EINVAL ); + insn_buf[11 ] =BPF_JMP_A (2 ); +/* e2big: s->cur = s->end = 0; return -E2BIG; */ + insn_buf[12 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + insn_buf[13 ] =BPF_MOV64_IMM (BPF_REG_0 , -E2BIG ); + +return 14 ; +}
The translation lgtm, do we want to avoid two checks at the start in case when r2 and r3 are known statically?
PATCH v2 2/4 bpf: Inline bpf_iter_num_next() kfunc
In reply to Puranjay Mohanbpf_iter_num_next() is called on every iteration of a bpf_for() loop and is the hot path of the numeric open-coded iterator. It only advances the on-stack iterator state and returns a pointer to it, so open-coding it in the verifier removes a function call from each loop iteration.
Inline it in bpf_fixup_kfunc_call() by replacing the call with an equivalent instruction sequence. R1 holds the pointer to the on-stack bpf_iter_num; the returned pointer to s->cur is R1 itself since s->cur is the first member.
s->cur and s->end are int, so the kfunc's (s64)(s->cur + 1) >= s->end test is equivalent to a signed 32-bit comparison of (s->cur + 1) against s->end: s->cur + 1 is computed as a 32-bit int in the kfunc as well, and sign-extending both sides of a comparison of two int values does not change its result. The inlined code therefore uses a 32-bit compare and needs no sign extension.
--- kernel/bpf/verifier.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
kernel/bpf/verifier.c +29 -0
@@ -19757,6 +19757,33 @@ static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)return 14 ; } +/* +* Inline bpf_iter_num_next(). R1 holds the pointer to the iterator. Keep in +* sync with the kfunc in kernel/bpf/bpf_iter.c. +*/ +static int inline_bpf_iter_num_next (struct bpf_insn *insn_buf) +{ +/* +* s->cur and s->end are int, so the (s64)(s->cur + 1) >= s->end check +* is equivalent to a signed 32-bit comparison of (s->cur + 1) against +* s->end and needs no sign extension. +*/ + insn_buf[0 ] =BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 ,0 ); + insn_buf[1 ] =BPF_ALU32_IMM (BPF_ADD ,BPF_REG_0 ,1 ); + insn_buf[2 ] =BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 ,4 ); +/* if ((s32)(s->cur + 1) >= (s32)s->end) goto done; */ + insn_buf[3 ] =BPF_JMP32_REG (BPF_JSGE ,BPF_REG_0 ,BPF_REG_2 ,3 ); +/* s->cur++; return &s->cur; */ + insn_buf[4 ] =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_0 ,0 ); + insn_buf[5 ] =BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ); + insn_buf[6 ] =BPF_JMP_A (2 ); +/* done: s->cur = s->end = 0; return NULL; */ + insn_buf[7 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + insn_buf[8 ] =BPF_MOV64_IMM (BPF_REG_0 ,0 ); + +return 9 ; +} +int bpf_fixup_kfunc_call (struct bpf_verifier_env *env,struct bpf_insn *insn,struct bpf_insn *insn_buf,int insn_idx,int *cnt) {
@@ -19888,6 +19915,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, *cnt =6 ; }else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) { *cnt =inline_bpf_iter_num_new (insn_buf); + }else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) { + *cnt =inline_bpf_iter_num_next (insn_buf); }if (env->insn_aux_data[insn_idx].arg_prog) {
Signature · 1 line
2.53.0-Meta
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 2/4] bpf: Inline bpf_iter_num_next() kfunc
On Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:
...
Quoted 12 lines
kernel/bpf/verifier.c +12
-0+static int inline_bpf_iter_num_next (struct bpf_insn *insn_buf) +{ +/* +* s->cur and s->end are int, so the (s64)(s->cur + 1) >= s->end check +* is equivalent to a signed 32-bit comparison of (s->cur + 1) against +* s->end and needs no sign extension. +*/ + insn_buf[0 ] =BPF_LDX_MEM (BPF_W ,BPF_REG_0 ,BPF_REG_1 ,0 ); + insn_buf[1 ] =BPF_ALU32_IMM (BPF_ADD ,BPF_REG_0 ,1 ); + insn_buf[2 ] =BPF_LDX_MEM (BPF_W ,BPF_REG_2 ,BPF_REG_1 ,4 ); +/* if ((s32)(s->cur + 1) >= (s32)s->end) goto done; */ + insn_buf[3 ] =BPF_JMP32_REG (BPF_JSGE ,BPF_REG_0 ,BPF_REG_2 ,3 );
kernel/bpf/verifier.c static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf) · line 19785
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* s->cur++; return &s->cur; */ + insn_buf[4 ] =BPF_STX_MEM (BPF_W ,BPF_REG_1 ,BPF_REG_0 ,0 ); + insn_buf[5 ] =BPF_MOV64_REG (BPF_REG_0 ,BPF_REG_1 ); + insn_buf[6 ] =BPF_JMP_A (2 ); +/* done: s->cur = s->end = 0; return NULL; */ + insn_buf[7 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + insn_buf[8 ] =BPF_MOV64_IMM (BPF_REG_0 ,0 ); + +return 9 ; +}
Nit: let's have this (and others) as:
int i = 0;
insn_buf[i++] = ...
insn_buf[i++] = ...
insn_buf[i++] = ...
return i;
The translation lgtm.
...
PATCH v2 3/4 bpf: Inline bpf_iter_num_destroy() kfunc
In reply to Puranjay Mohanbpf_iter_num_destroy() only zeroes the on-stack iterator state, which is a single 8-byte store. Open-code it in the verifier to drop the call emitted at the end of every bpf_for() loop.
Inline it in bpf_fixup_kfunc_call() by replacing the call with a store of zero to the iterator, matching the kfunc which clears both s->cur and s->end. R1 holds the pointer to the on-stack bpf_iter_num.
--- kernel/bpf/verifier.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
kernel/bpf/verifier.c +14 -0
@@ -19784,6 +19784,18 @@ static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)return 9 ; } +/* +* Inline bpf_iter_num_destroy(). R1 holds the pointer to the iterator. Keep in +* sync with the kfunc in kernel/bpf/bpf_iter.c. +*/ +static int inline_bpf_iter_num_destroy (struct bpf_insn *insn_buf) +{ +/* s->cur = s->end = 0; */ + insn_buf[0 ] =BPF_ST_MEM (BPF_DW ,BPF_REG_1 ,0 ,0 ); + +return 1 ; +} +int bpf_fixup_kfunc_call (struct bpf_verifier_env *env,struct bpf_insn *insn,struct bpf_insn *insn_buf,int insn_idx,int *cnt) {
@@ -19917,6 +19929,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, *cnt =inline_bpf_iter_num_new (insn_buf); }else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) { *cnt =inline_bpf_iter_num_next (insn_buf); + }else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_destroy]) { + *cnt =inline_bpf_iter_num_destroy (insn_buf); }if (env->insn_aux_data[insn_idx].arg_prog) {
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next v2 3/4] bpf: Inline bpf_iter_num_destroy() kfunc
Quoted 10 lines · Puranjay Mohan
bpf_iter_num_destroy()only zeroes the on-stack iterator state, which is a single 8-byte store. Open-code it in the verifier to drop the call emitted at the end of everybpf_for()loop.Inline it in
bpf_fixup_kfunc_call()by replacing the call with a store of zero to the iterator, matching the kfunc which clears boths->curands->end. R1 holds the pointer to the on-stackbpf_iter_num.Signed-off-by Puranjay Mohan <[email protected]>---
PATCH v2 4/4 selftests/bpf: Add bpf_for() benchmark
In reply to Puranjay MohanThe bpf_for() macro builds on the numeric open-coded iterator kfuncs bpf_iter_num_{new,next,destroy}(). Add a benchmark, modelled on the existing bpf_loop benchmark, that repeatedly runs a bpf_for() loop so the per-iteration cost of the iterator can be measured, e.g. to quantify the effect of inlining the iterator kfuncs in the verifier.
The BPF program runs an inner bpf_for(i, 0, nr_loops) loop with an empty body 1000 times per trigger and accounts nr_loops hits per outer iteration, mirroring bench_bpf_loop so the two are directly comparable.
nr_loops defaults to 1000 so that the per-iteration bpf_iter_num_next() cost, rather than the one-time bpf_iter_num_new()/destroy() setup and teardown, dominates the reported numbers:
$ ./bench -p 1 --nr_loops 1000 bpf-for
--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 4 + .../selftests/bpf/benchs/bench_bpf_for.c | 105 ++++++++++++++++++ .../selftests/bpf/benchs/run_bench_bpf_for.sh | 15 +++ .../selftests/bpf/progs/bpf_for_bench.c | 32 ++++++ 5 files changed, 158 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c
tools/testing/selftests/bpf/Makefile +2 -0
@@ -967,6 +967,7 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \ $(OUTPUT)/perfbuf_bench.skel.h $(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h $(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h +$(OUTPUT)/bench_bpf_for.o: $(OUTPUT)/bpf_for_bench.skel.h $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h
@@ -992,6 +993,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_ringbufs.o \ $(OUTPUT)/bench_bloom_filter_map.o \ $(OUTPUT)/bench_bpf_loop.o \ + $(OUTPUT)/bench_bpf_for.o \ $(OUTPUT)/bench_strncmp.o \ $(OUTPUT)/bench_bpf_hashmap_full_update.o \ $(OUTPUT)/bench_local_storage.o \
tools/testing/selftests/bpf/bench.c +4 -0
@@ -276,6 +276,7 @@ static const struct argp_option opts[] = {extern struct argp bench_ringbufs_argp;extern struct argp bench_bloom_map_argp;extern struct argp bench_bpf_loop_argp; +extern struct argp bench_bpf_for_argp;extern struct argp bench_local_storage_argp;extern struct argp bench_local_storage_rcu_tasks_trace_argp;extern struct argp bench_strncmp_argp;
@@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 }, { &bench_bloom_map_argp, 0, "Bloom filter map benchmark", 0 }, { &bench_bpf_loop_argp, 0, "bpf_loop helper benchmark", 0 }, + { &bench_bpf_for_argp, 0, "bpf_for loop benchmark", 0 }, { &bench_local_storage_argp, 0, "local_storage benchmark", 0 }, { &bench_strncmp_argp, 0, "bpf_strncmp helper benchmark", 0 }, { &bench_local_storage_rcu_tasks_trace_argp, 0,
@@ -557,6 +559,7 @@ extern const struct bench bench_bloom_false_positive;extern const struct bench bench_hashmap_without_bloom;extern const struct bench bench_hashmap_with_bloom;extern const struct bench bench_bpf_loop; +extern const struct bench bench_bpf_for;extern const struct bench bench_strncmp_no_helper;extern const struct bench bench_strncmp_helper;extern const struct bench bench_bpf_hashmap_full_update;
@@ -640,6 +643,7 @@ static const struct bench *benchs[] = { &bench_hashmap_without_bloom, &bench_hashmap_with_bloom, &bench_bpf_loop, + &bench_bpf_for, &bench_strncmp_no_helper, &bench_strncmp_helper, &bench_bpf_hashmap_full_update,
tools/testing/selftests/bpf/benchs/bench_bpf_for.c new file +105 -0
105 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh new file +15 -0
@@ -0,0 +1,15 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +for t in 1 4 8 12 16; do +for i in 10 100 500 1000 5000 10000 50000 100000 500000 1000000; do +subtitle "nr_loops: $i, nr_threads: $t" + summarize_ops "bpf_for: " \ + "$($RUN_BENCH -p $t --nr_loops $i bpf-for)" + printf "\n" +done +done
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +32 -0
@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +u32 nr_loops; +long hits; + +static int outer_loop (__u32 index,void *data) +{ +int i; + +/* +* Empty body: the work being measured is the open-coded numeric +* iterator itself (bpf_iter_num_new/next/destroy behind bpf_for()). +*/ +bpf_for (i,0 , nr_loops) + ; +__sync_add_and_fetch (&hits, nr_loops); +return 0 ; +} + +SEC("fentry/" SYS_PREFIX "sys_getpgid") +int benchmark (void *ctx) +{ +bpf_loop (1000 , outer_loop,NULL ,0 ); +return 0 ; +}
Signature · 1 line
2.53.0-Meta
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 4/4] selftests/bpf: Add bpf_for() benchmark
Quoted 10 lines · Puranjay Mohan
The
bpf_for()macro builds on the numeric open-coded iterator kfuncs bpf_iter_num_{new,next,destroy}(). Add a benchmark, modelled on the existingbpf_loopbenchmark, that repeatedly runs abpf_for()loop so the per-iteration cost of the iterator can be measured, e.g. to quantify the effect of inlining the iterator kfuncs in the verifier.The BPF program runs an inner
bpf_for(i, 0, nr_loops)loop with an empty body 1000 times per trigger and accountsnr_loopshits per outer iteration, mirroringbench_bpf_loopso the two are directly comparable.
Quoted 8 lines
nr_loopsdefaults to 1000 so that the per-iterationbpf_iter_num_next()cost, rather than the one-time bpf_iter_num_new()/destroy() setup and teardown, dominates the reported numbers:$ ./bench -p 1 --nr_loops 1000 bpf-forSigned-off-by Puranjay Mohan <[email protected]>---
Could you please add a few tests with __xlated tag, e.g. as in:
tools/testing/selftests/bpf/progs/epilogue_exit.c
To verify the shape of the inlined code?
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 4/4] selftests/bpf: Add bpf_for() benchmark
In reply to Puranjay MohanOn Fri, 2026-07-17 at 05:02 -0700, Puranjay Mohan wrote:
...
Quoted 18 lines
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +12
-0@@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +char _license[] SEC("license") = "GPL"; + +u32 nr_loops; +long hits; +
tools/testing/selftests/bpf/progs/bpf_for_bench.c line 22
Quoted 10 lines
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +10
-0+static int outer_loop (__u32 index,void *data) +{ +int i; + +/* +* Empty body: the work being measured is the open-coded numeric +* iterator itself (bpf_iter_num_new/next/destroy behind bpf_for()). +*/ +bpf_for (i,0 , nr_loops) + ;
I wonder how does this compare with regular for loop that also reads it's counter.
Quoted 10 lines
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +10
-0+__sync_add_and_fetch (&hits, nr_loops); +return 0 ; +} + +SEC("fentry/" SYS_PREFIX "sys_getpgid") +int benchmark (void *ctx) +{ +bpf_loop (1000 , outer_loop,NULL ,0 ); +return 0 ; +}