Skip to contentSkip to conversation

bpf: Inline the numeric open-coded iterator kfuncs

PATCH Superseded Puranjay Mohan 4 participants 14 messages Jul 29 – Jul 30

Subsystems (5)
Kselftest BPF BPF selftests BPF core +1 more subsystems

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover bpf: Inline the numeric open-coded iterator kfuncs

The bpf_for(i, start, end) macro is BPF's open-coded numeric iterator. It expands into calls to three kfuncs: bpf_iter_num_new() to set the iterator up, bpf_iter_num_next() once per iteration, and bpf_iter_num_destroy() to tear it down. The verifier emits these as ordinary kfunc calls, so a bpf_for() loop pays function-call overhead on setup, teardown, and -- most importantly -- on every single iteration via bpf_iter_num_next().

All three kfuncs are tiny and only touch the 8-byte on-stack iterator state (struct bpf_iter_num_kern { int cur; int end; }). That makes them good candidates for inlining, the same way several other special kfuncs are already open-coded in bpf_fixup_kfunc_call(). This series replaces each of the three calls with an equivalent inline BPF instruction sequence:

- bpf_iter_num_new(): the end - start range check is done with 32-bit arithmetic (start <= end is checked first, so the distance fits in a u32) and range-checked against BPF_MAX_LOOPS as unsigned. This avoids the cpuv4 sign-extension insns that some JITs do not implement. Returns the same -EINVAL / -E2BIG / 0 as the kfunc.

- bpf_iter_num_next(): the hot path. cur and end are int, so the kfunc's s->cur + 1 >= s->end test is an ordinary signed 32-bit compare and the inlined code needs no sign extension.

- bpf_iter_num_destroy(): the stack slot is no longer tracked as iterator state once destroy() returns, so nothing needs to be written to it. Both the kfunc and the inlined form become a no-op, which just drops the call.

The emitted instructions are plain BPF and remain valid for the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.

Benchmark (./bench -p 1 --nr_loops 1000000 {bpf-loop,bpf-for}):

    +--------+---------------------+---------------------+---------------------+
    |  arch  |       bpf_loop      | bpf_for non-inlined |   bpf_for inlined   |
    +--------+---------------------+---------------------+---------------------+
    | x86-64 |  4252 M/s (0.24 ns) |  3608 M/s (0.28 ns) |  7417 M/s (0.13 ns) |
    +--------+---------------------+---------------------+---------------------+
    | arm64  |  649 M/s (1.54 ns)  |  548 M/s (1.82 ns)  |  546 M/s (1.83 ns)  |
    +--------+---------------------+---------------------+---------------------+

On x86-64 removing the per-iteration call roughly doubles bpf_for() throughput. On arm64 it is neutral, and rather than guess why this was checked with perf: inlining removes ~28% of the executed instructions (the call) but leaves the cycle count unchanged -- IPC drops from ~4.2 to ~3.0 and backend stalls rise from ~50% to ~66%. The loop is bound by the latency of the iterator's on-stack counter, not by call overhead:
bpf_iter_num_next() loads s->cur from the stack, increments it and stores it back each iteration, and the next iteration's load depends on that store. The removed call instructions were executing in the shadow of that store->load stall and were never on the critical path.

A small userspace microbenchmark isolates the effect: a same-address store->load->add round-trip (the shape of the on-stack counter) costs ~6 cycles/iteration on the tested arm64 core but ~1 cycle on x86-64, where the core collapses the same-address round-trip into a register move (memory renaming / store-to-load-forwarding elimination). So on x86-64 the loop is not latency-bound and the per-iteration call dominates -- removing it is the ~2x win -- whereas on arm64 the call fits entirely inside the store->load stall the loop already has, so adding or removing it changes nothing.

bpf_loop() is shown for reference only; it is a different construct (a callback invoked per iteration) and this series does not change it. Its counter lives in a register rather than on the stack, so on arm64 it avoids the store->load latency above and is faster than bpf_for() there.

Changelog:
v3: https://lore.kernel.org/all/[email protected]/
Changes in v4:
- Drop the "elide range checks for constant bounds" patch (Andrii Nakryiko)
- bpf_iter_num_new(): range-check the distance against BPF_MAX_LOOPS with an unsigned compare (Andrii Nakryiko)
- bpf_iter_num_destroy(): make it a no-op in both the kfunc and the inlined form instead of zeroing the iterator state (Andrii Nakryiko)
- New patch: fix the misleading overflow comment in bpf_iter_num_next() and drop the redundant (s64) cast; the int wraparound is intentional and load-bearing (Andrii Nakryiko)
- bpf_for benchmark: nr_loops is int, matching what bpf_for() expects (Andrii Nakryiko)
- Corroborate the arm64/x86 benchmark difference with perf counters and a store-to-load-forwarding microbenchmark (Kumar Kartikeya Dwivedi, Andrii Nakryiko)

v2: https://lore.kernel.org/bpf/[email protected]/
Changes in v3:
- Elide the range checks in bpf_iter_num_new() when start and end are constant, marking the registers precise so paths reaching the call with different constants are not pruned (Eduard Zingerman)
- Add __xlated selftests pinning the inlined new()/next()/destroy() shapes (Eduard Zingerman)
- Use the insn_buf[i++] idiom in the inline helpers (Eduard Zingerman)
- Pick up Acked-by on patch 3

v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Don't emit sign-extending (movsx) moves; some JITs (e.g. x86-32, mips32, sparc64) decode them as a plain move and would miscompile the range check

Puranjay Mohan (6):
bpf: Correct the overflow check comment in bpf_iter_num_next()
bpf: Inline bpf_iter_num_new() kfunc
bpf: Inline bpf_iter_num_next() kfunc
bpf: Inline bpf_iter_num_destroy() as a no-op selftests/bpf: Verify inlined numeric iterator shape with __xlated selftests/bpf: Add bpf_for() benchmark

 kernel/bpf/bpf_iter.c                         |  20 ++--
 kernel/bpf/verifier.c                         |  83 ++++++++++++++
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 104 ++++++++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 +++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++++
 tools/testing/selftests/bpf/progs/iters.c     |  83 ++++++++++++++
 8 files changed, 335 insertions(+), 8 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c

base-commit: fdec474c65fd35d5a6e1497ed50a9f98c07192f0

Signature · 1 line
2.53.0-Meta
#1 lore raw

PATCH v4 1/6 bpf: Correct the overflow check comment in bpf_iter_num_next()

bpf_iter_num_next() decides whether the iterator is exhausted with:

	if ((s64)(s->cur + 1) >= s->end) {

The comment above it claimed the (s64) cast was needed to be careful about overflow, "e.g., if s->cur == s->end == INT_MAX, we can't just do s->cur + 1 >= s->end". That reasoning is wrong: s->cur + 1 is evaluated in int and wraps *before* the cast, so casting the already-wrapped result to s64 changes nothing. For s->cur == s->end == INT_MAX the plain s->cur + 1 >= s->end and the (s64) version both evaluate to false, and more generally the two are identical for all inputs (sign-extending both operands of a signed compare never changes its result).

The wraparound of s->cur + 1 is in fact intentional and load-bearing:
bpf_iter_num_new() initializes s->cur to start - 1, which wraps to INT_MAX when start == INT_MIN, and the wrapping s->cur + 1 recovers start on the first call. Using real 64-bit arithmetic ((s64)s->cur + 1) would instead break iterators starting at INT_MIN.

Drop the redundant cast and rewrite the comment to describe what actually happens. No functional change; the kernel builds with -fno-strict-overflow so the signed wraparound is well defined.

---
 kernel/bpf/bpf_iter.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/bpf/bpf_iter.c +7 -5
index f5eaeb2493d4a..f190f2b250048 100644
@@ -802,12 +802,14 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 {
 	struct bpf_iter_num_kern *s = (void *)it;
 
-	/* check failed initialization or if we are done (same behavior);
-	 * need to be careful about overflow, so convert to s64 for checks,
-	 * e.g., if s->cur == s->end == INT_MAX, we can't just do
-	 * s->cur + 1 >= s->end
+	/* Detect the end of the range, or a failed/empty iterator: all of these
+	 * leave s->cur + 1 >= s->end. bpf_iter_num_new() set s->cur to start - 1
+	 * (which wraps to INT_MAX when start == INT_MIN), so the s->cur + 1 below
+	 * is a deliberate 32-bit wraparound that recovers start. As s->cur and
+	 * s->end are int, this is an ordinary signed 32-bit compare, exactly what
+	 * the inlined bpf_iter_num_next() emits.
 	 */
-	if ((s64)(s->cur + 1) >= s->end) {
+	if (s->cur + 1 >= s->end) {
 		s->cur = s->end = 0;
 		return NULL;
 	}
Signature · 1 line
2.53.0-Meta
#2 lore raw

Re: [PATCH bpf-next v4 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The newly added multi-line comment in `bpf_iter_num_next()` does not follow the required BPF…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The newly added multi-line comment in bpf_iter_num_next() does not follow the required BPF subsystem comment style. --

--- Patch [1]: [PATCH bpf-next v4 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next() --- commit 385d10f67ceb46ec31888afb23333c5f7645c9f6
Author: Puranjay Mohan <[email protected]>

bpf: Correct the overflow check comment in bpf_iter_num_next()

This commit removes a redundant 64-bit cast in the bpf_iter_num_next() iterator loop and rewrites the associated comment to correctly describe that the 32-bit wraparound is intentional for handling starts at INT_MIN.

kernel/bpf/bpf_iter.c __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) · line 805

Quoted 13 lines
kernel/bpf/bpf_iter.c +1 -4
index f5eaeb2493d4a..f190f2b250048 100644
@@ -802,12 +802,14 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 {
 	struct bpf_iter_num_kern *s = (void *)it;
 
-	/* check failed initialization or if we are done (same behavior);
-	 * need to be careful about overflow, so convert to s64 for checks,
-	 * e.g., if s->cur == s->end == INT_MAX, we can't just do
-	 * s->cur + 1 >= s->end
+	/* Detect the end of the range, or a failed/empty iterator: all of these

[Severity: Low]
This isn't a bug, but does this multi-line comment follow the preferred BPF subsystem style?

The BPF guidelines specify that multi-line comments should have the opening /* on its own line, with the text starting on the next line. Since this comment block is being rewritten entirely, would it be better to update it to match the modern multi-line comment format?

Quoted 11 lines
kernel/bpf/bpf_iter.c +6 -1
+	 * leave s->cur + 1 >= s->end. bpf_iter_num_new() set s->cur to start - 1
+	 * (which wraps to INT_MAX when start == INT_MIN), so the s->cur + 1 below
+	 * is a deliberate 32-bit wraparound that recovers start. As s->cur and
+	 * s->end are int, this is an ordinary signed 32-bit compare, exactly what
+	 * the inlined bpf_iter_num_next() emits.
 	 */
-	if ((s64)(s->cur + 1) >= s->end) {
+	if (s->cur + 1 >= s->end) {
 		s->cur = s->end = 0;
 		return NULL;
 	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [PATCH bpf-next v4 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()

In reply to Puranjay Mohan
Does this comment reference code that does not exist yet? Grepping the tree at de0af07e8966 for 'inline_bpf_iter_num' returns nothing, and kernel/bpf/verifier.c only references bpf_iter_num_next via…
Quoted 8 lines
kernel/bpf/bpf_iter.c +0 -0
index f5eaeb2493d4..f190f2b25004 100644
@@ -802,12 +802,14 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 {
 	struct bpf_iter_num_kern *s = (void *)it;
 

kernel/bpf/bpf_iter.c __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) · line 810

Quoted 10 lines
kernel/bpf/bpf_iter.c +6 -4
-	/* check failed initialization or if we are done (same behavior);
-	 * need to be careful about overflow, so convert to s64 for checks,
-	 * e.g., if s->cur == s->end == INT_MAX, we can't just do
-	 * s->cur + 1 >= s->end
+	/* Detect the end of the range, or a failed/empty iterator: all of these
+	 * leave s->cur + 1 >= s->end. bpf_iter_num_new() set s->cur to start - 1
+	 * (which wraps to INT_MAX when start == INT_MIN), so the s->cur + 1 below
+	 * is a deliberate 32-bit wraparound that recovers start. As s->cur and
+	 * s->end are int, this is an ordinary signed 32-bit compare, exactly what
+	 * the inlined bpf_iter_num_next() emits.
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Does this comment reference code that does not exist yet? Grepping the tree at de0af07e8966 for 'inline_bpf_iter_num' returns nothing, and kernel/bpf/verifier.c only references bpf_iter_num_next via BTF_ID and is_bpf_iter_num_api_kfunc - bookkeeping for the KF_ITER_NEXT kfunc, not an open-coded instruction sequence.

The inlining is added three commits later in this series:
4bb5d9c0aa96 ("bpf: Inline bpf_iter_num_next() kfunc"), which introduces inline_bpf_iter_num_next() in kernel/bpf/verifier.c. So the comment added by this patch is a forward reference to code that does not exist yet, in a patch whose entire stated purpose is to make the comment "describe what actually happens".

Could the patch either drop that clause, or be reordered after 4bb5d9c0aa96 so the reference resolves?

kernel/bpf/bpf_iter.c __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) · line 815

Quoted 6 lines
kernel/bpf/bpf_iter.c +1 -1
 	 */
-	if ((s64)(s->cur + 1) >= s->end) {
+	if (s->cur + 1 >= s->end) {
 		s->cur = s->end = 0;
 		return NULL;
 	}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/30490674177

#4 lore raw

Re: [PATCH bpf-next v4 1/6] bpf: Correct the overflow check comment in bpf_iter_num_next()

In reply to Puranjay Mohan
Quoted 38 lines · Puranjay Mohan

bpf_iter_num_next() decides whether the iterator is exhausted with:

        if ((s64)(s->cur + 1) >= s->end) {

The comment above it claimed the (s64) cast was needed to be careful about overflow, "e.g., if s->cur == s->end == INT_MAX, we can't just do s->cur + 1 >= s->end". That reasoning is wrong: s->cur + 1 is evaluated in int and wraps *before* the cast, so casting the already-wrapped result to s64 changes nothing. For s->cur == s->end == INT_MAX the plain s->cur + 1 >= s->end and the (s64) version both evaluate to false, and more generally the two are identical for all inputs (sign-extending both operands of a signed compare never changes its result).

The wraparound of s->cur + 1 is in fact intentional and load-bearing:
bpf_iter_num_new() initializes s->cur to start - 1, which wraps to INT_MAX when start == INT_MIN, and the wrapping s->cur + 1 recovers start on the first call. Using real 64-bit arithmetic ((s64)s->cur + 1) would instead break iterators starting at INT_MIN.

Drop the redundant cast and rewrite the comment to describe what actually happens. No functional change; the kernel builds with -fno-strict-overflow so the signed wraparound is well defined.

---
 kernel/bpf/bpf_iter.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
kernel/bpf/bpf_iter.c +0 -1
index f5eaeb2493d4a..f190f2b250048 100644
@@ -802,12 +802,14 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 {
        struct bpf_iter_num_kern *s = (void *)it;
 
-       /* check failed initialization or if we are done (same behavior);

kernel/bpf/bpf_iter.c __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) · line 811

Quoted 10 lines
kernel/bpf/bpf_iter.c +6 -3
-        * need to be careful about overflow, so convert to s64 for checks,
-        * e.g., if s->cur == s->end == INT_MAX, we can't just do
-        * s->cur + 1 >= s->end
+       /* Detect the end of the range, or a failed/empty iterator: all of these
+        * leave s->cur + 1 >= s->end. bpf_iter_num_new() set s->cur to start - 1
+        * (which wraps to INT_MAX when start == INT_MIN), so the s->cur + 1 below
+        * is a deliberate 32-bit wraparound that recovers start. As s->cur and
+        * s->end are int, this is an ordinary signed 32-bit compare, exactly what
+        * the inlined bpf_iter_num_next() emits.
         */

this is really Claude-style verboseness. let's just say that num iter maintains an invariant that either s->cur < s->end or s->cur == s->end == 0, in both cases s->cur + 1 can't overflow int.

Quoted 8 lines
kernel/bpf/bpf_iter.c +1 -2
-       if ((s64)(s->cur + 1) >= s->end) {
+       if (s->cur + 1 >= s->end) {
                s->cur = s->end = 0;
                return NULL;
        }
--

2.53.0-Meta

#5 lore raw

PATCH v4 2/6 bpf: Inline bpf_iter_num_new() kfunc

In reply to Puranjay Mohan

The numeric iterator kfuncs bpf_iter_num_{new,next,destroy}() back the open-coded iterator macro bpf_for() and are emitted as regular kfunc calls by the verifier. bpf_iter_num_new() is small and only touches the on-stack iterator state, so the verifier can open-code it and avoid the call overhead of setting up an iterator.

Inline it in bpf_fixup_kfunc_call() by replacing the call with an equivalent instruction sequence. R1 holds the pointer to the on-stack bpf_iter_num, while R2 and R3 hold the start and end arguments. The arithmetic and the error paths mirror the kfunc exactly, so a program that inspects the return value keeps observing the same -EINVAL / -E2BIG / 0 results.

The kfunc guards against overflow with a (s64)end - (s64)start range check. The start > end case is rejected first, so start <= end at that point and the range end - start fits in a u32. The inlined code computes it with a 32-bit subtraction that zero-extends the distance into a 64-bit register and compares it against BPF_MAX_LOOPS as an unsigned value. Sign-extending the operands with movsx instead would emit a cpuv4 instruction after verification; JITs that do not implement movsx (e.g. x86-32, mips32, sparc64) decode it as a plain move and would miscompile the check.

The emitted instructions are plain BPF and are handled by the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.

---
 kernel/bpf/verifier.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
kernel/bpf/verifier.c +39 -0
index e6f35f4e715b6..7400515ae1296 100644
@@ -19773,6 +19773,43 @@ static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux,
 	*cnt = 4;
 }
 
+/*
+ * Inline bpf_iter_num_new(). R1 holds the pointer to the iterator, R2 and R3 hold the (int)
+ * start and end arguments. Keep in sync with the kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
+{
+	int i = 0;
+
+	/* if (start > end) goto einval; */
+	insn_buf[i++] = BPF_JMP32_REG(BPF_JSGT, BPF_REG_2, BPF_REG_3, 8);
+	/*
+	 * start <= end here, so the range end - start fits in a u32; compute it as a 32-bit
+	 * subtraction that zero-extends into r0 and range-check it as unsigned.
+	 */
+	insn_buf[i++] = BPF_MOV32_REG(BPF_REG_0, BPF_REG_3);
+	insn_buf[i++] = BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_2);
+	/* if (r0 > BPF_MAX_LOOPS) goto e2big; */
+	insn_buf[i++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_0, BPF_MAX_LOOPS, 8);
+	/* s->cur = start - 1; */
+	insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_2, -1);
+	insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, 0);
+	/* s->end = end; */
+	insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, 4);
+	/* return 0; */
+	insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+	insn_buf[i++] = BPF_JMP_A(5);
+	/* einval: s->cur = s->end = 0; return -EINVAL; */
+	insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+	insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -EINVAL);
+	insn_buf[i++] = BPF_JMP_A(2);
+	/* e2big: s->cur = s->end = 0; return -E2BIG; */
+	insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+	insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
+
+	return i;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19902,6 +19939,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		insn_buf[4] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1);
 		insn_buf[5] = BPF_ALU64_IMM(BPF_NEG, BPF_REG_0, 0);
 		*cnt = 6;
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
+		*cnt = inline_bpf_iter_num_new(insn_buf);
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
Signature · 1 line
2.53.0-Meta
#6 lore raw

Re: [PATCH bpf-next v4 2/6] bpf: Inline bpf_iter_num_new() kfunc

Quoted 48 lines · Puranjay Mohan

The numeric iterator kfuncs bpf_iter_num_{new,next,destroy}() back the open-coded iterator macro bpf_for() and are emitted as regular kfunc calls by the verifier. bpf_iter_num_new() is small and only touches the on-stack iterator state, so the verifier can open-code it and avoid the call overhead of setting up an iterator.

Inline it in bpf_fixup_kfunc_call() by replacing the call with an equivalent instruction sequence. R1 holds the pointer to the on-stack bpf_iter_num, while R2 and R3 hold the start and end arguments. The arithmetic and the error paths mirror the kfunc exactly, so a program that inspects the return value keeps observing the same -EINVAL / -E2BIG / 0 results.

The kfunc guards against overflow with a (s64)end - (s64)start range check. The start > end case is rejected first, so start <= end at that point and the range end - start fits in a u32. The inlined code computes it with a 32-bit subtraction that zero-extends the distance into a 64-bit register and compares it against BPF_MAX_LOOPS as an unsigned value. Sign-extending the operands with movsx instead would emit a cpuv4 instruction after verification; JITs that do not implement movsx (e.g. x86-32, mips32, sparc64) decode it as a plain move and would miscompile the check.

The emitted instructions are plain BPF and are handled by the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.

---
 kernel/bpf/verifier.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
kernel/bpf/verifier.c +7 -0
index e6f35f4e715b6..7400515ae1296 100644
@@ -19773,6 +19773,43 @@ static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux,
        *cnt = 4;
 }
 
+/*
+ * Inline bpf_iter_num_new(). R1 holds the pointer to the iterator, R2 and R3 hold the (int)
+ * start and end arguments. Keep in sync with the kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
+{
+       int i = 0;

kernel/bpf/verifier.c static void __fixup_collection_insert_kfunc(struct bpf_insn_aux_data *insn_aux, · line 19792

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+
+       /* if (start > end) goto einval; */
+       insn_buf[i++] = BPF_JMP32_REG(BPF_JSGT, BPF_REG_2, BPF_REG_3, 8);
+       /*
+        * start <= end here, so the range end - start fits in a u32; compute it as a 32-bit
+        * subtraction that zero-extends into r0 and range-check it as unsigned.
+        */
+       insn_buf[i++] = BPF_MOV32_REG(BPF_REG_0, BPF_REG_3);
+       insn_buf[i++] = BPF_ALU32_REG(BPF_SUB, BPF_REG_0, BPF_REG_2);
+       /* if (r0 > BPF_MAX_LOOPS) goto e2big; */

should we update bpf_iter_num_new() implementation to have the same u32-based comparison to keep everything in-sync (and leave *brief* comment on why this is ok)?

/* start <= end, so (u32)(end - start) fits u32 without overflow */ if ((u32)(end - start) > BPF_MAX_LOOPS) -> E2BIG ?

?

Quoted 24 lines
kernel/bpf/verifier.c +20 -0
+       insn_buf[i++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_0, BPF_MAX_LOOPS, 8);
+       /* s->cur = start - 1; */
+       insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_2, -1);
+       insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_2, 0);
+       /* s->end = end; */
+       insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_3, 4);
+       /* return 0; */
+       insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+       insn_buf[i++] = BPF_JMP_A(5);
+       /* einval: s->cur = s->end = 0; return -EINVAL; */
+       insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+       insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -EINVAL);
+       insn_buf[i++] = BPF_JMP_A(2);
+       /* e2big: s->cur = s->end = 0; return -E2BIG; */
+       insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+       insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
+
+       return i;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {

kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19943

Quoted 5 lines
kernel/bpf/verifier.c +2 -0
@@ -19902,6 +19939,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                insn_buf[4] = BPF_ALU64_REG(BPF_SUB, BPF_REG_0, BPF_REG_1);
                insn_buf[5] = BPF_ALU64_IMM(BPF_NEG, BPF_REG_0, 0);
                *cnt = 6;
+       } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
+               *cnt = inline_bpf_iter_num_new(insn_buf);

all other kfunc inlining logic is inlined in bpf_fixup_kfunc_call, let's keep it that way, drop inline_bpf_iter_num_new

pw-bot: cr

Quoted 6 lines
kernel/bpf/verifier.c +0 -1
        }
 
        if (env->insn_aux_data[insn_idx].arg_prog) {
--

2.53.0-Meta

#7 lore raw

PATCH v4 3/6 bpf: Inline bpf_iter_num_next() kfunc

In reply to Puranjay Mohan

bpf_iter_num_next() is called on every iteration of a bpf_for() loop and is the hot path of the numeric open-coded iterator. It only advances the on-stack iterator state and returns a pointer to it, so open-coding it in the verifier removes a function call from each loop iteration.

Inline it in bpf_fixup_kfunc_call() by replacing the call with an equivalent instruction sequence. R1 holds the pointer to the on-stack bpf_iter_num; the returned pointer to s->cur is R1 itself since s->cur is the first member.

s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end test is a signed 32-bit comparison of (s->cur + 1) against s->end, with s->cur + 1 computed as a 32-bit int. Sign-extending both sides of a comparison of two int values does not change its result, so the inlined code uses a 32-bit compare and needs no sign extension.

---
 kernel/bpf/verifier.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
kernel/bpf/verifier.c +30 -0
index 7400515ae1296..3a3d096f3966e 100644
@@ -19810,6 +19810,34 @@ static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
 	return i;
 }
 
+/*
+ * Inline bpf_iter_num_next(). R1 holds the pointer to the iterator. Keep in sync with the
+ * kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
+{
+	int i = 0;
+
+	/*
+	 * s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end check is a signed 32-bit
+	 * comparison of (s->cur + 1) against s->end and needs no sign extension.
+	 */
+	insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0);
+	insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1);
+	insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 4);
+	/* if ((s32)(s->cur + 1) >= (s32)s->end) goto done; */
+	insn_buf[i++] = BPF_JMP32_REG(BPF_JSGE, BPF_REG_0, BPF_REG_2, 3);
+	/* s->cur++; return &s->cur; */
+	insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0);
+	insn_buf[i++] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
+	insn_buf[i++] = BPF_JMP_A(2);
+	/* done: s->cur = s->end = 0; return NULL; */
+	insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+	insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+
+	return i;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19941,6 +19969,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		*cnt = 6;
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
 		*cnt = inline_bpf_iter_num_new(insn_buf);
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
+		*cnt = inline_bpf_iter_num_next(insn_buf);
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
Signature · 1 line
2.53.0-Meta
#8 lore raw

Re: [PATCH bpf-next v4 3/6] bpf: Inline bpf_iter_num_next() kfunc

Quoted 32 lines · Puranjay Mohan

bpf_iter_num_next() is called on every iteration of a bpf_for() loop and is the hot path of the numeric open-coded iterator. It only advances the on-stack iterator state and returns a pointer to it, so open-coding it in the verifier removes a function call from each loop iteration.

Inline it in bpf_fixup_kfunc_call() by replacing the call with an equivalent instruction sequence. R1 holds the pointer to the on-stack bpf_iter_num; the returned pointer to s->cur is R1 itself since s->cur is the first member.

s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end test is a signed 32-bit comparison of (s->cur + 1) against s->end, with s->cur + 1 computed as a 32-bit int. Sign-extending both sides of a comparison of two int values does not change its result, so the inlined code uses a 32-bit compare and needs no sign extension.

---
 kernel/bpf/verifier.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
kernel/bpf/verifier.c +2 -0
index 7400515ae1296..3a3d096f3966e 100644
@@ -19810,6 +19810,34 @@ static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf)
        return i;
 }
 
+/*
+ * Inline bpf_iter_num_next(). R1 holds the pointer to the iterator. Keep in sync with the

kernel/bpf/verifier.c static int inline_bpf_iter_num_new(struct bpf_insn *insn_buf) · line 19824

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+ * kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
+{
+       int i = 0;
+
+       /*
+        * s->cur and s->end are int, so the kfunc's s->cur + 1 >= s->end check is a signed 32-bit
+        * comparison of (s->cur + 1) against s->end and needs no sign extension.
+        */

let's keep all the explanations in C code implementation, and here just have brief C code we are implementing using BPF instructions

Quoted 31 lines
kernel/bpf/verifier.c +18 -1
+       insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1, 0);
+       insn_buf[i++] = BPF_ALU32_IMM(BPF_ADD, BPF_REG_0, 1);
+       insn_buf[i++] = BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 4);
+       /* if ((s32)(s->cur + 1) >= (s32)s->end) goto done; */
+       insn_buf[i++] = BPF_JMP32_REG(BPF_JSGE, BPF_REG_0, BPF_REG_2, 3);
+       /* s->cur++; return &s->cur; */
+       insn_buf[i++] = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0, 0);
+       insn_buf[i++] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
+       insn_buf[i++] = BPF_JMP_A(2);
+       /* done: s->cur = s->end = 0; return NULL; */
+       insn_buf[i++] = BPF_ST_MEM(BPF_DW, BPF_REG_1, 0, 0);
+       insn_buf[i++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+
+       return i;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19941,6 +19969,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                *cnt = 6;
        } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_new]) {
                *cnt = inline_bpf_iter_num_new(insn_buf);
+       } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
+               *cnt = inline_bpf_iter_num_next(insn_buf);
        }
 
        if (env->insn_aux_data[insn_idx].arg_prog) {
--

2.53.0-Meta

#9 lore raw

PATCH v4 4/6 bpf: Inline bpf_iter_num_destroy() as a no-op

In reply to Puranjay Mohan

bpf_iter_num_destroy() ends every bpf_for() loop. The kfunc only zeroed the 8-byte on-stack iterator state, but once destroy() returns that slot is no longer tracked as iterator state, so nothing reads it and the zeroing is dead work.

Drop it on both sides: make the bpf_iter_num_destroy() kfunc a no-op, and inline the call in bpf_fixup_kfunc_call() to a single BPF_JA 0 (nop) so the per-loop call is removed without emitting any store. The nop is elided by the JITs.

The emitted instruction is plain BPF and is handled by the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.

---
 kernel/bpf/bpf_iter.c |  8 +++++---
 kernel/bpf/verifier.c | 14 ++++++++++++++
 2 files changed, 19 insertions(+), 3 deletions(-)
kernel/bpf/bpf_iter.c +5 -3
index f190f2b250048..ff2055399f3c0 100644
@@ -821,9 +821,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 
 __bpf_kfunc void bpf_iter_num_destroy(struct bpf_iter_num *it)
 {
-	struct bpf_iter_num_kern *s = (void *)it;
-
-	s->cur = s->end = 0;
+	/*
+	 * Nothing to do: the stack slot backing the iterator is no longer
+	 * tracked as iterator state once destroy() returns, so its contents do
+	 * not matter. The verifier inlines this call away entirely.
+	 */
 }
 
 __bpf_kfunc_end_defs();
kernel/bpf/verifier.c +14 -0
index 3a3d096f3966e..36f3e80d2f0b9 100644
@@ -19838,6 +19838,18 @@ static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
 	return i;
 }
 
+/*
+ * Inline bpf_iter_num_destroy(). The stack slot is no longer tracked as iterator state after
+ * destroy(), so nothing has to be done to it; emit a nop just to drop the call. Keep in sync
+ * with the kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_destroy(struct bpf_insn *insn_buf)
+{
+	insn_buf[0] = BPF_JMP_A(0);
+
+	return 1;
+}
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19971,6 +19983,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		*cnt = inline_bpf_iter_num_new(insn_buf);
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
 		*cnt = inline_bpf_iter_num_next(insn_buf);
+	} else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_destroy]) {
+		*cnt = inline_bpf_iter_num_destroy(insn_buf);
 	}
 
 	if (env->insn_aux_data[insn_idx].arg_prog) {
Signature · 1 line
2.53.0-Meta
#10 lore raw

Re: [PATCH bpf-next v4 4/6] bpf: Inline bpf_iter_num_destroy() as a no-op

Quoted 28 lines · Puranjay Mohan

bpf_iter_num_destroy() ends every bpf_for() loop. The kfunc only zeroed the 8-byte on-stack iterator state, but once destroy() returns that slot is no longer tracked as iterator state, so nothing reads it and the zeroing is dead work.

Drop it on both sides: make the bpf_iter_num_destroy() kfunc a no-op, and inline the call in bpf_fixup_kfunc_call() to a single BPF_JA 0 (nop) so the per-loop call is removed without emitting any store. The nop is elided by the JITs.

The emitted instruction is plain BPF and is handled by the interpreter, so interpreter fallback stays correct and no jit_required marking is needed.

---
 kernel/bpf/bpf_iter.c |  8 +++++---
 kernel/bpf/verifier.c | 14 ++++++++++++++
 2 files changed, 19 insertions(+), 3 deletions(-)
kernel/bpf/bpf_iter.c +0 -0
index f190f2b250048..ff2055399f3c0 100644
@@ -821,9 +821,11 @@ __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it)
 

kernel/bpf/bpf_iter.c __bpf_kfunc int *bpf_iter_num_next(struct bpf_iter_num* it) · line 828

Quoted 10 lines
kernel/bpf/bpf_iter.c +5 -3
 __bpf_kfunc void bpf_iter_num_destroy(struct bpf_iter_num *it)
 {
-       struct bpf_iter_num_kern *s = (void *)it;
-
-       s->cur = s->end = 0;
+       /*
+        * Nothing to do: the stack slot backing the iterator is no longer
+        * tracked as iterator state once destroy() returns, so its contents do
+        * not matter. The verifier inlines this call away entirely.
+        */

verbose, drop the comment or have /* no-op */

Quoted 12 lines
kernel/bpf/bpf_iter.c +0 -0
 }
 
 __bpf_kfunc_end_defs();
kernel/bpf/verifier.c +1 -0
index 3a3d096f3966e..36f3e80d2f0b9 100644
@@ -19838,6 +19838,18 @@ static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf)
        return i;
 }
 
+/*

kernel/bpf/verifier.c static int inline_bpf_iter_num_next(struct bpf_insn *insn_buf) · line 19851

Quoted 10 lines
kernel/bpf/verifier.c +10 -0
+ * Inline bpf_iter_num_destroy(). The stack slot is no longer tracked as iterator state after
+ * destroy(), so nothing has to be done to it; emit a nop just to drop the call. Keep in sync
+ * with the kfunc in kernel/bpf/bpf_iter.c.
+ */
+static int inline_bpf_iter_num_destroy(struct bpf_insn *insn_buf)
+{
+       insn_buf[0] = BPF_JMP_A(0);
+
+       return 1;
+}

again, inline the inlining logic

also, bpf_patch_insn_single doesn't support deleting an instruction, we have to have BPF_JMP_A?

Quoted 16 lines
kernel/bpf/verifier.c +3 -1
+
 int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                     struct bpf_insn *insn_buf, int insn_idx, int *cnt)
 {
@@ -19971,6 +19983,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
                *cnt = inline_bpf_iter_num_new(insn_buf);
        } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_next]) {
                *cnt = inline_bpf_iter_num_next(insn_buf);
+       } else if (desc->func_id == special_kfunc_list[KF_bpf_iter_num_destroy]) {
+               *cnt = inline_bpf_iter_num_destroy(insn_buf);
        }
 
        if (env->insn_aux_data[insn_idx].arg_prog) {
--

2.53.0-Meta

#11 lore raw

PATCH v4 5/6 selftests/bpf: Verify inlined numeric iterator shape with __xlated

In reply to Puranjay Mohan

Add an __xlated test to the open-coded iterator suite that pins the shape of the inlined bpf_iter_num_{new,next,destroy}() rewrites. The program is a naked function, so there is no compiler-generated glue and the whole inlined program is matched instruction for instruction: bpf_iter_num_new() emits the range check (distance computation and both the -EINVAL and -E2BIG paths), and bpf_iter_num_next()/destroy() are inlined too.

The tests are pinned to x86_64 and arm64 (bpf_jit_needs_zext() == false):
on arches that need explicit zero-extension the verifier interleaves "wN = wN" insns into the inlined body, which would not match the fixed instruction sequence. The inlining itself is arch independent, so checking it on these arches is sufficient.

---
 tools/testing/selftests/bpf/progs/iters.c | 83 +++++++++++++++++++++++
 1 file changed, 83 insertions(+)
tools/testing/selftests/bpf/progs/iters.c +83 -0
index 0fa70b133d932..62d7df9e80beb 100644
@@ -88,6 +88,89 @@ int iter_err_unsafe_asm_loop(const void *ctx)
 	return 0;
 }
 
+/*
+ * Naked function, so there is no compiler-generated glue and the whole inlined program can be
+ * matched. Pinned to arches whose JITs zero-extend 32-bit writes implicitly
+ * (bpf_jit_needs_zext() == false); on arches that need explicit zero-extension the verifier
+ * interleaves "wN = wN" insns and the fixed shape below would not match. The inlining itself is
+ * arch independent, so checking it on these arches is sufficient.
+ *
+ * bpf_iter_num_new() emits the full range check (distance computation and both the -EINVAL and
+ * -E2BIG error paths); bpf_iter_num_next() and bpf_iter_num_destroy() are inlined too.
+ */
+SEC("raw_tp")
+__arch_x86_64
+__arch_arm64
+__success
+__xlated("r6 = r10")
+__xlated("r6 += -8")
+__xlated("call unknown")
+__xlated("r3 = r0")
+__xlated("r3 &= 65535")
+__xlated("r1 = r6")
+__xlated("r2 = 0")
+/* bpf_iter_num_new(&it, 0, <non-const>) with the range check kept */
+__xlated("if w2 s> w3 goto pc+8")
+__xlated("w0 = w3")
+__xlated("w0 -= w2")
+__xlated("if r0 > 0x800000 goto pc+8")
+__xlated("w2 += -1")
+__xlated("*(u32 *)(r1 +0) = r2")
+__xlated("*(u32 *)(r1 +4) = r3")
+__xlated("r0 = 0")
+__xlated("goto pc+5")
+__xlated("*(u64 *)(r1 +0) = 0")
+__xlated("r0 = -22")
+__xlated("goto pc+2")
+__xlated("*(u64 *)(r1 +0) = 0")
+__xlated("r0 = -7")
+__xlated("r1 = r6")
+/* bpf_iter_num_next(&it) */
+__xlated("r0 = *(u32 *)(r1 +0)")
+__xlated("w0 += 1")
+__xlated("r2 = *(u32 *)(r1 +4)")
+__xlated("if w0 s>= w2 goto pc+3")
+__xlated("*(u32 *)(r1 +0) = r0")
+__xlated("r0 = r1")
+__xlated("goto pc+2")
+__xlated("*(u64 *)(r1 +0) = 0")
+__xlated("r0 = 0")
+__xlated("if r0 != 0x0 goto pc-11")
+__xlated("r1 = r6")
+/* bpf_iter_num_destroy(&it) is inlined to a nop */
+__xlated("goto pc+0")
+__xlated("r0 = 0")
+__xlated("exit")
+int __naked iter_num_new_inlined(void)
+{
+	asm volatile (
+		/* r6 points to struct bpf_iter_num on the stack */
+		"r6 = r10;"
+		"r6 += -8;"
+		/* non-constant end so the range checks are kept */
+		"call %[bpf_get_prandom_u32];"
+		"r3 = r0;"
+		"r3 &= 0xffff;"
+		"r1 = r6;"
+		"r2 = 0;"
+		"call %[bpf_iter_num_new];"
+	"1:"
+		"r1 = r6;"
+		"call %[bpf_iter_num_next];"
+		"if r0 != 0 goto 1b;"
+		"r1 = r6;"
+		"call %[bpf_iter_num_destroy];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(bpf_get_prandom_u32),
+		  __imm(bpf_iter_num_new),
+		  __imm(bpf_iter_num_next),
+		  __imm(bpf_iter_num_destroy)
+		: __clobber_common, "r6"
+	);
+}
+
 SEC("raw_tp")
 __success
 int iter_while_loop(const void *ctx)
Signature · 1 line
2.53.0-Meta
#12 lore raw

PATCH v4 6/6 selftests/bpf: Add bpf_for() benchmark

In reply to Puranjay Mohan

The bpf_for() macro builds on the numeric open-coded iterator kfuncs bpf_iter_num_{new,next,destroy}(). Add a benchmark, modelled on the existing bpf_loop benchmark, that repeatedly runs a bpf_for() loop so the per-iteration cost of the iterator can be measured, e.g. to quantify the effect of inlining the iterator kfuncs in the verifier.

The BPF program runs an inner bpf_for(i, 0, nr_loops) loop with an empty body 1000 times per trigger and accounts nr_loops hits per outer iteration, mirroring bench_bpf_loop so the two are directly comparable.

nr_loops defaults to 1000 so that the per-iteration bpf_iter_num_next() cost, rather than the one-time bpf_iter_num_new()/destroy() setup and teardown, dominates the reported numbers:

  $ ./bench -p 1 --nr_loops 1000 bpf-for
---
 tools/testing/selftests/bpf/Makefile          |   2 +
 tools/testing/selftests/bpf/bench.c           |   4 +
 .../selftests/bpf/benchs/bench_bpf_for.c      | 104 ++++++++++++++++++
 .../selftests/bpf/benchs/run_bench_bpf_for.sh |  15 +++
 .../selftests/bpf/progs/bpf_for_bench.c       |  32 ++++++
 5 files changed, 157 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_for.c
 create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh
 create mode 100644 tools/testing/selftests/bpf/progs/bpf_for_bench.c
tools/testing/selftests/bpf/Makefile +2 -0
index 55d394438705a..3174c2f18e715 100644
@@ -965,6 +965,7 @@ $(OUTPUT)/bench_ringbufs.o: $(OUTPUT)/ringbuf_bench.skel.h \
 			    $(OUTPUT)/perfbuf_bench.skel.h
 $(OUTPUT)/bench_bloom_filter_map.o: $(OUTPUT)/bloom_filter_bench.skel.h
 $(OUTPUT)/bench_bpf_loop.o: $(OUTPUT)/bpf_loop_bench.skel.h
+$(OUTPUT)/bench_bpf_for.o: $(OUTPUT)/bpf_for_bench.skel.h
 $(OUTPUT)/bench_strncmp.o: $(OUTPUT)/strncmp_bench.skel.h
 $(OUTPUT)/bench_bpf_hashmap_full_update.o: $(OUTPUT)/bpf_hashmap_full_update_bench.skel.h
 $(OUTPUT)/bench_local_storage.o: $(OUTPUT)/local_storage_bench.skel.h
@@ -990,6 +991,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \
 		 $(OUTPUT)/bench_ringbufs.o \
 		 $(OUTPUT)/bench_bloom_filter_map.o \
 		 $(OUTPUT)/bench_bpf_loop.o \
+		 $(OUTPUT)/bench_bpf_for.o \
 		 $(OUTPUT)/bench_strncmp.o \
 		 $(OUTPUT)/bench_bpf_hashmap_full_update.o \
 		 $(OUTPUT)/bench_local_storage.o \
tools/testing/selftests/bpf/bench.c +4 -0
index 3d9d2cd7764bd..b86b73456d3ca 100644
@@ -276,6 +276,7 @@ static const struct argp_option opts[] = {
 extern struct argp bench_ringbufs_argp;
 extern struct argp bench_bloom_map_argp;
 extern struct argp bench_bpf_loop_argp;
+extern struct argp bench_bpf_for_argp;
 extern struct argp bench_local_storage_argp;
 extern struct argp bench_local_storage_rcu_tasks_trace_argp;
 extern struct argp bench_strncmp_argp;
@@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = {
 	{ &bench_ringbufs_argp, 0, "Ring buffers benchmark", 0 },
 	{ &bench_bloom_map_argp, 0, "Bloom filter map benchmark", 0 },
 	{ &bench_bpf_loop_argp, 0, "bpf_loop helper benchmark", 0 },
+	{ &bench_bpf_for_argp, 0, "bpf_for loop benchmark", 0 },
 	{ &bench_local_storage_argp, 0, "local_storage benchmark", 0 },
 	{ &bench_strncmp_argp, 0, "bpf_strncmp helper benchmark", 0 },
 	{ &bench_local_storage_rcu_tasks_trace_argp, 0,
@@ -557,6 +559,7 @@ extern const struct bench bench_bloom_false_positive;
 extern const struct bench bench_hashmap_without_bloom;
 extern const struct bench bench_hashmap_with_bloom;
 extern const struct bench bench_bpf_loop;
+extern const struct bench bench_bpf_for;
 extern const struct bench bench_strncmp_no_helper;
 extern const struct bench bench_strncmp_helper;
 extern const struct bench bench_bpf_hashmap_full_update;
@@ -640,6 +643,7 @@ static const struct bench *benchs[] = {
 	&bench_hashmap_without_bloom,
 	&bench_hashmap_with_bloom,
 	&bench_bpf_loop,
+	&bench_bpf_for,
 	&bench_strncmp_no_helper,
 	&bench_strncmp_helper,
 	&bench_bpf_hashmap_full_update,
tools/testing/selftests/bpf/benchs/bench_bpf_for.c new file +104 -0

104 lines not on this page · raw message

tools/testing/selftests/bpf/benchs/run_bench_bpf_for.sh new file +15 -0
new file mode 100755
index 0000000000000..7da6453920dab
@@ -0,0 +1,15 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+source ./benchs/run_common.sh
+
+set -eufo pipefail
+
+for t in 1 4 8 12 16; do
+for i in 10 100 500 1000 5000 10000 50000 100000 500000 1000000; do
+subtitle "nr_loops: $i, nr_threads: $t"
+	summarize_ops "bpf_for: " \
+	    "$($RUN_BENCH -p $t --nr_loops $i bpf-for)"
+	printf "\n"
+done
+done
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +32 -0
new file mode 100644
index 0000000000000..f9c723051fc74
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+int nr_loops;
+long hits;
+
+static int outer_loop(__u32 index, void *data)
+{
+	int i;
+
+	/*
+	 * Empty body: the work being measured is the open-coded numeric iterator itself
+	 * (bpf_iter_num_new/next/destroy behind bpf_for()).
+	 */
+	bpf_for(i, 0, nr_loops)
+		;
+	__sync_add_and_fetch(&hits, nr_loops);
+	return 0;
+}
+
+SEC("fentry/" SYS_PREFIX "sys_getpgid")
+int benchmark(void *ctx)
+{
+	bpf_loop(1000, outer_loop, NULL, 0);
+	return 0;
+}
Signature · 1 line
2.53.0-Meta
#13 lore raw

Re: [PATCH bpf-next v4 6/6] selftests/bpf: Add bpf_for() benchmark

This isn't a bug, but since this ends up identical to bench_bpf_loop.c apart from the skeleton name and the nr_loops default, would it be worth sharing the argp/validate/producer/measure scaffolding…
Quoted 27 lines
tools/testing/selftests/bpf/benchs/bench_bpf_for.c new file +21 -0
new file mode 100644
index 000000000000..730c51ad2dec
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <argp.h>
+#include "bench.h"
+#include "bpf_for_bench.skel.h"
+
+/* BPF triggering benchmarks */
+static struct ctx {
+	struct bpf_for_bench *skel;
+} ctx;
+
+static struct {
+	__u32 nr_loops;
+} args = {
+	/*
+	 * Default to a large loop count so the per-iteration bpf_iter_num_next() cost dominates
+	 * the one-time bpf_iter_num_new()/destroy() setup and teardown.
+	 */
+	.nr_loops = 1000,
+};

[ ... ]

Quoted 24 lines
tools/testing/selftests/bpf/benchs/bench_bpf_for.c new file +24 -0
+static void *producer(void *input)
+{
+	while (true)
+		/* trigger the bpf program */
+		syscall(__NR_getpgid);
+
+	return NULL;
+}
+
+static void measure(struct bench_res *res)
+{
+	res->hits = atomic_swap(&ctx.skel->bss->hits, 0);
+}
+
+static void setup(void)
+{
+	struct bpf_link *link;
+
+	setup_libbpf();
+
+	ctx.skel = bpf_for_bench__open_and_load();
+	if (!ctx.skel) {
+		fprintf(stderr, "failed to open skeleton\n");
+		exit(1);

tools/testing/selftests/bpf/benchs/bench_bpf_for.c line 93

Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_bpf_for.c new file +10 -0
+	}
+
+	link = bpf_program__attach(ctx.skel->progs.benchmark);
+	if (!link) {
+		fprintf(stderr, "failed to attach program!\n");
+		exit(1);
+	}
+
+	ctx.skel->bss->nr_loops = args.nr_loops;
+}

This isn't a bug, but since this ends up identical to bench_bpf_loop.c apart from the skeleton name and the nr_loops default, would it be worth sharing the argp/validate/producer/measure scaffolding between the two, or is a standalone file per benchmark the preferred shape for benchs/?

Quoted 18 lines
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +12 -0
new file mode 100644
index 000000000000..f9c723051fc7
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+int nr_loops;
+long hits;
+

tools/testing/selftests/bpf/progs/bpf_for_bench.c line 22

Quoted 10 lines
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +10 -0
+static int outer_loop(__u32 index, void *data)
+{
+	int i;
+
+	/*
+	 * Empty body: the work being measured is the open-coded numeric iterator itself
+	 * (bpf_iter_num_new/next/destroy behind bpf_for()).
+	 */
+	bpf_for(i, 0, nr_loops)
+		;

This isn't a bug, but would a shorter note here - just that the loop body is intentionally empty - read closer to bpf_loop_bench.c, given the changelog already explains what is being measured?

tools/testing/selftests/bpf/progs/bpf_for_bench.c line 32

Quoted 10 lines
tools/testing/selftests/bpf/progs/bpf_for_bench.c new file +10 -0
+	__sync_add_and_fetch(&hits, nr_loops);
+	return 0;
+}
+
+SEC("fentry/" SYS_PREFIX "sys_getpgid")
+int benchmark(void *ctx)
+{
+	bpf_loop(1000, outer_loop, NULL, 0);
+	return 0;
+}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/30490674177