Skip to contentSkip to conversation

selftests/bpf: XDP LB benchmark fixes

PATCH Superseded Puranjay Mohan 5 participants 11 messages May 19

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover selftests/bpf: XDP LB benchmark fixes

Three bug fixes and one improvement for the XDP LB and batch-timing benchmarks.

The cold_lru validation was failing a lot because batch_hash could compute to zero when batch_gen matched the CPU id. Similarly, pre-populated UDP LRU entries had atime=0 so they'd expire immediately on any CPU that calibration didn't warm. Both are fixed in patches 1-2.

Patch 3 lowers CALIBRATE_MAX_BATCH to stay under BPF_MAX_LOOPS.

Patch 4 adds IQR outlier filtering to the timing stats to stabilize scenarios with high stddev.

Puranjay Mohan (4):
selftests/bpf: Fix cold_lru producing zero batch_hash in XDP LB
benchmark
selftests/bpf: Fix expired UDP LRU entries in XDP LB benchmark selftests/bpf: Cap batch-timing calibration at BPF may_goto loop limit selftests/bpf: Filter timing outliers with IQR in batch-timing library

 .../selftests/bpf/benchs/bench_bpf_timing.c   | 28 ++++++++++++++++++-
 .../selftests/bpf/benchs/bench_xdp_lb.c       | 11 ++++++++
 .../selftests/bpf/progs/xdp_lb_bench.c        |  2 +-
 3 files changed, 39 insertions(+), 2 deletions(-)
Signature · 1 line
2.53.0-Meta
#1 lore raw

PATCH 1/4 selftests/bpf: Fix cold_lru producing zero batch_hash in XDP LB benchmark

The cold_lru mechanism generates a per-batch hash to XOR into the source address, ensuring each batch tests a fresh (uncached) flow:

    batch_hash = (batch_gen ^ cpu_id) * KNUTH_HASH_MULT;
    *saddr ^= batch_hash;

When batch_gen equals the CPU ID, the XOR is zero and batch_hash becomes zero. The source address is left unchanged, so every iteration hits the warm LRU entry instead of testing cold lookups.

During validation, batch_gen is 2 (one increment from seeding, one from the validation run itself). If the BPF program happens to execute on CPU 2, batch_hash is zero and all LRU lookups hit, producing:

[udp-v4-lru-miss] COUNTER FAIL: LRU misses=0, expected 1 Validation FAILED - aborting benchmark

This fails roughly 50% of the time depending on scheduler placement.

Fix by forcing the multiplier input to be non-zero with | 1. Since KNUTH_HASH_MULT (2654435761) is odd, the product of two odd numbers is always odd, so batch_hash can never be zero.

Fixes 4b4f2229104c ("selftests/bpf: Add XDP load-balancer BPF program")
---
 tools/testing/selftests/bpf/progs/xdp_lb_bench.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/bpf/progs/xdp_lb_bench.c +1 -1
index b9fd848c035d..f40e36aa183e 100644
@@ -618,7 +618,7 @@ int xdp_lb_bench(struct xdp_md *xdp)
 		__u32 *saddr = data + saddr_off;
 
 		batch_gen++;
-		batch_hash = (batch_gen ^ bpf_get_smp_processor_id()) * KNUTH_HASH_MULT;
+		batch_hash = ((batch_gen ^ bpf_get_smp_processor_id()) | 1) * KNUTH_HASH_MULT;
 		if ((void *)(saddr + 1) <= data_end)
 			*saddr ^= batch_hash;
 	}
Signature · 1 line
2.53.0-Meta
#2 lore raw

Re: [PATCH bpf-next 1/4] selftests/bpf: Fix cold_lru producing zero batch_hash in XDP LB benchmark

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The fix for zero `batch_hash` introduces a regression in the `cold_lru` benchmark by destroying…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The fix for zero batch_hash introduces a regression in the cold_lru benchmark by destroying the lowest bit of entropy, causing 50% of consecutive batches to generate identical hashes. --

commit d60966744103a5bacba48698e7e96189a24c69c1
Author: Puranjay Mohan <[email protected]>

selftests/bpf: Fix cold_lru producing zero batch_hash in XDP LB benchmark

This commit addresses an issue in the cold_lru benchmark where a batch_hash of zero causes the benchmark to incorrectly hit warm LRU entries. It attempts to fix this by bitwise-ORing the input to the hash multiplier with 1.

tools/testing/selftests/bpf/progs/xdp_lb_bench.c int xdp_lb_bench(struct xdp_md *xdp) · line 621

Quoted 9 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c +1 -1
@@ -618,7 +618,7 @@ int xdp_lb_bench(struct xdp_md *xdp)
 		__u32 *saddr = data + saddr_off;
 
 		batch_gen++;
-		batch_hash = (batch_gen ^ bpf_get_smp_processor_id()) * KNUTH_HASH_MULT;
+		batch_hash = ((batch_gen ^ bpf_get_smp_processor_id()) | 1) * KNUTH_HASH_MULT;

[Severity: Medium]
Does this code introduce a regression where using | 1 inadvertently destroys the lowest bit of entropy from the batch generator?

Since batch_gen increments sequentially by 1 per batch execution, forcing the least significant bit to 1 maps pairs of adjacent values to the same number. For example, if the CPU ID evaluates to 0, both batch_gen = 2 and batch_gen = 3 will yield 3 before the multiplication.

Could this cause 50% of consecutive batches to generate the exact same batch_hash as the previous batch, causing those iterations to hit the warm LRU cache entries that were just populated and defeating the purpose of the cold_lru benchmark?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [PATCH bpf-next 1/4] selftests/bpf: Fix cold_lru producing zero batch_hash in XDP LB benchmark

In reply to Puranjay Mohan
Does forcing the LSB to 1 with '| 1' inadvertently destroy the lowest bit of entropy from the batch generator?

tools/testing/selftests/bpf/progs/xdp_lb_bench.c int xdp_lb_bench(struct xdp_md *xdp) · line 621

Quoted 10 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c +1 -1
index b9fd848c035d..f40e36aa183e 100644
@@ -618,7 +618,7 @@ int xdp_lb_bench(struct xdp_md *xdp)
 		__u32 *saddr = data + saddr_off;
 
 		batch_gen++;
-		batch_hash = (batch_gen ^ bpf_get_smp_processor_id()) * KNUTH_HASH_MULT;
+		batch_hash = ((batch_gen ^ bpf_get_smp_processor_id()) | 1) * KNUTH_HASH_MULT;
                                                                           ^^^^

Does forcing the LSB to 1 with '| 1' inadvertently destroy the lowest bit of entropy from the batch generator?

Since batch_gen increments sequentially by 1 per batch execution, forcing the least significant bit to 1 maps pairs of adjacent values to the same number. For example, if the CPU ID evaluates to 0, both batch_gen = 2 and batch_gen = 3 will yield 3 before the multiplication:

  batch_gen = 2, CPU = 0:  (2 ^ 0) | 1 = 2 | 1 = 3
  batch_gen = 3, CPU = 0:  (3 ^ 0) | 1 = 3 | 1 = 3

Could this cause 50% of consecutive batches to generate the exact same batch_hash as the previous batch, causing those iterations to hit the warm LRU cache entries that were just populated and defeating the purpose of the cold_lru benchmark?

Reference: https://lore.kernel.org/bpf/[email protected]/

tools/testing/selftests/bpf/progs/xdp_lb_bench.c int xdp_lb_bench(struct xdp_md *xdp) · line 624

Quoted 3 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c +0 -0
 		if ((void *)(saddr + 1) <= data_end)
 			*saddr ^= batch_hash;
 	}

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/26112027147

#4 lore raw

Re: [PATCH bpf-next 1/4] selftests/bpf: Fix cold_lru producing zero batch_hash in XDP LB benchmark

Quoted 21 lines · On Tue, May 19, 2026 at 6:14 PM <[email protected]> wrote:

tools/testing/selftests/bpf/progs/xdp_lb_bench.c int xdp_lb_bench(struct xdp_md *xdp) · line 621

Quoted 11 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c +1 -1
index b9fd848c035d..f40e36aa183e 100644
@@ -618,7 +618,7 @@ int xdp_lb_bench(struct xdp_md *xdp)
              __u32 *saddr = data + saddr_off;
 
              batch_gen++;
-             batch_hash = (batch_gen ^ bpf_get_smp_processor_id()) * KNUTH_HASH_MULT;
+             batch_hash = ((batch_gen ^ bpf_get_smp_processor_id()) | 1) * KNUTH_HASH_MULT;
                                                                           ^^^^

Does forcing the LSB to 1 with '| 1' inadvertently destroy the lowest bit of entropy from the batch generator?

Since batch_gen increments sequentially by 1 per batch execution, forcing the least significant bit to 1 maps pairs of adjacent values to the same number. For example, if the CPU ID evaluates to 0, both batch_gen = 2 and batch_gen = 3 will yield 3 before the multiplication:

Quoted 7 lines
  batch_gen = 2, CPU = 0:  (2 ^ 0) | 1 = 2 | 1 = 3
  batch_gen = 3, CPU = 0:  (3 ^ 0) | 1 = 3 | 1 = 3

Could this cause 50% of consecutive batches to generate the exact same batch_hash as the previous batch, causing those iterations to hit the warm LRU cache entries that were just populated and defeating the purpose of the cold_lru benchmark?

Yeah, this analysis is correct, I need to fix it another way.

#5 lore raw

PATCH 2/4 selftests/bpf: Fix expired UDP LRU entries in XDP LB benchmark

In reply to Puranjay Mohan

populate_lru() inserts LRU entries with atime zero-initialized:

    struct real_pos_lru lru = { .pos = real_idx };

The BPF program's connection_table_lookup() enforces a 30-second timeout for UDP flows:

    if (cur_time - dst_lru->atime > LRU_UDP_TIMEOUT)
        return NULL;

Since cur_time (system uptime) is always much larger than 30 seconds, every pre-populated UDP entry is treated as expired on first access. The lookup returns NULL, counting as an LRU miss.

This is masked by calibration: the BPF program runs during calibration on one CPU, misses the expired entry, and re-inserts it with a fresh atime. But if the scheduler moves the thread to a different CPU for validation, that CPU still has the atime=0 entry from populate_lru(), causing intermittent validation failures for UDP LRU-hit scenarios:

[udp-v4-lru-hit] COUNTER FAIL: LRU misses=1, expected 0

Fix by initializing atime with the current CLOCK_MONOTONIC time for UDP flows, matching the clock source used by bpf_ktime_get_ns() in the BPF program.

Fixes a4b5ba8187cb ("selftests/bpf: Add XDP load-balancer benchmark driver")
---
 tools/testing/selftests/bpf/benchs/bench_xdp_lb.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c +11 -0
index 0b6709a2b03c..8e25bccbde92 100644
@@ -563,12 +563,23 @@ static void create_per_cpu_lru_maps(struct xdp_lb_bench *skel)
 	nr_inner_maps = nr_cpus;
 }
 
+static __u64 ktime_get_ns(void)
+{
+	struct timespec ts;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	return (__u64)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
+}
+
 static void populate_lru(const struct test_scenario *sc, __u32 real_idx)
 {
 	struct real_pos_lru lru = { .pos = real_idx };
 	struct flow_key fk;
 	int i, err;
 
+	if (sc->ip_proto == IPPROTO_UDP)
+		lru.atime = ktime_get_ns();
+
 	build_flow_key(&fk, sc);
 
 	/* Insert into every per-CPU inner LRU so the entry is found
Signature · 1 line
2.53.0-Meta
#6 lore raw

PATCH 3/4 selftests/bpf: Cap batch-timing calibration at BPF may_goto loop limit

In reply to Puranjay Mohan

BENCH_BPF_LOOP uses the may_goto instruction (via can_loop) to bound its iteration loop. The kernel enforces a maximum of BPF_MAX_LOOPS (8,388,608) iterations per may_goto instruction.

CALIBRATE_MAX_BATCH was set to 10,000,000, exceeding this limit. If calibration computed a batch_iters above 8M, the BPF loop would exit early when can_loop becomes false. The timing sample would cover fewer iterations than batch_iters, causing the per-op calculation to underestimate the true cost. Validation would also fail since map counters would not reach the expected batch_iters + 1.

No current benchmark triggers this (the fastest, bpf-nop at ~1.8 ns/op, calibrates to ~5.5M), but future sub-1.25 ns/op benchmarks would.

Lower the cap to 8,000,000 to stay within the may_goto limit.

Fixes 08158c111d7d ("selftests/bpf: Add BPF batch-timing library")
---
 tools/testing/selftests/bpf/benchs/bench_bpf_timing.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c +1 -1
index 75a39da69655..081e2e860cb4 100644
@@ -166,7 +166,7 @@ void bpf_bench_timing_report(struct bpf_bench_timing *t, const char *name, const
 
 #define CALIBRATE_SEED_BATCH	100
 #define CALIBRATE_MIN_BATCH	100
-#define CALIBRATE_MAX_BATCH	10000000
+#define CALIBRATE_MAX_BATCH	8000000
 #define CALIBRATE_TARGET_MS	10
 #define CALIBRATE_RUNS		5
 #define PROPORTIONALITY_TOL	0.05	/* 5% */
Signature · 1 line
2.53.0-Meta
#7 lore raw

Re: [PATCH bpf-next 3/4] selftests/bpf: Cap batch-timing calibration at BPF may_goto loop limit

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The new CALIBRATE_MAX_BATCH cap of 8,000,000 is mathematically incorrect and fails the patch's…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The new CALIBRATE_MAX_BATCH cap of 8,000,000 is mathematically incorrect and fails the patch's stated goal of keeping the benchmark within the BPF may_goto limit. --

commit d9779455687fa035b58a0d4ba0d495256b32ed13
Author: Puranjay Mohan <[email protected]>

selftests/bpf: Cap batch-timing calibration at BPF may_goto loop limit

This commit lowers CALIBRATE_MAX_BATCH from 10000000 to 8000000. This is intended to keep the benchmark iterations within the BPF_MAX_LOOPS limit of 8388608, avoiding early loop exits that cause underestimated costs.

tools/testing/selftests/bpf/benchs/bench_bpf_timing.c void bpf_bench_timing_report(struct bpf_bench_timing *t, const char *name, const · line 169

Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c +1 -1
index 75a39da696552..081e2e860cb49 100644
@@ -166,7 +166,7 @@ void bpf_bench_timing_report(struct bpf_bench_timing *t, const char *name, const
 
 #define CALIBRATE_SEED_BATCH	100
 #define CALIBRATE_MIN_BATCH	100
-#define CALIBRATE_MAX_BATCH	10000000
+#define CALIBRATE_MAX_BATCH	8000000

[Severity: Medium]
Will this new cap of 8000000 still cause the benchmark to hit the BPF may_goto limit during calibration?

Looking at bpf_bench_calibrate(), the proportionality check explicitly requests a measurement of t->batch_iters * 2:

	time_n = measure_elapsed(t, run_fn, run_ctx, t->batch_iters, CALIBRATE_RUNS);
	time_2n = measure_elapsed(t, run_fn, run_ctx, t->batch_iters * 2, CALIBRATE_RUNS);

With a cap of 8000000, t->batch_iters * 2 can reach up to 16000000 iterations. Since this exceeds the BPF_MAX_LOOPS limit of 8388608, won't the BPF program's can_loop condition return false early during the time_2n measurement?

This would cause time_2n to record the time for only ~8.38M iterations instead of 16M. When compared to time_n (8M iterations), the ratio would be ~1.05 instead of the expected 2.0. This completely violates the 5% PROPORTIONALITY_TOL and triggers a bogus proportionality check warning for fast benchmarks.

To safely avoid the limit across all phases of calibration, shouldn't the cap be no more than BPF_MAX_LOOPS / 2 (e.g., 4000000)?

Quoted 3 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c +0 -0
 #define CALIBRATE_TARGET_MS	10
 #define CALIBRATE_RUNS		5
 #define PROPORTIONALITY_TOL	0.05	/* 5% */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#8 lore raw

Re: [PATCH bpf-next 3/4] selftests/bpf: Cap batch-timing calibration at BPF may_goto loop limit

In reply to Puranjay Mohan
Quoted 4 lines · Puranjay Mohan

BENCH_BPF_LOOP uses the may_goto instruction (via can_loop) to bound its iteration loop. The kernel enforces a maximum of BPF_MAX_LOOPS (8,388,608) iterations per may_goto instruction.

claude is incorrect.
may_goto is timer based. It aborts at 1/4 of a second.
Old BPF_MAX_LOOPS limit applies for bpf_loop and bpf_for() iters only. Is this a theoretical issue?

#9 lore raw

Re: [PATCH bpf-next 3/4] selftests/bpf: Cap batch-timing calibration at BPF may_goto loop limit

Quoted 11 lines · Alexei Starovoitov
Quoted 4 lines · Puranjay Mohan

BENCH_BPF_LOOP uses the may_goto instruction (via can_loop) to bound its iteration loop. The kernel enforces a maximum of BPF_MAX_LOOPS (8,388,608) iterations per may_goto instruction.

claude is incorrect.
may_goto is timer based. It aborts at 1/4 of a second.
Old BPF_MAX_LOOPS limit applies for bpf_loop and bpf_for() iters only. Is this a theoretical issue?

Yes, this is theoretical, the benchmark will never cross this limit because it runs the loop for 20ms at max.
So, I will drop this patch in the next version.

#10 lore raw

PATCH 4/4 selftests/bpf: Filter timing outliers with IQR in batch-timing library

In reply to Puranjay Mohan

System noise (timer interrupts, scheduling) can produce outlier batch samples that inflate the reported stddev. For example, tcp-v4-syn showed stddev 37.86 ns without filtering vs 0.16 ns with filtering on the same run, because a handful of interrupt-hit batches dominated the variance.

Apply IQR-based outlier filtering (1.5 * IQR fences) before computing statistics. Samples outside [Q1 - 1.5*IQR, Q3 + 1.5*IQR] are discarded. This removes system noise while preserving genuine operational variance: scenarios with inherently wide distributions (e.g., LRU-miss with eviction pressure) have large IQR, so their fences are wide and the filter has minimal effect.

---
 .../selftests/bpf/benchs/bench_bpf_timing.c   | 26 +++++++++++++++++++
 1 file changed, 26 insertions(+)
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c +26 -0
index 081e2e860cb4..1c3a74807c1c 100644
@@ -65,6 +65,31 @@ static int collect_samples(struct bpf_bench_timing *t,
 	return total;
 }
 
+static int filter_outliers_iqr(double *sorted, int n)
+{
+	double q1, q3, iqr, lo, hi;
+	int start = 0, end = n;
+
+	if (n < 8)
+		return n;
+
+	q1 = sorted[n / 4];
+	q3 = sorted[3 * n / 4];
+	iqr = q3 - q1;
+	lo = q1 - 1.5 * iqr;
+	hi = q3 + 1.5 * iqr;
+
+	while (start < end && sorted[start] < lo)
+		start++;
+	while (end > start && sorted[end - 1] > hi)
+		end--;
+
+	if (start > 0)
+		memmove(sorted, sorted + start, (end - start) * sizeof(double));
+
+	return end - start;
+}
+
 static void compute_stats(const double *sorted, int n,
 			  struct timing_stats *s)
 {
@@ -150,6 +175,7 @@ void bpf_bench_timing_report(struct bpf_bench_timing *t, const char *name, const
 		return;
 	}
 
+	total = filter_outliers_iqr(all, total);
 	compute_stats(all, total, &s);
 
 	if (t->machine_readable) {
Signature · 1 line
2.53.0-Meta