selftests/bpf: Add XDP load-balancer benchmark
Subsystems (4)
cover selftests/bpf: Add XDP load-balancer benchmark
Changelog:
RFC: https://lore.kernel.org/all/[email protected]/
Changes in v1:
- Replace bpf_get_cpu_time_counter() with bpf_ktime_get_ns()
- Replace bpf_repeat() with plain for loop and may_goto
- Refactor collect_measurements() to reuse bench_force_done()
- Remove histogram, verbose calibration output, and per-scenario status prints
- Trim run script table to p50/stddev/p99
- Set env.quiet when --machine-readable is passed
- Add || true to run script benchmark invocation for set -e safety
- Add bpf-nop benchmark as timing overhead baseline (patch 3)
- Use named struct for LRU inner map to fix build on older toolchains
This series adds an XDP load-balancer benchmark (based on Katran) to the BPF selftest bench framework.
Motivation
----------
Existing BPF bench tests measure individual operations (map lookups, kprobes, ring buffers) in isolation. Production BPF programs combine parsing, map lookups, branching, and packet rewriting in a single call chain. The performance characteristics of such programs depend on the interaction of these operations -- register pressure, spills, inlining decisions, branch layout -- which isolated micro-benchmarks do not capture.
This benchmark implements a simplified L4 load-balancer modeled after katran [1]. The BPF program reproduces katran's core datapath:
L3/L4 parsing -> VIP hash lookup -> per-CPU LRU connection table with consistent-hash fallback -> real server selection -> per-VIP and per-real stats -> IPIP/IP6IP6 encapsulation
The BPF code exercises hash maps, array-of-maps (per-CPU LRU), percpu arrays, jhash, bpf_xdp_adjust_head(), bpf_ktime_get_ns(), and bpf_get_smp_processor_id() in a single pipeline.
This is intended as the first in a series of BPF workload benchmarks covering other use cases (sched_ext, etc.).
Design
------
A userspace loop calling bpf_prog_test_run_opts(repeat=1) would measure syscall overhead, not BPF program cost -- the ~4 ns early-exit paths would be buried under kernel entry/exit. Using repeat=N is also unsuitable: the kernel re-runs the same packet without resetting state between iterations, so the second iteration of an encap scenario would process an already-encapsulated packet.
Instead, timing is measured inside the BPF program using bpf_ktime_get_ns(). BENCH_BPF_LOOP() brackets N iterations with timestamp reads using a plain for loop with may_goto, runs a caller-supplied reset block between iterations to undo side effects (e.g. strip encapsulation), and records the elapsed time per batch. One extra untimed iteration runs afterward for output validation.
Auto-calibration picks a batch size targeting ~10 ms per invocation. A proportionality sanity check verifies that 2N iterations take ~2x as long as N.
24 scenarios cover the code-path matrix:
- Protocol: TCP, UDP
- Address family: IPv4, IPv6, cross-AF (IPv4-in-IPv6)
- LRU state: hit, miss (16M flow space), diverse (4K flows), cold
- Consistent-hash: direct (LRU bypass)
- TCP flags: SYN (skip LRU, force CH), RST (skip LRU insert)
- Early exits: unknown VIP, non-IP, ICMP, fragments, IP options
Each scenario validates correctness before benchmarking by comparing the output packet byte-for-byte against a pre-built expected packet and checking BPF map counters.
Sample single-scenario output:
$ sudo ./bench xdp-lb --scenario tcp-v4-lru-hit Setting up benchmark 'xdp-lb'... Benchmark 'xdp-lb' started. tcp-v4-lru-hit: median 74.51 ns/op, stddev 0.11, p99 74.81 (202 samples)
Sample run script output:
$ ./benchs/run_bench_xdp_lb.sh
XDP load-balancer benchmark
===========================
+----------------------------------+----------+---------+----------+ | Single-flow baseline | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-hit | 74.30 | 0.08 | 74.48 | | tcp-v4-ch | 101.73 | 0.11 | 102.01 | | tcp-v6-lru-hit | 76.77 | 0.14 | 77.04 | | tcp-v6-ch | 121.40 | 0.10 | 121.65 | | udp-v4-lru-hit | 107.42 | 0.22 | 107.90 | | udp-v6-lru-hit | 110.21 | 0.12 | 110.45 | | tcp-v4v6-lru-hit | 74.82 | 0.35 | 75.43 | +----------------------------------+----------+---------+----------+ | Diverse flows (4K src addrs) | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-diverse | 86.63 | 0.37 | 89.04 | | tcp-v4-ch-diverse | 104.09 | 0.19 | 105.67 | | tcp-v6-lru-diverse | 89.34 | 0.42 | 90.70 | | tcp-v6-ch-diverse | 122.20 | 0.21 | 123.78 | | udp-v4-lru-diverse | 119.37 | 0.58 | 123.10 | +----------------------------------+----------+---------+----------+ | TCP flags | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-syn | 165.52 | 15.68 | 198.34 | | tcp-v4-rst-miss | 161.34 | 2.69 | 172.64 | +----------------------------------+----------+---------+----------+ | LRU stress | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-miss | 440.39 | 35.75 | 550.62 | | udp-v4-lru-miss | 571.88 | 57.38 | 680.61 | | tcp-v4-lru-warmup | 317.75 | 9.55 | 356.20 | +----------------------------------+----------+---------+----------+ | Early exits | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | pass-v4-no-vip | 18.26 | 0.13 | 18.66 | | pass-v6-no-vip | 19.08 | 0.01 | 19.10 | | pass-v4-icmp | 6.81 | 0.02 | 6.86 | | pass-non-ip | 5.71 | 0.03 | 5.76 | | drop-v4-frag | 6.09 | 0.01 | 6.10 | | drop-v4-options | 5.88 | 0.00 | 5.89 | | drop-v6-frag | 6.00 | 0.03 | 6.04 | +----------------------------------+----------+---------+----------+
Patches
-------
Patch 1 adds bench_force_done() to the bench framework so benchmarks can signal early completion when enough samples have been collected.
Patch 2 adds the shared BPF batch-timing library (BPF-side timing arrays, BENCH_BPF_LOOP macro, userspace statistics and calibration).
Patch 3 adds a bpf-nop benchmark as a timing overhead baseline and usage example for the timing library.
Patch 4 adds the common header shared between the BPF program and userspace (flow_key, vip_definition, real_definition, encap helpers).
Patch 5 adds the XDP load-balancer BPF program.
Patch 6 adds the userspace benchmark driver with 24 scenarios, packet construction, validation, and bench framework integration.
Patch 7 adds the run script for running all scenarios.
[1] https://github.com/facebookincubator/katran
Puranjay Mohan (7):
selftests/bpf: Add bench_force_done() for early benchmark completion selftests/bpf: Add BPF batch-timing library selftests/bpf: Add bpf-nop benchmark for timing overhead baseline selftests/bpf: Add XDP load-balancer common definitions selftests/bpf: Add XDP load-balancer BPF program selftests/bpf: Add XDP load-balancer benchmark driver selftests/bpf: Add XDP load-balancer benchmark run script
tools/testing/selftests/bpf/Makefile | 6 + tools/testing/selftests/bpf/bench.c | 20 +- tools/testing/selftests/bpf/bench.h | 1 + .../testing/selftests/bpf/bench_bpf_timing.h | 50 + .../selftests/bpf/benchs/bench_bpf_nop.c | 84 ++ .../selftests/bpf/benchs/bench_bpf_timing.c | 272 ++++ .../selftests/bpf/benchs/bench_xdp_lb.c | 1113 +++++++++++++++++ .../selftests/bpf/benchs/run_bench_xdp_lb.sh | 79 ++ .../bpf/progs/bench_bpf_timing.bpf.h | 69 + .../selftests/bpf/progs/bpf_nop_bench.c | 14 + .../selftests/bpf/progs/xdp_lb_bench.c | 647 ++++++++++ .../selftests/bpf/xdp_lb_bench_common.h | 112 ++ 12 files changed, 2462 insertions(+), 5 deletions(-) create mode 100644 tools/testing/selftests/bpf/bench_bpf_timing.h create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_nop.c create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_timing.c create mode 100644 tools/testing/selftests/bpf/benchs/bench_xdp_lb.c create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh create mode 100644 tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h create mode 100644 tools/testing/selftests/bpf/progs/bpf_nop_bench.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_lb_bench.c create mode 100644 tools/testing/selftests/bpf/xdp_lb_bench_common.h
Signature · 1 line
2.52.0
PATCH 1/7 selftests/bpf: Add bench_force_done() for early benchmark completion
The bench framework waits for duration_sec to elapse before collecting results. Benchmarks that know exactly how many samples they need can call bench_force_done() to signal completion early, avoiding wasted wall-clock time.
Also refactor collect_measurements() to reuse bench_force_done() instead of open-coding the same mutex/cond_signal sequence.
--- tools/testing/selftests/bpf/bench.c | 14 +++++++++----- tools/testing/selftests/bpf/bench.h | 1 + 2 files changed, 10 insertions(+), 5 deletions(-)
tools/testing/selftests/bpf/bench.c +9 -5
@@ -741,6 +741,13 @@ static void setup_benchmark(void)static pthread_mutex_t bench_done_mtx =PTHREAD_MUTEX_INITIALIZER ;static pthread_cond_t bench_done =PTHREAD_COND_INITIALIZER ; +void bench_force_done (void ) +{ +pthread_mutex_lock (&bench_done_mtx); +pthread_cond_signal (&bench_done); +pthread_mutex_unlock (&bench_done_mtx); +} +static void collect_measurements (long delta_ns) {int iter = state.res_cnt++;struct bench_res *res = &state.results[iter];
@@ -750,11 +757,8 @@ static void collect_measurements(long delta_ns) {if (bench->report_progress) bench->report_progress (iter, res, delta_ns); -if (iter == env.duration_sec + env.warmup_sec) { -pthread_mutex_lock (&bench_done_mtx); -pthread_cond_signal (&bench_done); -pthread_mutex_unlock (&bench_done_mtx); - } +if (iter == env.duration_sec + env.warmup_sec) +bench_force_done (); }int main (int argc,char **argv)
tools/testing/selftests/bpf/bench.h +1 -0
@@ -70,6 +70,7 @@ extern struct env env;extern const struct bench *bench;void setup_libbpf (void ); +void bench_force_done (void );void hits_drops_report_progress (int iter,struct bench_res *res,long delta_ns);void hits_drops_report_final (struct bench_res res[],int res_cnt);void false_hits_report_progress (int iter,struct bench_res *res,long delta_ns);
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 1/7] selftests/bpf: Add bench_force_done() for early benchmark completion
Sashiko AI review found 3 potential issue(s): - [High] Calling non-async-signal-safe POSIX functions in a signal handler creates a self-deadlock risk. - [Medium] Missing state predicate for…
Sashiko AI review found 3 potential issue(s):
- [High] Calling non-async-signal-safe POSIX functions in a signal handler creates a self-deadlock risk.
- [Medium] Missing state predicate for condition variable causes lost wakeups, defeating the patch's purpose.
- [Medium] Early benchmark completion passes negative sample counts to reporting functions, yielding NaN outputs. --
commit 7ae4c67dc4880455d3c6c1f70bd9e8d56eb4e69b
Author: Puranjay Mohan <[email protected]>
selftests/bpf: Add bench_force_done() for early benchmark completion
The bench framework waits for duration_sec to elapse before collecting results. Benchmarks that know exactly how many samples they need can call bench_force_done() to signal completion early, avoiding wasted wall-clock time.
Quoted 4 lines
tools/testing/selftests/bpf/bench.c static void setup_benchmark(void) · line 750
Quoted 10 lines
tools/testing/selftests/bpf/bench.c +7
-0@@ -741,6 +741,13 @@ static void setup_benchmark(void)static pthread_mutex_t bench_done_mtx =PTHREAD_MUTEX_INITIALIZER ;static pthread_cond_t bench_done =PTHREAD_COND_INITIALIZER ; +void bench_force_done (void ) +{ +pthread_mutex_lock (&bench_done_mtx); +pthread_cond_signal (&bench_done); +pthread_mutex_unlock (&bench_done_mtx); +} +
Since bench_force_done() is now exported for concurrent benchmark worker threads to call directly, could this lead to lost wakeups?
Condition variables are stateless. If a fast-running benchmark finishes and signals the condition variable before main() reaches its wait statement:
tools/testing/selftests/bpf/bench.c:main() {
...
pthread_cond_wait(&bench_done, &bench_done_mtx);
...
}Will the signal be permanently lost, causing main() to sleep for the full duration and defeating the goal of the patch?
Also, if a benchmark legitimately completes its target operations and calls bench_force_done() before the warmup period is fully satisfied:
tools/testing/selftests/bpf/bench.c:main() {
...
if (bench->report_final)
bench->report_final(state.results + env.warmup_sec,
state.res_cnt - env.warmup_sec);
}Could state.res_cnt - env.warmup_sec evaluate to a negative integer?
Passing a negative sample count to reporting functions might cause reporter loops to fail bounds checks and yield nan outputs.
Quoted 5 lines
tools/testing/selftests/bpf/bench.c +0
-0static void collect_measurements (long delta_ns) {int iter = state.res_cnt++;struct bench_res *res = &state.results[iter];@@ -750,11 +757,8 @@ static void collect_measurements(long delta_ns) {if (bench->report_progress)
tools/testing/selftests/bpf/bench.c static void collect_measurements(long delta_ns) { · line 762
Quoted 10 lines
tools/testing/selftests/bpf/bench.c +2
-5bench->report_progress (iter, res, delta_ns); -if (iter == env.duration_sec + env.warmup_sec) { -pthread_mutex_lock (&bench_done_mtx); -pthread_cond_signal (&bench_done); -pthread_mutex_unlock (&bench_done_mtx); - } +if (iter == env.duration_sec + env.warmup_sec) +bench_force_done (); }
collect_measurements() executes in the context of a timer signal handler.
Posix specifies that pthread_mutex_lock() and pthread_cond_signal() are not async-signal-safe.
If the timer signal interrupts a thread exactly while it already holds bench_done_mtx, will the signal handler attempt to acquire the same non-recursive mutex and cause a self-deadlock?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[email protected]
AI reviewRe: [PATCH bpf-next 1/7] selftests/bpf: Add bench_force_done() for early benchmark completion
In reply to Puranjay MohanIs it safe to use pthread_mutex_lock() and pthread_cond_signal() here given that collect_measurements() calls bench_force_done() from a SIGALRM signal handler (sigalarm_handler)?
Quoted 5 lines
tools/testing/selftests/bpf/bench.c +0
-0
tools/testing/selftests/bpf/bench.c static void setup_benchmark(void) · line 750
Quoted 10 lines
tools/testing/selftests/bpf/bench.c +7
-0@@ -741,6 +741,13 @@ static void setup_benchmark(void)static pthread_mutex_t bench_done_mtx =PTHREAD_MUTEX_INITIALIZER ;static pthread_cond_t bench_done =PTHREAD_COND_INITIALIZER ; +void bench_force_done (void ) +{ +pthread_mutex_lock (&bench_done_mtx); +pthread_cond_signal (&bench_done); +pthread_mutex_unlock (&bench_done_mtx); +} +
Is it safe to use pthread_mutex_lock() and pthread_cond_signal() here given that collect_measurements() calls bench_force_done() from a SIGALRM signal handler (sigalarm_handler)?
If the signal interrupts a thread that is already holding bench_done_mtx (for example, a worker thread executing bench_force_done()), the signal handler will attempt to lock the same mutex:
Thread context:
bench_force_done()
pthread_mutex_lock(&bench_done_mtx); // lock acquired [SIGALRM arrives]
Signal handler context:
sigalarm_handler()
collect_measurements()
bench_force_done()
pthread_mutex_lock(&bench_done_mtx); // deadlock
This causes a self-deadlock because pthread_mutex_lock() and pthread_cond_signal() are not async-signal-safe according to POSIX.
Could an async-signal-safe mechanism like an atomic flag with sem_post() be used instead?
This concern was raised in the v1 and v2 reviews by Sashiko AI on lore:
https://lore.kernel.org/bpf/[email protected]/
Could this code lose wakeups because the condition variable is signaled without a shared state predicate?
If a worker thread completes quickly and calls bench_force_done() before main() reaches pthread_cond_wait(), the signal will be lost because POSIX condition variables do not retain state:
Worker thread:
bench_force_done()
pthread_cond_signal(&bench_done); // signal sent
Main thread (not waiting yet):
[still in setup]
pthread_cond_wait(&bench_done, &bench_done_mtx); // misses signal
This would cause main() to block for the full timer duration, defeating the purpose of bench_force_done(). Should a boolean flag be added to track the done state?
This concern was raised in the v1 and v2 reviews by Sashiko AI on lore:
https://lore.kernel.org/bpf/[email protected]/
Quoted 5 lines
tools/testing/selftests/bpf/bench.c +0
-0static void collect_measurements (long delta_ns) {int iter = state.res_cnt++;struct bench_res *res = &state.results[iter];@@ -750,11 +757,8 @@ static void collect_measurements(long delta_ns) {if (bench->report_progress)
tools/testing/selftests/bpf/bench.c static void collect_measurements(long delta_ns) { · line 762
Quoted 10 lines
tools/testing/selftests/bpf/bench.c +2
-5bench->report_progress (iter, res, delta_ns); -if (iter == env.duration_sec + env.warmup_sec) { -pthread_mutex_lock (&bench_done_mtx); -pthread_cond_signal (&bench_done); -pthread_mutex_unlock (&bench_done_mtx); - } +if (iter == env.duration_sec + env.warmup_sec) +bench_force_done (); }
[ ... ]
What happens if bench_force_done() is called before the warmup period fully elapses?
In main(), the final sample count passed to bench->report_final() is calculated as:
state.res_cnt - env.warmup_sec
If state.res_cnt is strictly less than env.warmup_sec, the resulting negative count will bypass the iteration loops in the report_final callbacks (like ops_report_final()), leaving aggregate variables at zero.
Would subsequent calculations divide by zero and corrupt the benchmark summary?
This concern was raised in the v1 and v2 reviews by Sashiko AI on lore:
https://lore.kernel.org/bpf/[email protected]/
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
Re: [PATCH bpf-next 1/7] selftests/bpf: Add bench_force_done() for early benchmark completion
Quoted 21 lines · On Tue, Apr 28, 2026 at 1:05 AM <[email protected]> wrote:
Quoted 6 lines
tools/testing/selftests/bpf/bench.c +0
-0tools/testing/selftests/bpf/bench.c static void setup_benchmark(void) · line 750
Quoted 10 lines
tools/testing/selftests/bpf/bench.c +7
-0@@ -741,6 +741,13 @@ static void setup_benchmark(void)static pthread_mutex_t bench_done_mtx =PTHREAD_MUTEX_INITIALIZER ;static pthread_cond_t bench_done =PTHREAD_COND_INITIALIZER ; +void bench_force_done (void ) +{ +pthread_mutex_lock (&bench_done_mtx); +pthread_cond_signal (&bench_done); +pthread_mutex_unlock (&bench_done_mtx); +} +Is it safe to use
pthread_mutex_lock()andpthread_cond_signal()here given thatcollect_measurements()callsbench_force_done()from a SIGALRM signal handler (sigalarm_handler)?
Quoted 8 lines
If the signal interrupts a thread that is already holding
bench_done_mtx(for example, a worker thread executingbench_force_done()), the signal handler will attempt to lock the same mutex:Thread context:
bench_force_done()
pthread_mutex_lock(&bench_done_mtx); // lock acquired [SIGALRM arrives]
bench_force_done() is expected to be called from SIGALARM handlers only, it is never called in Thread context.
Quoted 28 lines
Signal handler context:
sigalarm_handler()
collect_measurements()
bench_force_done()
pthread_mutex_lock(&bench_done_mtx); // deadlockThis causes a self-deadlock because
pthread_mutex_lock()andpthread_cond_signal()are not async-signal-safe according to POSIX.Could an async-signal-safe mechanism like an atomic flag with
sem_post()be used instead?This concern was raised in the v1 and v2 reviews by Sashiko AI on lore:
https://lore.kernel.org/bpf/[email protected]/Could this code lose wakeups because the condition variable is signaled without a shared state predicate?
If a worker thread completes quickly and calls
bench_force_done()beforemain()reachespthread_cond_wait(), the signal will be lost because POSIX condition variables do not retain state:Worker thread:
bench_force_done()
pthread_cond_signal(&bench_done); // signal sent
Quoted 7 lines
Main thread (not waiting yet):
[still in setup]
pthread_cond_wait(&bench_done, &bench_done_mtx); // misses signalThis would cause
main()to block for the full timer duration, defeating the purpose ofbench_force_done(). Should a boolean flag be added to track the done state?
t->warmup_ticks++;
if (t->warmup_ticks < env.warmup_sec)
return;
if (t->warmup_ticks == env.warmup_sec) {
*t->timing_enabled = 1;
return;
}
// ... only here can bench_force_done() is calledSo bench_force_done() fires at earliest on tick warmup_sec + 1, meaning state.res_cnt >= warmup_sec + 1 when it fires. The subtraction is always positive
Quoted 46 lines
This concern was raised in the v1 and v2 reviews by Sashiko AI on lore:
https://lore.kernel.org/bpf/[email protected]/Quoted 5 lines
tools/testing/selftests/bpf/bench.c +0
-0static void collect_measurements (long delta_ns) {int iter = state.res_cnt++;struct bench_res *res = &state.results[iter];@@ -750,11 +757,8 @@ static void collect_measurements(long delta_ns) {if (bench->report_progress)tools/testing/selftests/bpf/bench.c static void collect_measurements(long delta_ns) { · line 762
Quoted 10 lines
tools/testing/selftests/bpf/bench.c +2
-5bench->report_progress (iter, res, delta_ns); -if (iter == env.duration_sec + env.warmup_sec) { -pthread_mutex_lock (&bench_done_mtx); -pthread_cond_signal (&bench_done); -pthread_mutex_unlock (&bench_done_mtx); - } +if (iter == env.duration_sec + env.warmup_sec) +bench_force_done (); }[ ... ]
What happens if
bench_force_done()is called before the warmup period fully elapses?In
main(), the final sample count passed tobench->report_final()is calculated as:state.res_cnt - env.warmup_sec
If state.res_cnt is strictly less than env.warmup_sec, the resulting negative count will bypass the iteration loops in the
report_finalcallbacks (likeops_report_final()), leaving aggregate variables at zero.Would subsequent calculations divide by zero and corrupt the benchmark summary?
This concern was raised in the v1 and v2 reviews by Sashiko AI on lore:
https://lore.kernel.org/bpf/[email protected]/---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
PATCH 2/7 selftests/bpf: Add BPF batch-timing library
In reply to Puranjay MohanAdd a reusable timing library for BPF benchmarks that need to measure BPF program execution time.
The BPF side (progs/bench_bpf_timing.bpf.h) provides per-CPU sample arrays and BENCH_BPF_LOOP(), a macro that brackets batch_iters iterations with bpf_ktime_get_ns() reads and records the elapsed time. One extra untimed iteration runs afterward for output validation.
The userspace side (benchs/bench_bpf_timing.c) collects samples from the skeleton BSS, computes percentile statistics, and auto-calibrates batch_iters to target ~10 ms per batch.
--- tools/testing/selftests/bpf/Makefile | 2 + .../testing/selftests/bpf/bench_bpf_timing.h | 50 ++++ .../selftests/bpf/benchs/bench_bpf_timing.c | 272 ++++++++++++++++++ .../bpf/progs/bench_bpf_timing.bpf.h | 69 +++++ 4 files changed, 393 insertions(+) create mode 100644 tools/testing/selftests/bpf/bench_bpf_timing.h create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_timing.c create mode 100644 tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h
tools/testing/selftests/bpf/Makefile +2 -0
@@ -906,6 +906,7 @@ $(OUTPUT)/bench_htab_mem.o: $(OUTPUT)/htab_mem_bench.skel.h $(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h $(OUTPUT)/bench_sockmap.o: $(OUTPUT)/bench_sockmap_prog.skel.h $(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_trie_map.skel.h +$(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm $(OUTPUT)/bench: $(OUTPUT)/bench.o \
@@ -928,6 +929,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_bpf_crypto.o \ $(OUTPUT)/bench_sockmap.o \ $(OUTPUT)/bench_lpm_trie_map.o \ + $(OUTPUT)/bench_bpf_timing.o \ $(OUTPUT)/usdt_1.o \ $(OUTPUT)/usdt_2.o \ #
tools/testing/selftests/bpf/bench_bpf_timing.h new file +50 -0
50 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +272 -0
272 lines not on this page · raw message
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +69 -0
69 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 2/7] selftests/bpf: Add BPF batch-timing library
Sashiko AI review found 4 potential issue(s): - [Medium] Integer division truncation in `bpf_bench_calibrate()` breaks auto-calibration for very fast BPF programs. - [Medium] Zero warmup time (`-w…
Sashiko AI review found 4 potential issue(s):
- [Medium] Integer division truncation in bpf_bench_calibrate() breaks auto-calibration for very fast BPF programs.
- [Medium] Zero warmup time (-w 0) causes the benchmark to silently collect no samples.
- [Medium] BPF can_loop (bpf_may_goto) can prematurely terminate the timed loop, silently invalidating the userspace ns/op calculation.
- [Low] The BPF header includes a non-existent file <bpf_may_goto.h>.
--
commit ecbc1ddd2ab895fa706970f55f150bb87ba2790e
Author: Puranjay Mohan <[email protected]>
selftests/bpf: Add BPF batch-timing library
This commit adds a reusable timing library for BPF benchmarks that need to measure BPF program execution time. It provides a BPF macro to record elapsed time and a userspace library to auto-calibrate batch iterations and compute statistics.
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +0
-0
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +6
-0+static int collect_samples (struct bpf_bench_timing *t, +double *out,int max_out) +{ +unsigned int nr_cpus =bpf_num_possible_cpus (); +__u32 timed_iters = t->batch_iters; +int total =0 ;
[ ... ]
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 60
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +6
-0+for (__u32 i =0 ; i < count && total < max_out; i++) { +__u64 sample = t->samples[cpu][i]; + +if (sample ==0 ) +continue ; + out[total++] = (double )sample / timed_iters;
If the loop in BENCH_BPF_LOOP is terminated early by the can_loop condition (e.g., due to instruction limits or yielding), the actual number of completed iterations will be less than timed_iters.
Since the BPF program only records the elapsed time, will dividing by timed_iters produce artificially low ns/op calculations?
Should the BPF program explicitly record and export the actual number of completed iterations for this calculation?
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +6
-0+ } + } + +qsort (out, total,sizeof (double ), cmp_double); +return total; +}
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +5
-0+void bpf_bench_timing_measure (struct bpf_bench_timing *t,struct bench_res *res) +{ +unsigned int nr_cpus; +__u32 total_samples; +int i;
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 110
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +10
-0+ + t->warmup_ticks++; + +if (t->warmup_ticks < env.warmup_sec) +return ; + +if (t->warmup_ticks == env.warmup_sec) { + *t->timing_enabled =1 ; +return ; + }
If the benchmark is run with zero warmup (env.warmup_sec == 0), t->warmup_ticks becomes 1 on the first tick.
Does this mean t->warmup_ticks == env.warmup_sec will never be true, bypassing the assignment and preventing timing_enabled from ever being set to 1?
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +6
-0+void bpf_bench_calibrate (struct bpf_bench_timing *t,bpf_bench_run_fn run_fn,void *run_ctx) +{ +__u64 elapsed, per_op_ns; +__u64 time_n, time_2n; +double ratio; +
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 253
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +10
-0+ elapsed =measure_elapsed (t, run_fn, run_ctx,CALIBRATE_SEED_BATCH ,CALIBRATE_RUNS ); +if (elapsed ==0 ) { +fprintf (stderr,"calibration: no timing samples, using default\n" ); + t->batch_iters =10000 ; + *t->batch_iters_bss = t->batch_iters; +reset_timing (t); +return ; + } + + per_op_ns = elapsed /CALIBRATE_SEED_BATCH ;
If the BPF operation executes in less than 1 ns (e.g., an empty loop or bpf-nop), elapsed could be less than 100 since CALIBRATE_SEED_BATCH is 100.
Will the integer division truncate per_op_ns to 0 here, causing compute_batch_iters() to incorrectly fall back to CALIBRATE_MIN_BATCH instead of scaling up the iterations?
Quoted 1 line
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +1
-0+ t->batch_iters =compute_batch_iters (per_op_ns);
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +0
-0
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h line 10
Quoted 10 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +10
-0@@ -0,0 +1,69 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#ifndef __BENCH_BPF_TIMING_BPF_H__ +#define __BENCH_BPF_TIMING_BPF_H__ + +#include <stdbool.h> +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include <bpf_may_goto.h>
Does bpf_may_goto.h exist in the kernel source tree?
I noticed that can_loop is typically defined in bpf_experimental.h. Will including this cause a No such file or directory build error?
Quoted 2 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +2
-0+ +#ifndef BENCH_NR_SAMPLES
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[email protected]
AI reviewRe: [PATCH bpf-next 2/7] selftests/bpf: Add BPF batch-timing library
In reply to Puranjay MohanWill this integer division truncation break calibration for operations taking less than 1 ns? If the total elapsed time for 100 iterations is under 100 ns, per_op_ns evaluates to 0, forcing…
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +0
-0
[ ... ]
Quoted 25 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +25
-0+static __u32 compute_batch_iters (__u64 per_op_ns) +{ +__u64 target_ns = (__u64 )CALIBRATE_TARGET_MS *1000000ULL ; +__u32 iters; + +if (per_op_ns ==0 ) +return CALIBRATE_MIN_BATCH ; + + iters = target_ns / per_op_ns; + +if (iters <CALIBRATE_MIN_BATCH ) + iters =CALIBRATE_MIN_BATCH ; +if (iters >CALIBRATE_MAX_BATCH ) + iters =CALIBRATE_MAX_BATCH ; + +return iters; +} + +void bpf_bench_calibrate (struct bpf_bench_timing *t,bpf_bench_run_fn run_fn,void *run_ctx) +{ +__u64 elapsed, per_op_ns; +__u64 time_n, time_2n; +double ratio; + + elapsed =measure_elapsed (t, run_fn, run_ctx,CALIBRATE_SEED_BATCH ,CALIBRATE_RUNS );
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 254
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +10
-0+if (elapsed ==0 ) { +fprintf (stderr,"calibration: no timing samples, using default\n" ); + t->batch_iters =10000 ; + *t->batch_iters_bss = t->batch_iters; +reset_timing (t); +return ; + } + + per_op_ns = elapsed /CALIBRATE_SEED_BATCH ; + t->batch_iters =compute_batch_iters (per_op_ns);
Will this integer division truncation break calibration for operations taking less than 1 ns? If the total elapsed time for 100 iterations is under 100 ns, per_op_ns evaluates to 0, forcing compute_batch_iters() to return CALIBRATE_MIN_BATCH (100) instead of scaling up to reach the 10 ms target.
Quoted 3 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +3
-0+ + time_n =measure_elapsed (t, run_fn, run_ctx, t->batch_iters,CALIBRATE_RUNS ); + time_2n =measure_elapsed (t, run_fn, run_ctx, t->batch_iters *2 ,CALIBRATE_RUNS );
[ ... ]
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 179
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +6
-0+static void reset_timing (struct bpf_bench_timing *t) +{ + *t->timing_enabled =0 ; +memset (t->samples,0 ,sizeof (__u64) *BENCH_NR_CPUS *BENCH_NR_SAMPLES ); +memset (t->idx,0 ,sizeof (__u32) *BENCH_NR_CPUS ); +}
Does this need CPU memory barriers to prevent hardware reordering? While timing_enabled is volatile, weakly-ordered architectures like arm64 might reorder the memory stores across the flag toggle. This could allow background BPF executions to write samples while the arrays are being zeroed.
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +0
-0
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +6
-0+static __always_inline void bench_record_sample(__u64 elapsed_ns) +{ +__u32 cpu, idx; + +if (!timing_enabled) +return ;
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h line 41
Quoted 10 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +10
-0+ + cpu =bpf_get_smp_processor_id () &BENCH_CPU_MASK ; + idx = timing_idx[cpu]; + +if (idx >=BENCH_NR_SAMPLES ) +return ; + + timing_samples[cpu][idx] = elapsed_ns; + timing_idx[cpu] = idx +1 ; +}
Can this read-modify-write pattern cause lost samples or out-of-bounds writes? If machines have more than 256 logical CPUs, BENCH_CPU_MASK will alias multiple CPUs to the same index. Concurrent thread executions could race and overwrite the index, potentially corrupting the timing statistics.
Quoted 16 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +16
-0+ +/* +* @body: expression to time; return value (int) stored in __bench_result. +* @reset: undo body's side-effects so each iteration starts identically. +* May reference __bench_result. Use ({}) for empty reset. +* +* Runs batch_iters timed iterations, then one untimed iteration whose +* return value the macro evaluates to (for validation). +*/ +#define BENCH_BPF_LOOP (body, reset) ({ \ + __u64 __bench_start = bpf_ktime_get_ns(); \ + __u32 __bench_i; \ + int __bench_result; \ + \ + for (__bench_i = 0; \ + __bench_i < batch_iters && can_loop; \
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h line 67
Quoted 10 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +10
-0+ __bench_i++) { \ + __bench_result = (body); \ + reset; \ + } \ + \ +bench_record_sample (bpf_ktime_get_ns () - __bench_start); \ + \ + __bench_result = (body); \ + __bench_result; \ +})
Should the final untimed iteration also evaluate the reset expression? Without calling reset here, any side effects from the last body execution (like memory allocations or map modifications) will leak into the next program invocation and could violate benchmark isolation.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
Re: [PATCH bpf-next 2/7] selftests/bpf: Add BPF batch-timing library
Quoted 41 lines · On Tue, Apr 28, 2026 at 1:18 AM <[email protected]> wrote:
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +0
-0[ ... ]
Quoted 32 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +32
-0+static __u32 compute_batch_iters (__u64 per_op_ns) +{ +__u64 target_ns = (__u64 )CALIBRATE_TARGET_MS *1000000ULL ; +__u32 iters; + +if (per_op_ns ==0 ) +return CALIBRATE_MIN_BATCH ; + + iters = target_ns / per_op_ns; + +if (iters <CALIBRATE_MIN_BATCH ) + iters =CALIBRATE_MIN_BATCH ; +if (iters >CALIBRATE_MAX_BATCH ) + iters =CALIBRATE_MAX_BATCH ; + +return iters; +} + +void bpf_bench_calibrate (struct bpf_bench_timing *t,bpf_bench_run_fn run_fn,void *run_ctx) +{ +__u64 elapsed, per_op_ns; +__u64 time_n, time_2n; +double ratio; + + elapsed =measure_elapsed (t, run_fn, run_ctx,CALIBRATE_SEED_BATCH ,CALIBRATE_RUNS ); +if (elapsed ==0 ) { +fprintf (stderr,"calibration: no timing samples, using default\n" ); + t->batch_iters =10000 ; + *t->batch_iters_bss = t->batch_iters; +reset_timing (t); +return ; + }
Quoted 8 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 254
Quoted 3 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +3
-0+ + per_op_ns = elapsed /CALIBRATE_SEED_BATCH ; + t->batch_iters =compute_batch_iters (per_op_ns);Will this integer division truncation break calibration for operations taking less than 1 ns? If the total elapsed time for 100 iterations is under 100 ns,
per_op_nsevaluates to 0, forcingcompute_batch_iters()to returnCALIBRATE_MIN_BATCH(100) instead of scaling up to reach the 10 ms target.
If individual operations take less than 1 ns then we can't meaninfully measure them, so returning 100 here is good enough.
Quoted 7 lines
Quoted 4 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +3
-0+ + time_n =measure_elapsed (t, run_fn, run_ctx, t->batch_iters,CALIBRATE_RUNS ); + time_2n =measure_elapsed (t, run_fn, run_ctx, t->batch_iters *2 ,CALIBRATE_RUNS );[ ... ]
Quoted 11 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c line 179
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_bpf_timing.c new file +6
-0+static void reset_timing (struct bpf_bench_timing *t) +{ + *t->timing_enabled =0 ; +memset (t->samples,0 ,sizeof (__u64) *BENCH_NR_CPUS *BENCH_NR_SAMPLES ); +memset (t->idx,0 ,sizeof (__u32) *BENCH_NR_CPUS ); +}Does this need CPU memory barriers to prevent hardware reordering? While
timing_enabledis volatile, weakly-ordered architectures like arm64 might reorder the memory stores across the flag toggle. This could allow background BPF executions to write samples while the arrays are being zeroed.
reset_timing() is only called during calibration, which runs in setup() — before any producer threads are started. There's no concurrent BPF execution to race with.
Quoted 23 lines
[ ... ]
Quoted 5 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +0
-0[ ... ]
Quoted 13 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +13
-0+static __always_inline void bench_record_sample(__u64 elapsed_ns) +{ +__u32 cpu, idx; + +if (!timing_enabled) +return ; + + cpu =bpf_get_smp_processor_id () &BENCH_CPU_MASK ; + idx = timing_idx[cpu]; + +if (idx >=BENCH_NR_SAMPLES ) +return ; +
Quoted 8 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h line 41
Quoted 3 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +3
-0+ timing_samples[cpu][idx] = elapsed_ns; + timing_idx[cpu] = idx +1 ; +}Can this read-modify-write pattern cause lost samples or out-of-bounds writes? If machines have more than 256 logical CPUs,
BENCH_CPU_MASKwill alias multiple CPUs to the same index. Concurrent thread executions could race and overwrite the index, potentially corrupting the timing statistics.
xdp_lb_validate() checks bpf_num_possible_cpus() > BENCH_NR_CPUS and exits with an error.
Quoted 24 lines
Quoted 24 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +23
-0+ +/* +* @body: expression to time; return value (int) stored in __bench_result. +* @reset: undo body's side-effects so each iteration starts identically. +* May reference __bench_result. Use ({}) for empty reset. +* +* Runs batch_iters timed iterations, then one untimed iteration whose +* return value the macro evaluates to (for validation). +*/ +#define BENCH_BPF_LOOP (body, reset) ({ \ + __u64 __bench_start = bpf_ktime_get_ns(); \ + __u32 __bench_i; \ + int __bench_result; \ + \ + for (__bench_i = 0; \ + __bench_i < batch_iters && can_loop; \ + __bench_i++) { \ + __bench_result = (body); \ + reset; \ + } \ + \ + bench_record_sample(bpf_ktime_get_ns() - __bench_start); \ + \
Quoted 8 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h line 67
Quoted 3 lines
tools/testing/selftests/bpf/progs/bench_bpf_timing.bpf.h new file +3
-0+ __bench_result = (body); \ + __bench_result; \ +})Should the final untimed iteration also evaluate the reset expression? Without calling reset here, any side effects from the last body execution (like memory allocations or map modifications) will leak into the next program invocation and could violate benchmark isolation.
This is by design, we want to have the last untimed iteration to have side effects so we can validate output.
Quoted 6 lines
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.mdCI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
PATCH 3/7 selftests/bpf: Add bpf-nop benchmark for timing overhead baseline
In reply to Puranjay MohanAdd a minimal benchmark that measures the overhead of the batch-timing infrastructure itself. The BPF program runs an empty BENCH_BPF_LOOP body (~1.5-2 ns/op), establishing the floor cost that all timing-library benchmarks include.
[root@virtme-ng tools/testing/selftests/bpf]# sudo ./bench -a -p8 bpf-nop Setting up benchmark 'bpf-nop'...
Benchmark 'bpf-nop' started.
bpf-nop: median 1.82 ns/op, stddev 0.01, p99 1.86 (1754 samples)
--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 2 + .../selftests/bpf/benchs/bench_bpf_nop.c | 84 +++++++++++++++++++ .../selftests/bpf/progs/bpf_nop_bench.c | 14 ++++ 4 files changed, 102 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_bpf_nop.c create mode 100644 tools/testing/selftests/bpf/progs/bpf_nop_bench.c
tools/testing/selftests/bpf/Makefile +2 -0
@@ -906,6 +906,7 @@ $(OUTPUT)/bench_htab_mem.o: $(OUTPUT)/htab_mem_bench.skel.h $(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h $(OUTPUT)/bench_sockmap.o: $(OUTPUT)/bench_sockmap_prog.skel.h $(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_trie_map.skel.h +$(OUTPUT)/bench_bpf_nop.o: $(OUTPUT)/bpf_nop_bench.skel.h bench_bpf_timing.h $(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm
@@ -930,6 +931,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_sockmap.o \ $(OUTPUT)/bench_lpm_trie_map.o \ $(OUTPUT)/bench_bpf_timing.o \ + $(OUTPUT)/bench_bpf_nop.o \ $(OUTPUT)/usdt_1.o \ $(OUTPUT)/usdt_2.o \ #
tools/testing/selftests/bpf/bench.c +2 -0
@@ -575,6 +575,7 @@ extern const struct bench bench_lpm_trie_insert;extern const struct bench bench_lpm_trie_update;extern const struct bench bench_lpm_trie_delete;extern const struct bench bench_lpm_trie_free; +extern const struct bench bench_bpf_nop;static const struct bench *benchs[] = { &bench_count_global,
@@ -653,6 +654,7 @@ static const struct bench *benchs[] = { &bench_lpm_trie_update, &bench_lpm_trie_delete, &bench_lpm_trie_free, + &bench_bpf_nop, };static void find_benchmark (void )
tools/testing/selftests/bpf/benchs/bench_bpf_nop.c new file +84 -0
84 lines not on this page · raw message
tools/testing/selftests/bpf/progs/bpf_nop_bench.c new file +14 -0
@@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bench_bpf_timing.bpf.h" + +SEC("syscall") +int bench_nop (void *ctx) +{ +return BENCH_BPF_LOOP (0 , ({})); +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.52.0
PATCH 4/7 selftests/bpf: Add XDP load-balancer common definitions
In reply to Puranjay MohanAdd the shared header for the XDP load-balancer benchmark. This defines the data structures used by both the BPF program and
userspace: flow_key, vip_definition, real_definition, and the stats/control structures.
Also provides the encapsulation source-address helpers shared between the BPF datapath (for encap) and userspace (for building expected output packets used in validation).
--- .../selftests/bpf/xdp_lb_bench_common.h | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 tools/testing/selftests/bpf/xdp_lb_bench_common.h
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +112 -0
@@ -0,0 +1,112 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#ifndef XDP_LB_BENCH_COMMON_H +#define XDP_LB_BENCH_COMMON_H + +#define F_IPV6 (1 << 0) +#define F_LRU_BYPASS (1 << 1) + +#define CH_RING_SIZE 65537/* per-VIP consistent hash ring slots */ +#define MAX_VIPS 16 +#define CH_RINGS_SIZE (MAX_VIPS * CH_RING_SIZE) +#define MAX_REALS 512 +#define DEFAULT_LRU_SIZE 100000/* connection tracking cache size */ +#define ONE_SEC 1000000000U/* 1 sec in nanosec */ +#define MAX_CONN_RATE 100000000/* high enough to never trigger in bench */ +#define LRU_UDP_TIMEOUT 30000000000ULL/* 30 sec in nanosec */ +#define PCKT_FRAGMENTED 0x3FFF +#define KNUTH_HASH_MULT 2654435761U +#define IPIP_V4_PREFIX 4268/* 172.16/12 in network order */ +#define IPIP_V6_PREFIX1 1/* 0100::/64 (RFC 6666 discard) */ +#define IPIP_V6_PREFIX2 0 +#define IPIP_V6_PREFIX3 0 + +/* Stats indices (0..MAX_VIPS-1 are per-VIP packet/byte counters) */ +#define STATS_LRU (MAX_VIPS + 0)/* v1: total VIP packets, v2: LRU misses */ +#define STATS_XDP_TX (MAX_VIPS + 1) +#define STATS_XDP_PASS (MAX_VIPS + 2) +#define STATS_XDP_DROP (MAX_VIPS + 3) +#define STATS_NEW_CONN (MAX_VIPS + 4)/* v1: conn count, v2: last reset ts */ +#define STATS_LRU_MISS (MAX_VIPS + 5)/* v1: TCP LRU misses */ +#define STATS_SIZE (MAX_VIPS + 6) + +#ifdef __BPF__ +#define lb_htons (x) bpf_htons(x) +#define LB_INLINE static __always_inline +#else +#define lb_htons (x) htons(x) +#define LB_INLINE static inline +#endif + +LB_INLINE __be32 create_encap_ipv4_src(__u16 port, __be32 src) +{ +__u32 ip_suffix =lb_htons (port); + + ip_suffix <<=16 ; + ip_suffix ^= src; +return (0xFFFF0000 & ip_suffix) |IPIP_V4_PREFIX ; +} + +LB_INLINE void create_encap_ipv6_src(__u16 port, __be32 src, __be32 *saddr) +{ + saddr[0 ] =IPIP_V6_PREFIX1 ; + saddr[1 ] =IPIP_V6_PREFIX2 ; + saddr[2 ] =IPIP_V6_PREFIX3 ; + saddr[3 ] = src ^ port; +} + +struct flow_key { +union { +__be32 src; +__be32 srcv6[4 ]; + }; +union { +__be32 dst; +__be32 dstv6[4 ]; + }; +union { +__u32 ports; +__u16 port16[2 ]; + }; +__u8 proto; +__u8 pad[3 ]; +}; + +struct vip_definition { +union { +__be32 vip; +__be32 vipv6[4 ]; + }; +__u16 port; +__u8 proto; +__u8 pad; +}; + +struct vip_meta { +__u32 flags; +__u32 vip_num; +}; + +struct real_pos_lru { +__u32 pos; +__u64 atime; +}; + +struct real_definition { +__be32 dst; +__be32 dstv6[4 ]; +__u8 flags; +}; + +struct lb_stats { +__u64 v1; +__u64 v2; +}; + +struct ctl_value { +__u8 mac[6 ]; +__u8 pad[2 ]; +}; + +#endif /* XDP_LB_BENCH_COMMON_H */
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 4/7] selftests/bpf: Add XDP load-balancer common definitions
Does this mix host-endian arithmetic with network-order types?
Quoted 11 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +5
-0@@ -0,0 +1,112 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#ifndef XDP_LB_BENCH_COMMON_H +#define XDP_LB_BENCH_COMMON_H
[ ... ]
Quoted 6 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +6
-0+#ifdef __BPF__ +#define lb_htons (x) bpf_htons(x) +#define LB_INLINE static __always_inline +#else +#define lb_htons (x) htons(x) +#define LB_INLINE static inline
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 49
Quoted 10 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +10
-0+#endif + +LB_INLINE __be32 create_encap_ipv4_src(__u16 port, __be32 src) +{ +__u32 ip_suffix =lb_htons (port); + + ip_suffix <<=16 ; + ip_suffix ^= src; +return (0xFFFF0000 & ip_suffix) |IPIP_V4_PREFIX ; +}
Does this mix host-endian arithmetic with network-order types?
Shifting and masking __be32 values using host operations modifies different IP octets depending on the architecture.
Also, on big-endian systems, will the host-order constant IPIP_V4_PREFIX (4268) map to 0.0.16.172 instead of 172.16.0.0?
This concern was raised in review of v1:
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 57
Quoted 8 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +8
-0+ +LB_INLINE void create_encap_ipv6_src(__u16 port, __be32 src, __be32 *saddr) +{ + saddr[0 ] =IPIP_V6_PREFIX1 ; + saddr[1 ] =IPIP_V6_PREFIX2 ; + saddr[2 ] =IPIP_V6_PREFIX3 ; + saddr[3 ] = src ^ port; +}
Are we mixing endianness here by applying a host-endian XOR operation between a __be32 source and a __u16 port?
Looking at the parallel IPv4 version above, it converts port with lb_htons(port) before the XOR operation. Shouldn't this be:
saddr[3] = src ^ lb_htons(port);
to match the IPv4 version's byte order handling?
On little-endian architectures, when port=0x1234, the XOR will use 0x3412 (byte-swapped during 32-bit promotion) instead of the intended value. This will cause packet validation failures and incorrect routing behavior when these functions are eventually used.
This concern was raised in review of v1:
Quoted 27 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +27
-0+ +struct flow_key { +union { +__be32 src; +__be32 srcv6[4 ]; + }; +union { +__be32 dst; +__be32 dstv6[4 ]; + }; +union { +__u32 ports; +__u16 port16[2 ]; + }; +__u8 proto; +__u8 pad[3 ]; +}; + +struct vip_definition { +union { +__be32 vip; +__be32 vipv6[4 ]; + }; +__u16 port; +__u8 proto; +__u8 pad; +};
[ ... ]
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 100
Quoted 5 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +5
-0+struct real_definition { +__be32 dst; +__be32 dstv6[4 ]; +__u8 flags; +};
Should dst and dstv6 be wrapped in an anonymous union?
The other dual-stack IP structures in this header (flow_key and vip_definition) use a union to save memory in BPF maps and maintain structural layout.
Also, does this structure need explicit trailing padding? There are currently 3 bytes of implicit trailing padding which might cause UAPI ABI mismatch issues between 32-bit and 64-bit architectures.
These concerns were raised in review of v1:
https://lore.kernel.org/bpf/[email protected]/
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
[email protected]
AI reviewRe: [PATCH bpf-next 4/7] selftests/bpf: Add XDP load-balancer common definitions
In reply to Puranjay MohanSashiko AI review found 3 potential issue(s): - [Medium] Endianness bug and invalid arithmetic on network-order values in IP encapsulation helpers. - [Medium] Implicit compiler padding in shared…
Sashiko AI review found 3 potential issue(s):
- [Medium] Endianness bug and invalid arithmetic on network-order values in IP encapsulation helpers.
- [Medium] Implicit compiler padding in shared BPF/userspace structures causes cross-architecture ABI breakage.
- [Low] Missing anonymous union for IPv4/IPv6 addresses in struct real_definition.
--
commit 27405cc11d849dd578c45222803d3bbd40b0e500
Author: Puranjay Mohan <[email protected]>
selftests/bpf: Add XDP load-balancer common definitions
This commit adds the shared header for the XDP load-balancer benchmark. It defines the data structures used by both the BPF program and userspace, along with encapsulation source-address helpers.
Quoted 5 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +0
-0
[ ... ]
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 49
Quoted 8 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +8
-0+LB_INLINE __be32 create_encap_ipv4_src(__u16 port, __be32 src) +{ +__u32 ip_suffix =lb_htons (port); + + ip_suffix <<=16 ; + ip_suffix ^= src; +return (0xFFFF0000 & ip_suffix) |IPIP_V4_PREFIX ; +}
Is it safe to perform bitwise operations directly on network-order values like this? Since lb_htons() returns a network-order port, left-shifting it by 16 and masking with host-order constants like 0xFFFF0000 or combining it with IPIP_V4_PREFIX (4268) seems to assume a little-endian architecture. Could this generate corrupted packet headers on big-endian systems?
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 57
Quoted 8 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +8
-0+ +LB_INLINE void create_encap_ipv6_src(__u16 port, __be32 src, __be32 *saddr) +{ + saddr[0 ] =IPIP_V6_PREFIX1 ; + saddr[1 ] =IPIP_V6_PREFIX2 ; + saddr[2 ] =IPIP_V6_PREFIX3 ; + saddr[3 ] = src ^ port; +}
Will saddr[0] evaluate to the correct prefix on all architectures? Because IPIP_V6_PREFIX1 is defined as 1, it evaluates to 0100:: on little-endian but ::1 on big-endian.
Also, is it intended to XOR the network-order __be32 src with the host-order __u16 port without a prior byte-order conversion?
[ ... ]
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 94
Quoted 4 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +4
-0+struct real_pos_lru { +__u32 pos; +__u64 atime; +};
Could this shared structure cause ABI breakage between 32-bit userspace and 64-bit BPF environments? In 64-bit environments, the compiler will insert 4 bytes of padding after pos, but 32-bit environments only require 4-byte alignment for __u64. Should an explicit padding field be added to ensure identical memory layout?
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 100
Quoted 6 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +6
-0+ +struct real_definition { +__be32 dst; +__be32 dstv6[4 ]; +__u8 flags; +};
This isn't a bug, but earlier structures like flow_key and vip_definition use an anonymous union to overlay their IPv4 and IPv6 addresses. Should dst and dstv6 be placed in a union here as well to save memory and maintain consistency?
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH bpf-next 4/7] selftests/bpf: Add XDP load-balancer common definitions
Quoted 24 lines · On Tue, Apr 28, 2026 at 1:38 AM <[email protected]> wrote:
Sashiko AI review found 3 potential issue(s):
- [Medium] Endianness bug and invalid arithmetic on network-order values in IP encapsulation helpers.
- [Medium] Implicit compiler padding in shared BPF/userspace structures causes cross-architecture ABI breakage.
- [Low] Missing anonymous union for IPv4/IPv6 addresses instruct real_definition.
--commit
27405cc11d849dd578c45222803d3bbd40b0e500
Author: Puranjay Mohan <[email protected]>selftests/bpf: Add XDP load-balancer common definitions
This commit adds the shared header for the XDP load-balancer benchmark. It defines the data structures used by both the BPF program and userspace, along with encapsulation source-address helpers.
Quoted 5 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +0
-0[ ... ]
Quoted 14 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 49
Quoted 8 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +8
-0+LB_INLINE __be32 create_encap_ipv4_src(__u16 port, __be32 src) +{ +__u32 ip_suffix =lb_htons (port); + + ip_suffix <<=16 ; + ip_suffix ^= src; +return (0xFFFF0000 & ip_suffix) |IPIP_V4_PREFIX ; +}Is it safe to perform bitwise operations directly on network-order values like this? Since
lb_htons()returns a network-order port, left-shifting it by 16 and masking with host-order constants like0xFFFF0000or combining it withIPIP_V4_PREFIX(4268) seems to assume a little-endian architecture. Could this generate corrupted packet headers on big-endian systems?
It is safe, everything happens on the same host, this is a benchmark!
Quoted 30 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 57
Quoted 9 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +8
-0+ +LB_INLINE void create_encap_ipv6_src(__u16 port, __be32 src, __be32 *saddr) +{ + saddr[0 ] =IPIP_V6_PREFIX1 ; + saddr[1 ] =IPIP_V6_PREFIX2 ; + saddr[2 ] =IPIP_V6_PREFIX3 ; + saddr[3 ] = src ^ port; +}Will saddr[0] evaluate to the correct prefix on all architectures? Because
IPIP_V6_PREFIX1is defined as 1, it evaluates to 0100:: on little-endian but ::1 on big-endian.Also, is it intended to XOR the network-order
__be32src with the host-order__u16port without a prior byte-order conversion?[ ... ]
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 94
Quoted 4 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +4
-0+struct real_pos_lru { +__u32 pos; +__u64 atime; +};Could this shared structure cause ABI breakage between 32-bit userspace and 64-bit BPF environments? In 64-bit environments, the compiler will insert 4 bytes of padding after pos, but 32-bit environments only require 4-byte alignment for
__u64. Should an explicit padding field be added to ensure identical memory layout?
Quoted 11 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h line 100
Quoted 6 lines
tools/testing/selftests/bpf/xdp_lb_bench_common.h new file +6
-0+ +struct real_definition { +__be32 dst; +__be32 dstv6[4 ]; +__u8 flags; +};This isn't a bug, but earlier structures like
flow_keyandvip_definitionuse an anonymous union to overlay their IPv4 and IPv6 addresses. Should dst and dstv6 be placed in a union here as well to save memory and maintain consistency?
No, some scenarios use both.
Quoted 5 lines
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH 5/7 selftests/bpf: Add XDP load-balancer BPF program
In reply to Puranjay MohanAdd the BPF datapath for the XDP load-balancer benchmark, a simplified L4 load-balancer inspired by katran.
The pipeline: L3/L4 parse -> VIP lookup -> per-CPU LRU connection table or consistent-hash fallback -> real server lookup -> per-VIP and per-real stats -> IPIP/IP6IP6 encapsulation. TCP SYN forces the consistent-hash path (skipping LRU); TCP RST skips LRU insert to avoid polluting the table.
process_packet() is marked __noinline so that the BENCH_BPF_LOOP reset block (which strips encapsulation) operates on valid packet pointers after bpf_xdp_adjust_head().
--- .../selftests/bpf/progs/xdp_lb_bench.c | 647 ++++++++++++++++++ 1 file changed, 647 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/xdp_lb_bench.c
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +647 -0
@@ -0,0 +1,647 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ + +#include <stddef.h> +#include <stdbool.h> +#include <linux/bpf.h> +#include <linux/if_ether.h> +#include <linux/ip.h> +#include <linux/ipv6.h> +#include <linux/in.h> +#include <linux/tcp.h> +#include <linux/udp.h> +#include <bpf/bpf_helpers.h> +#include <bpf/bpf_endian.h> +#include "bpf_compiler.h" +#include "xdp_lb_bench_common.h" +#include "bench_bpf_timing.bpf.h" + +#ifndef IPPROTO_FRAGMENT +#define IPPROTO_FRAGMENT 44 +#endif + +/* jhash helpers */ + +static inline __u32 rol32 (__u32 word,unsigned int shift) +{ +return (word << shift) | (word >> ((-shift) &31 )); +} + +#define __jhash_mix (a, b, c) \ +{ \ + a -= c; a ^= rol32(c, 4); c += b; \ + b -= a; b ^= rol32(a, 6); a += c; \ + c -= b; c ^= rol32(b, 8); b += a; \ + a -= c; a ^= rol32(c, 16); c += b; \ + b -= a; b ^= rol32(a, 19); a += c; \ + c -= b; c ^= rol32(b, 4); b += a; \ +} + +#define __jhash_final (a, b, c) \ +{ \ + c ^= b; c -= rol32(b, 14); \ + a ^= c; a -= rol32(c, 11); \ + b ^= a; b -= rol32(a, 25); \ + c ^= b; c -= rol32(b, 16); \ + a ^= c; a -= rol32(c, 4); \ + b ^= a; b -= rol32(a, 14); \ + c ^= b; c -= rol32(b, 24); \ +} + +#define JHASH_INITVAL 0xdeadbeef + +static inline __u32 __jhash_nwords (__u32 a,__u32 b,__u32 c,__u32 initval) +{ + a += initval; + b += initval; + c += initval; +__jhash_final (a, b, c); +return c; +} + +static inline __u32 jhash_2words (__u32 a,__u32 b,__u32 initval) +{ +return __jhash_nwords (a, b,0 , initval +JHASH_INITVAL + (2 <<2 )); +} + +static inline __u32 jhash2_4words (const __u32 *k,__u32 initval) +{ +__u32 a, b, c; + + a = b = c =JHASH_INITVAL + (4 <<2 ) + initval; + + a += k[0 ]; b += k[1 ]; c += k[2 ]; +__jhash_mix (a, b, c); + + a += k[3 ]; +__jhash_final (a, b, c); + +return c; +} + +static __always_inline void ipv4_csum(struct iphdr *iph) +{ +__u16 *next_iph = (__u16 *)iph; +__u32 csum =0 ; +int i; + +__pragma_loop_unroll_full + for (i = 0; i < (int)(sizeof(*iph) >> 1); i++) + csum += *next_iph++; + + csum = (csum &0xffff ) + (csum >>16 ); + csum = (csum &0xffff ) + (csum >>16 ); + iph->check = ~csum; +} + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 64); + __type(key, struct vip_definition); + __type(value, struct vip_meta); +} vip_map SEC(".maps"); + +struct lru_inner_map { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __type(key, struct flow_key); + __type(value, struct real_pos_lru); + __uint(max_entries, DEFAULT_LRU_SIZE); +} lru_inner SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS); + __type(key, __u32); + __type(value, __u32); + __uint(max_entries, BENCH_NR_CPUS); + __array(values, struct lru_inner_map); +} lru_mapping SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, CH_RINGS_SIZE); + __type(key, __u32); + __type(value, __u32); +} ch_rings SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, MAX_REALS); + __type(key, __u32); + __type(value, struct real_definition); +} reals SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, STATS_SIZE); + __type(key, __u32); + __type(value, struct lb_stats); +} stats SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, MAX_REALS); + __type(key, __u32); + __type(value, struct lb_stats); +} reals_stats SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, struct ctl_value); +} ctl_array SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, __u32); + __type(value, struct vip_definition); +} vip_miss_stats SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, MAX_REALS); + __type(key, __u32); + __type(value, __u32); +} lru_miss_stats SEC(".maps"); + +volatile __u32 flow_mask; +volatile __u32 cold_lru; +__u32 batch_gen; + +/* +* old_eth MUST be read BEFORE writing the outer header because +* bpf_xdp_adjust_head makes them overlap. +*/ +static __always_inline int encap_v4(struct xdp_md *xdp, __be32 saddr, __be32 daddr, +__u16 payload_len,const __u8 *dst_mac) +{ +struct ethhdr *new_eth, *old_eth; +void *data, *data_end; +struct iphdr *iph; + +if (bpf_xdp_adjust_head (xdp, -(int )sizeof (struct iphdr ))) +return -1 ; + + data = (void *)(long )xdp->data; + data_end = (void *)(long )xdp->data_end; + + new_eth = data; + iph = data +sizeof (struct ethhdr ); + old_eth = data +sizeof (struct iphdr ); + +if (new_eth +1 > data_end || old_eth +1 > data_end || iph +1 > data_end) +return -1 ; + +__builtin_memcpy (new_eth->h_source, old_eth->h_dest,sizeof (new_eth->h_source)); +__builtin_memcpy (new_eth->h_dest, dst_mac,sizeof (new_eth->h_dest)); + new_eth->h_proto =bpf_htons (ETH_P_IP ); + +__builtin_memset (iph,0 ,sizeof (*iph)); + iph->version =4 ; + iph->ihl =sizeof (*iph) >>2 ; + iph->protocol =IPPROTO_IPIP ; + iph->tot_len =bpf_htons (payload_len +sizeof (*iph)); + iph->ttl =64 ; + iph->saddr = saddr; + iph->daddr = daddr; +ipv4_csum (iph); + +return 0 ; +} + +static __always_inline int encap_v6(struct xdp_md *xdp, const __be32 saddr[4], +const __be32 daddr[4 ],__u8 nexthdr,__u16 payload_len, +const __u8 *dst_mac) +{ +struct ethhdr *new_eth, *old_eth; +void *data, *data_end; +struct ipv6hdr *ip6h; + +if (bpf_xdp_adjust_head (xdp, -(int )sizeof (struct ipv6hdr ))) +return -1 ; + + data = (void *)(long )xdp->data; + data_end = (void *)(long )xdp->data_end; + + new_eth = data; + ip6h = data +sizeof (struct ethhdr ); + old_eth = data +sizeof (struct ipv6hdr ); + +if (new_eth +1 > data_end || old_eth +1 > data_end || ip6h +1 > data_end) +return -1 ; + +__builtin_memcpy (new_eth->h_source, old_eth->h_dest,sizeof (new_eth->h_source)); +__builtin_memcpy (new_eth->h_dest, dst_mac,sizeof (new_eth->h_dest)); + new_eth->h_proto =bpf_htons (ETH_P_IPV6 ); + +__builtin_memset (ip6h,0 ,sizeof (*ip6h)); + ip6h->version =6 ; + ip6h->nexthdr = nexthdr; + ip6h->payload_len =bpf_htons (payload_len); + ip6h->hop_limit =64 ; +__builtin_memcpy (&ip6h->saddr, saddr,sizeof (ip6h->saddr)); +__builtin_memcpy (&ip6h->daddr, daddr,sizeof (ip6h->daddr)); + +return 0 ; +} + +static __always_inline void update_stats(void *map, __u32 key, __u16 bytes) +{ +struct lb_stats *st =bpf_map_lookup_elem (map, &key); + +if (st) { + st->v1 +=1 ; + st->v2 += bytes; + } +} + +static __always_inline void count_action(int action) +{ +struct lb_stats *st; +__u32 key; + +if (action ==XDP_TX ) + key =STATS_XDP_TX ; +else if (action ==XDP_PASS ) + key =STATS_XDP_PASS ; +else + key =STATS_XDP_DROP ; + + st =bpf_map_lookup_elem (&stats, &key); +if (st) + st->v1 +=1 ; +} + +static __always_inline bool is_under_flood(void) +{ +__u32 key =STATS_NEW_CONN ; +struct lb_stats *conn_st =bpf_map_lookup_elem (&stats, &key); +__u64 cur_time; + +if (!conn_st) +return true; + + cur_time =bpf_ktime_get_ns (); +if ((cur_time - conn_st->v2) >ONE_SEC ) { + conn_st->v1 =1 ; + conn_st->v2 = cur_time; + }else { + conn_st->v1 +=1 ; +if (conn_st->v1 >MAX_CONN_RATE ) +return true; + } +return false; +} + +static __always_inline struct real_definition *connection_table_lookup(void *lru_map, +struct flow_key *flow, +__u32 *out_pos) +{ +struct real_pos_lru *dst_lru; +struct real_definition *real; +__u32 key; + + dst_lru =bpf_map_lookup_elem (lru_map, flow); +if (!dst_lru) +return NULL ; + +/* UDP connections use atime-based timeout instead of FIN/RST */ +if (flow->proto ==IPPROTO_UDP ) { +__u64 cur_time =bpf_ktime_get_ns (); + +if (cur_time - dst_lru->atime >LRU_UDP_TIMEOUT ) +return NULL ; + dst_lru->atime = cur_time; + } + + key = dst_lru->pos; + *out_pos = key; + real =bpf_map_lookup_elem (&reals, &key); +return real; +} + +static __always_inline bool get_packet_dst(struct real_definition **real, struct flow_key *flow, +struct vip_meta *vip_info,bool is_v6,void *lru_map, +bool is_rst,__u32 *out_pos) +{ +bool under_flood; +__u32 hash, ch_key; +__u32 *ch_val; +__u32 real_pos; + + under_flood =is_under_flood (); + +if (is_v6) { +__u32 src_hash =jhash2_4words ((__u32 *)flow->srcv6,MAX_VIPS ); + + hash =jhash_2words (src_hash, flow->ports,CH_RING_SIZE ); + }else { + hash =jhash_2words (flow->src, flow->ports,CH_RING_SIZE ); + } + + ch_key =CH_RING_SIZE * vip_info->vip_num + hash %CH_RING_SIZE ; + ch_val =bpf_map_lookup_elem (&ch_rings, &ch_key); +if (!ch_val) +return false; + real_pos = *ch_val; + + *real =bpf_map_lookup_elem (&reals, &real_pos); +if (!(*real)) +return false; + +if (!(vip_info->flags &F_LRU_BYPASS ) && !under_flood && !is_rst) { +struct real_pos_lru new_lru = { .pos = real_pos }; + +if (flow->proto ==IPPROTO_UDP ) + new_lru.atime =bpf_ktime_get_ns (); +bpf_map_update_elem (lru_map, flow, &new_lru,BPF_ANY ); + } + + *out_pos = real_pos; +return true; +} + +static __always_inline void update_vip_lru_miss_stats(struct vip_definition *vip, bool is_v6, +__u32 real_idx) +{ +struct vip_definition *miss_vip; +__u32 key =0 ; +__u32 *cnt; + + miss_vip =bpf_map_lookup_elem (&vip_miss_stats, &key); +if (!miss_vip) +return ; + +if (is_v6) { +if (miss_vip->vipv6[0 ] != vip->vipv6[0 ] || miss_vip->vipv6[1 ] != vip->vipv6[1 ] || + miss_vip->vipv6[2 ] != vip->vipv6[2 ] || miss_vip->vipv6[3 ] != vip->vipv6[3 ]) +return ; + }else { +if (miss_vip->vip != vip->vip) +return ; + } + +if (miss_vip->port != vip->port || miss_vip->proto != vip->proto) +return ; + + cnt =bpf_map_lookup_elem (&lru_miss_stats, &real_idx); +if (cnt) + *cnt +=1 ; +} + +static __noinline int process_packet(struct xdp_md *xdp) +{ +void *data = (void *)(long )xdp->data; +void *data_end = (void *)(long )xdp->data_end; +struct ethhdr *eth = data; +struct real_definition *dst =NULL ; +struct vip_definition vip_def = {}; +struct ctl_value *cval; +struct flow_key flow = {}; +struct vip_meta *vip_info; +struct lb_stats *data_stats; +struct udphdr *uh; +__be32 tnl_src[4 ]; +void *lru_map; +void *l4; +__u16 payload_len; +__u32 real_pos =0 , cpu_num, key; +__u8 proto; +int action =XDP_DROP ; +bool is_v6, is_syn = false, is_rst = false; + +if (eth +1 > data_end) + goto out; + +if (eth->h_proto ==bpf_htons (ETH_P_IPV6 )) { + is_v6 = true; + }else if (eth->h_proto ==bpf_htons (ETH_P_IP )) { + is_v6 = false; + }else { + action =XDP_PASS ; + goto out; + } + +if (is_v6) { +struct ipv6hdr *ip6h = (void *)(eth +1 ); + +if (ip6h +1 > data_end) + goto out; +if (ip6h->nexthdr ==IPPROTO_FRAGMENT ) + goto out; + + payload_len =sizeof (struct ipv6hdr ) +bpf_ntohs (ip6h->payload_len); + proto = ip6h->nexthdr; + +__builtin_memcpy (flow.srcv6, &ip6h->saddr,sizeof (flow.srcv6)); +__builtin_memcpy (flow.dstv6, &ip6h->daddr,sizeof (flow.dstv6)); +__builtin_memcpy (vip_def.vipv6, &ip6h->daddr,sizeof (vip_def.vipv6)); + l4 = (void *)(ip6h +1 ); + }else { +struct iphdr *iph = (void *)(eth +1 ); + +if (iph +1 > data_end) + goto out; +if (iph->ihl !=5 ) + goto out; +if (iph->frag_off &bpf_htons (PCKT_FRAGMENTED )) + goto out; + + payload_len =bpf_ntohs (iph->tot_len); + proto = iph->protocol; + + flow.src = iph->saddr; + flow.dst = iph->daddr; + vip_def.vip = iph->daddr; + l4 = (void *)(iph +1 ); + } + +/* TCP and UDP share the same port layout at offset 0 */ +if (proto !=IPPROTO_TCP && proto !=IPPROTO_UDP ) { + action =XDP_PASS ; + goto out; + } + + uh = l4; +if ((void *)(uh +1 ) > data_end) + goto out; + flow.port16[0 ] = uh->source; + flow.port16[1 ] = uh->dest; + +if (proto ==IPPROTO_TCP ) { +struct tcphdr *th = l4; + +if ((void *)(th +1 ) > data_end) + goto out; + is_syn = th->syn; + is_rst = th->rst; + } + + flow.proto = proto; + vip_def.port = flow.port16[1 ]; + vip_def.proto = proto; + + vip_info =bpf_map_lookup_elem (&vip_map, &vip_def); +if (!vip_info) { + action =XDP_PASS ; + goto out; + } + + key =STATS_LRU ; + data_stats =bpf_map_lookup_elem (&stats, &key); +if (!data_stats) + goto out; + data_stats->v1 +=1 ; + + cpu_num =bpf_get_smp_processor_id (); + lru_map =bpf_map_lookup_elem (&lru_mapping, &cpu_num); +if (!lru_map) + goto out; + +if (!(vip_info->flags &F_LRU_BYPASS ) && !is_syn) + dst =connection_table_lookup (lru_map, &flow, &real_pos); + +if (!dst) { +if (flow.proto ==IPPROTO_TCP ) { +struct lb_stats *miss_st; + + key =STATS_LRU_MISS ; + miss_st =bpf_map_lookup_elem (&stats, &key); +if (miss_st) + miss_st->v1 +=1 ; + } + +if (!get_packet_dst (&dst, &flow, vip_info, is_v6, lru_map, is_rst, &real_pos)) + goto out; + +update_vip_lru_miss_stats (&vip_def, is_v6, real_pos); + data_stats->v2 +=1 ; + } + + key =0 ; + cval =bpf_map_lookup_elem (&ctl_array, &key); +if (!cval) + goto out; + +update_stats (&stats, vip_info->vip_num, payload_len); +update_stats (&reals_stats, real_pos, payload_len); + +if (is_v6) { +create_encap_ipv6_src (flow.port16[0 ], flow.srcv6[0 ], tnl_src); +if (encap_v6 (xdp, tnl_src, dst->dstv6,IPPROTO_IPV6 , payload_len, cval->mac)) + goto out; + }else if (dst->flags &F_IPV6 ) { +create_encap_ipv6_src (flow.port16[0 ], flow.src, tnl_src); +if (encap_v6 (xdp, tnl_src, dst->dstv6,IPPROTO_IPIP , payload_len, cval->mac)) + goto out; + }else { +if (encap_v4 (xdp,create_encap_ipv4_src (flow.port16[0 ], flow.src), dst->dst, + payload_len, cval->mac)) + goto out; + } + + action =XDP_TX ; + +out: +count_action (action); +return action; +} + +static __always_inline int strip_encap(struct xdp_md *xdp, const struct ethhdr *saved_eth) +{ +void *data = (void *)(long )xdp->data; +void *data_end = (void *)(long )xdp->data_end; +struct ethhdr *eth = data; +int hdr_sz; + +if (eth +1 > data_end) +return -1 ; + + hdr_sz = (eth->h_proto ==bpf_htons (ETH_P_IPV6 )) ? (int )sizeof (struct ipv6hdr ) + : (int )sizeof (struct iphdr ); + +if (bpf_xdp_adjust_head (xdp, hdr_sz)) +return -1 ; + + data = (void *)(long )xdp->data; + data_end = (void *)(long )xdp->data_end; + eth = data; + +if (eth +1 > data_end) +return -1 ; + +__builtin_memcpy (eth, saved_eth,sizeof (*saved_eth)); +return 0 ; +} + +static __always_inline void randomize_src(struct xdp_md *xdp, int saddr_off, __u32 *rand_state) +{ +void *data = (void *)(long )xdp->data; +void *data_end = (void *)(long )xdp->data_end; +__u32 *saddr = data + saddr_off; + + *rand_state ^= *rand_state <<13 ; + *rand_state ^= *rand_state >>17 ; + *rand_state ^= *rand_state <<5 ; + +if ((void *)(saddr +1 ) <= data_end) + *saddr = *rand_state & flow_mask; +} + +SEC("xdp") +int xdp_lb_bench (struct xdp_md *xdp) +{ +void *data = (void *)(long )xdp->data; +void *data_end = (void *)(long )xdp->data_end; +struct ethhdr *eth = data; +struct ethhdr saved_eth; +__u32 rand_state =0 ; +__u32 batch_hash =0 ; +int saddr_off =0 ; +bool is_v6; + +if (eth +1 > data_end) +return XDP_DROP ; + +__builtin_memcpy (&saved_eth, eth,sizeof (saved_eth)); + + is_v6 = (saved_eth.h_proto ==bpf_htons (ETH_P_IPV6 )); + + saddr_off =sizeof (struct ethhdr ) + (is_v6 ? offsetof(struct ipv6hdr , saddr) : + offsetof(struct iphdr , saddr)); + +if (flow_mask) + rand_state =bpf_get_prandom_u32 () |1 ; + +if (cold_lru) { +__u32 *saddr = data + saddr_off; + + batch_gen++; + batch_hash = (batch_gen ^bpf_get_smp_processor_id ()) *KNUTH_HASH_MULT ; +if ((void *)(saddr +1 ) <= data_end) + *saddr ^= batch_hash; + } + +return BENCH_BPF_LOOP ( +process_packet (xdp), + ({ +if (__bench_result ==XDP_TX ) { +if (strip_encap (xdp, &saved_eth)) +return XDP_DROP ; +if (rand_state) +randomize_src (xdp, saddr_off, &rand_state); + } +if (cold_lru) { +void *d = (void *)(long )xdp->data; +void *de = (void *)(long )xdp->data_end; +__u32 *__sa = d + saddr_off; + +if ((void *)(__sa +1 ) <= de) + *__sa ^= batch_hash; + } + }) + ); +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 5/7] selftests/bpf: Add XDP load-balancer BPF program
Does the cold_lru simulation actually generate the intended cache misses?
Quoted 6 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +0
-0@@ -0,0 +1,647 @@
[ ... ]
Quoted 40 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +40
-0+SEC("xdp") +int xdp_lb_bench (struct xdp_md *xdp) +{ +void *data = (void *)(long )xdp->data; +void *data_end = (void *)(long )xdp->data_end; +struct ethhdr *eth = data; +struct ethhdr saved_eth; +__u32 rand_state =0 ; +__u32 batch_hash =0 ; +int saddr_off =0 ; +bool is_v6; + +if (eth +1 > data_end) +return XDP_DROP ; + +__builtin_memcpy (&saved_eth, eth,sizeof (saved_eth)); + + is_v6 = (saved_eth.h_proto ==bpf_htons (ETH_P_IPV6 )); + + saddr_off =sizeof (struct ethhdr ) + (is_v6 ? offsetof(struct ipv6hdr , saddr) : + offsetof(struct iphdr , saddr)); + +if (flow_mask) + rand_state =bpf_get_prandom_u32 () |1 ; + +if (cold_lru) { +__u32 *saddr = data + saddr_off; + + batch_gen++; + batch_hash = (batch_gen ^bpf_get_smp_processor_id ()) *KNUTH_HASH_MULT ; +if ((void *)(saddr +1 ) <= data_end) + *saddr ^= batch_hash; + } + + return BENCH_BPF_LOOP( + process_packet(xdp), + ({ + if (__bench_result == XDP_TX) { + if (strip_encap(xdp, &saved_eth)) + return XDP_DROP;
tools/testing/selftests/bpf/progs/xdp_lb_bench.c line 641
Quoted 10 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +10
-0+if (rand_state) +randomize_src (xdp, saddr_off, &rand_state); + } +if (cold_lru) { +void *d = (void *)(long )xdp->data; +void *de = (void *)(long )xdp->data_end; +__u32 *__sa = d + saddr_off; + +if ((void *)(__sa +1 ) <= de) + *__sa ^= batch_hash;
^^^^
Does the cold_lru simulation actually generate the intended cache misses?
Inside BENCH_BPF_LOOP, the source IP address is XORed with the constant batch_hash value on every iteration. Since batch_hash does not change within the loop, this XOR operation will toggle the IP address between only two unique values.
If the LRU cache is sized to hold both IP addresses, after the first two iterations the system will continuously hit cached entries rather than experiencing the cache misses that the cold_lru simulation is intended to produce.
Would using a different random or incrementing value for each iteration ensure a continuous stream of unique addresses?
This concern was raised by sashiko-bot in the RFC v1 patch series but remains unaddressed in v2.
tools/testing/selftests/bpf/progs/xdp_lb_bench.c line 647
Quoted 6 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +6
-0+ } + }) + ); +} + +char _license[] SEC("license") = "GPL";
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
[email protected]
AI reviewRe: [PATCH bpf-next 5/7] selftests/bpf: Add XDP load-balancer BPF program
In reply to Puranjay MohanSashiko AI review found 1 potential issue(s): - [Medium] The `cold_lru` logic inside the benchmark loop's `reset` block toggles the IP source address back and forth, breaking the benchmark's LRU…
Sashiko AI review found 1 potential issue(s):
- [Medium] The cold_lru logic inside the benchmark loop's reset block toggles the IP source address back and forth, breaking the benchmark's LRU cache miss measurement. --
commit 295ed705f4fdd57c01615b43803c47313840bbf1
Author: Puranjay Mohan <[email protected]>
selftests/bpf: Add XDP load-balancer BPF program
Add the BPF datapath for the XDP load-balancer benchmark, a simplified L4 load-balancer inspired by katran.
Quoted 5 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +0
-0
[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +3
-0+SEC("xdp") +int xdp_lb_bench (struct xdp_md *xdp) +{
[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +15
-0+if (cold_lru) { +__u32 *saddr = data + saddr_off; + + batch_gen++; + batch_hash = (batch_gen ^bpf_get_smp_processor_id ()) *KNUTH_HASH_MULT ; +if ((void *)(saddr +1 ) <= data_end) + *saddr ^= batch_hash; + } + + return BENCH_BPF_LOOP( + process_packet(xdp), + ({ + if (__bench_result == XDP_TX) { + if (strip_encap(xdp, &saved_eth)) + return XDP_DROP;
tools/testing/selftests/bpf/progs/xdp_lb_bench.c line 641
Quoted 10 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +10
-0+if (rand_state) +randomize_src (xdp, saddr_off, &rand_state); + } +if (cold_lru) { +void *d = (void *)(long )xdp->data; +void *de = (void *)(long )xdp->data_end; +__u32 *__sa = d + saddr_off; + +if ((void *)(__sa +1 ) <= de) + *__sa ^= batch_hash;
Does this code toggle the source IP back and forth on alternating iterations?
Because batch_hash is computed outside the loop and remains constant, XORing the source IP on every reset block will cause it to alternate between two values.
Won't this allow the LRU cache to easily cache both addresses, leading to a near 100% cache hit rate and defeating the purpose of measuring cold_lru cache misses?
Quoted 4 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +4
-0+ } + }) + ); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Re: [PATCH bpf-next 5/7] selftests/bpf: Add XDP load-balancer BPF program
Quoted 51 lines · On Tue, Apr 28, 2026 at 2:05 AM <[email protected]> wrote:
Sashiko AI review found 1 potential issue(s):
- [Medium] Thecold_lrulogic inside the benchmark loop'sresetblock toggles the IP source address back and forth, breaking the benchmark's LRU cache miss measurement. --commit
295ed705f4fdd57c01615b43803c47313840bbf1
Author: Puranjay Mohan <[email protected]>selftests/bpf: Add XDP load-balancer BPF program
Add the BPF datapath for the XDP load-balancer benchmark, a simplified L4 load-balancer inspired by katran.
Quoted 5 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +0
-0[ ... ]
Quoted 3 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +3
-0+SEC("xdp") +int xdp_lb_bench (struct xdp_md *xdp) +{[ ... ]
Quoted 15 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +15
-0+if (cold_lru) { +__u32 *saddr = data + saddr_off; + + batch_gen++; + batch_hash = (batch_gen ^bpf_get_smp_processor_id ()) *KNUTH_HASH_MULT ; +if ((void *)(saddr +1 ) <= data_end) + *saddr ^= batch_hash; + } + + return BENCH_BPF_LOOP( + process_packet(xdp), + ({ + if (__bench_result == XDP_TX) { + if (strip_encap(xdp, &saved_eth)) + return XDP_DROP;tools/testing/selftests/bpf/progs/xdp_lb_bench.c line 641
Quoted 10 lines
tools/testing/selftests/bpf/progs/xdp_lb_bench.c new file +10
-0+if (rand_state) +randomize_src (xdp, saddr_off, &rand_state); + } +if (cold_lru) { +void *d = (void *)(long )xdp->data; +void *de = (void *)(long )xdp->data_end; +__u32 *__sa = d + saddr_off; + +if ((void *)(__sa +1 ) <= de) + *__sa ^= batch_hash;Does this code toggle the source IP back and forth on alternating iterations?
Quoted 7 lines
Because
batch_hashis computed outside the loop and remains constant, XORing the source IP on every reset block will cause it to alternate between two values.Won't this allow the LRU cache to easily cache both addresses, leading to a near 100% cache hit rate and defeating the purpose of measuring
cold_lrucache misses?
All cold_lru scenarios also have flow_mask set, which means randomize_src() runs in the reset block on every iteration, producing a different source IP each time via an xorshift PRNG. The cold_lru XOR is an additional per-batch perturbation, not the sole source of flow diversity. With flow_mask = 0xFFFFFF (16M flow space) and a 100K LRU, most iterations genuinely miss.
PATCH 6/7 selftests/bpf: Add XDP load-balancer benchmark driver
In reply to Puranjay MohanWire up the userspace side of the XDP load-balancer benchmark.
24 scenarios cover the full code-path matrix: TCP/UDP, IPv4/IPv6, cross-AF encap, LRU hit/miss/diverse/cold, consistent-hash bypass, SYN/RST flag handling, and early exits (unknown VIP, non-IP, ICMP, fragments, IP options).
Before benchmarking each scenario validates correctness: the output packet is compared byte-for-byte against a pre-built expected packet and BPF map counters are checked against the expected values.
Usage:
sudo ./bench -a -w3 -p1 xdp-lb --scenario tcp-v4-lru-hit sudo ./bench xdp-lb --list-scenarios
--- tools/testing/selftests/bpf/Makefile | 2 + tools/testing/selftests/bpf/bench.c | 4 + .../selftests/bpf/benchs/bench_xdp_lb.c | 1113 +++++++++++++++++ 3 files changed, 1119 insertions(+) create mode 100644 tools/testing/selftests/bpf/benchs/bench_xdp_lb.c
tools/testing/selftests/bpf/Makefile +2 -0
@@ -907,6 +907,7 @@ $(OUTPUT)/bench_bpf_crypto.o: $(OUTPUT)/crypto_bench.skel.h $(OUTPUT)/bench_sockmap.o: $(OUTPUT)/bench_sockmap_prog.skel.h $(OUTPUT)/bench_lpm_trie_map.o: $(OUTPUT)/lpm_trie_bench.skel.h $(OUTPUT)/lpm_trie_map.skel.h $(OUTPUT)/bench_bpf_nop.o: $(OUTPUT)/bpf_nop_bench.skel.h bench_bpf_timing.h +$(OUTPUT)/bench_xdp_lb.o: $(OUTPUT)/xdp_lb_bench.skel.h bench_bpf_timing.h $(OUTPUT)/bench_bpf_timing.o: bench_bpf_timing.h $(OUTPUT)/bench.o: bench.h testing_helpers.h $(BPFOBJ) $(OUTPUT)/bench: LDLIBS += -lm
@@ -932,6 +933,7 @@ $(OUTPUT)/bench: $(OUTPUT)/bench.o \ $(OUTPUT)/bench_lpm_trie_map.o \ $(OUTPUT)/bench_bpf_timing.o \ $(OUTPUT)/bench_bpf_nop.o \ + $(OUTPUT)/bench_xdp_lb.o \ $(OUTPUT)/usdt_1.o \ $(OUTPUT)/usdt_2.o \ #
tools/testing/selftests/bpf/bench.c +4 -0
@@ -286,6 +286,7 @@ extern struct argp bench_trigger_batch_argp;extern struct argp bench_crypto_argp;extern struct argp bench_sockmap_argp;extern struct argp bench_lpm_trie_map_argp; +extern struct argp bench_xdp_lb_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },
@@ -302,6 +303,7 @@ static const struct argp_child bench_parsers[] = { { &bench_crypto_argp, 0, "bpf crypto benchmark", 0 }, { &bench_sockmap_argp, 0, "bpf sockmap benchmark", 0 }, { &bench_lpm_trie_map_argp, 0, "LPM trie map benchmark", 0 }, + { &bench_xdp_lb_argp, 0, "XDP load-balancer benchmark", 0 }, {}, };
@@ -576,6 +578,7 @@ extern const struct bench bench_lpm_trie_update;extern const struct bench bench_lpm_trie_delete;extern const struct bench bench_lpm_trie_free;extern const struct bench bench_bpf_nop; +extern const struct bench bench_xdp_lb;static const struct bench *benchs[] = { &bench_count_global,
@@ -655,6 +658,7 @@ static const struct bench *benchs[] = { &bench_lpm_trie_delete, &bench_lpm_trie_free, &bench_bpf_nop, + &bench_xdp_lb, };static void find_benchmark (void )
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +1113 -0
1,113 lines not on this page · raw message
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 6/7] selftests/bpf: Add XDP load-balancer benchmark driver
Does the IP header checksum calculation exclude the IP options when set_ip_options is true? Since ip_checksum(&iph, sizeof(iph)) only computes over the 20-byte stack structure iph, and the 4 bytes…
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +0
-0
[ ... ]
Quoted 7 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +7
-0+static void build_packet (int idx) +{ +const struct test_scenario *sc = &scenarios[idx]; +__u8 *p = pkt_buf[idx]; +struct ethhdr eth = {}; +__u16 proto; +__u32 off =0 ;
[ ... ]
Quoted 19 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +19
-0+ }else { +struct iphdr iph = {}; +__u32 ip_off = off; + + iph.version =4 ; + iph.ihl = sc->set_ip_options ?6 :5 ; + iph.ttl =64 ; + iph.protocol = sc->ip_proto; + iph.saddr =htonl (sc->src_addr); + iph.daddr =htonl (sc->vip_addr); + iph.frag_off = sc->set_frag ?htons (IP_MF ) :0 ; + off +=sizeof (iph); + +if (sc->set_ip_options) { +/* NOP option padding (4 bytes = 1 word) */ +__u32 nop =htonl (0x01010101 ); + +memcpy (p + off, &nop,sizeof (nop)); + off +=sizeof (nop);
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c line 502
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +10
-0+ } + +build_l4 (sc, p, &off); + +memcpy (p + off,"bench___payload!" ,16 ); + off +=16 ; + + iph.tot_len =htons (off - ip_off); + iph.check =ip_checksum (&iph,sizeof (iph)); +memcpy (p + ip_off, &iph,sizeof (iph));
Does the IP header checksum calculation exclude the IP options when set_ip_options is true? Since ip_checksum(&iph, sizeof(iph)) only computes over the 20-byte stack structure iph, and the 4 bytes of NOP options are added separately to the packet buffer after the base header, those option bytes are not covered by the checksum.
Could this cause the BPF program to see an invalid checksum and drop the packet prematurely instead of processing the intended options logic?
This concern was raised by [email protected] in the RFC v1 thread
https://lore.kernel.org/bpf/[email protected]/
and does not appear to have been addressed.
Quoted 4 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +4
-0+ } + + pkt_len[idx] = off; +}
[ ... ]
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c line 570
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +5
-0+static void populate_lru (const struct test_scenario *sc,__u32 real_idx) +{ +struct real_pos_lru lru = { .pos = real_idx }; +struct flow_key fk; +int i, err;
When atime is not explicitly initialized, does this leave it at zero? If the machine running the selftest has been up for more than 30 seconds (LRU_UDP_TIMEOUT), would lookups against this pre-populated entry see an age of bpf_ktime_get_ns() - 0 and immediately evict it?
This could lead to unintended LRU misses in UDP scenarios that expect LRU hits (like S_UDP_V4_LRU_HIT, S_UDP_V6_LRU_HIT) and cause validation failures.
This concern was raised by [email protected] in the RFC v1 thread
https://lore.kernel.org/bpf/[email protected]/
and does not appear to have been addressed.
Quoted 15 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +15
-0+ +build_flow_key (&fk, sc); + +/* Insert into every per-CPU inner LRU so the entry is found +* regardless of which CPU runs the BPF program. +*/ +for (i =0 ; i < nr_inner_maps; i++) { + err =bpf_map_update_elem (lru_inner_fds[i], &fk, &lru,BPF_ANY ); +if (err) { +fprintf (stderr,"lru_inner[%d] [%s]: %s\n" , i, sc->name, +strerror (errno)); +exit (1 ); + } + } +}
[ ... ]
Quoted 9 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +9
-0+static void build_expected_packet (int idx) +{ +const struct test_scenario *sc = &scenarios[idx]; +__u8 *p = expected_buf[idx]; +struct ethhdr eth = {}; +const __u8 *in = pkt_buf[idx]; +__u32 in_len = pkt_len[idx]; +__u32 off =0 ; +__u32 inner_len = in_len -sizeof (struct ethhdr );
[ ... ]
Quoted 23 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +23
-0+if (sc->encap_v6_outer) { +struct ipv6hdr ip6h = {}; +__u8 nexthdr = sc->is_v6 ?IPPROTO_IPV6 :IPPROTO_IPIP ; + + ip6h.version =6 ; + ip6h.nexthdr = nexthdr; + ip6h.payload_len =htons (inner_len); + ip6h.hop_limit =64 ; + +create_encap_ipv6_src (htons (sc->src_port), + sc->is_v6 ?htonl (sc->src_addr_v6[0 ]) + :htonl (sc->src_addr), + (__be32 *)&ip6h.saddr); +htonl_v6 ((__be32 *)&ip6h.daddr, sc->tunnel_dst_v6); + +memcpy (p + off, &ip6h,sizeof (ip6h)); + off +=sizeof (ip6h); + }else { +struct iphdr iph = {}; + + iph.version =4 ; + iph.ihl =sizeof (iph) >>2 ; + iph.protocol =IPPROTO_IPIP ;
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c line 724
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +10
-0+ iph.tot_len =htons (inner_len +sizeof (iph)); + iph.ttl =64 ; + iph.saddr =create_encap_ipv4_src (htons (sc->src_port), +htonl (sc->src_addr)); + iph.daddr =htonl (sc->tunnel_dst); + iph.check =ip_checksum (&iph,sizeof (iph)); + +memcpy (p + off, &iph,sizeof (iph)); + off +=sizeof (iph); + }
Are the port values being double byte-swapped here? The code passes htons(sc->src_port) into create_encap_ipv6_src() and create_encap_ipv4_src(), but looking at xdp_lb_bench_common.h, those functions internally call lb_htons(port) again. On little-endian architectures, would this cause the port to be swapped twice, ending up in host order instead of network order?
Additionally, the encapsulation helpers use bitwise shifts and XORs on host integers (like IPIP_V4_PREFIX) directly into network-order types. On big-endian architectures, could this place the prefix in the wrong bytes, altering the IP address semantics and causing validation failures?
This concern was raised by [email protected] in the RFC v1 thread
https://lore.kernel.org/bpf/[email protected]/
and does not appear to have been addressed.
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c line 730
Quoted 6 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +6
-0+ +memcpy (p + off, in +sizeof (struct ethhdr ), inner_len); + off += inner_len; + + expected_len[idx] = off; +}
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/25025533577
[email protected]
AI reviewRe: [PATCH bpf-next 6/7] selftests/bpf: Add XDP load-balancer benchmark driver
In reply to Puranjay MohanSashiko AI review found 1 potential issue(s): - [Low] Incorrect IPv4 checksum calculation for packets with IP options in test packet generator --
Sashiko AI review found 1 potential issue(s):
- [Low] Incorrect IPv4 checksum calculation for packets with IP options in test packet generator --
commit cc1cb16090fdc833cb82a1c6a168d6cb11005220
Author: Puranjay Mohan <[email protected]> selftests/bpf: Add XDP load-balancer benchmark driver
This commit wires up the userspace side of the XDP load-balancer benchmark. It covers 24 scenarios and validates correctness before benchmarking by comparing the output packet byte-for-byte against an expected packet.
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +0
-0
[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +2
-0+static void build_packet (int idx) +{
[ ... ]
Quoted 1 line
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +1
-0+if (sc->is_v6) {
[ ... ]
Quoted 18 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +18
-0+ }else { +struct iphdr iph = {}; +__u32 ip_off = off; + + iph.version =4 ; + iph.ihl = sc->set_ip_options ?6 :5 ; + iph.ttl =64 ; + iph.protocol = sc->ip_proto; + iph.saddr =htonl (sc->src_addr); + iph.daddr =htonl (sc->vip_addr); + iph.frag_off = sc->set_frag ?htons (IP_MF ) :0 ; + off +=sizeof (iph); + +if (sc->set_ip_options) { +/* NOP option padding (4 bytes = 1 word) */ +__u32 nop =htonl (0x01010101 ); + +memcpy (p + off, &nop,sizeof (nop));
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c line 501
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +10
-0+ off +=sizeof (nop); + } + +build_l4 (sc, p, &off); + +memcpy (p + off,"bench___payload!" ,16 ); + off +=16 ; + + iph.tot_len =htons (off - ip_off); + iph.check =ip_checksum (&iph,sizeof (iph));
Does this calculation exclude the appended IP options from the checksum?
When sc->set_ip_options is true, the header length is set to 6 words (24 bytes) and 4 bytes of NOP options are copied directly into the packet buffer. However, ip_checksum() is only passed sizeof(iph), which is exactly 20 bytes, and computes the checksum over the local iph struct alone.
While this logically malformed packet might not cause the current tests to fail because the BPF program validates iph->ihl == 5 before verifying the IP checksum, could this mask bugs or produce false test failures if the BPF program is later modified to validate L3 checksums?
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +5
-0+memcpy (p + ip_off, &iph,sizeof (iph)); + } + + pkt_len[idx] = off; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [PATCH bpf-next 6/7] selftests/bpf: Add XDP load-balancer benchmark driver
Quoted 60 lines · On Tue, Apr 28, 2026 at 2:29 AM <[email protected]> wrote:
Sashiko AI review found 1 potential issue(s):
- [Low] Incorrect IPv4 checksum calculation for packets with IP options in test packet generator --commit
cc1cb16090fdc833cb82a1c6a168d6cb11005220
Author: Puranjay Mohan <[email protected]> selftests/bpf: Add XDP load-balancer benchmark driverThis commit wires up the userspace side of the XDP load-balancer benchmark. It covers 24 scenarios and validates correctness before benchmarking by comparing the output packet byte-for-byte against an expected packet.
Quoted 5 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +0
-0[ ... ]
Quoted 2 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +2
-0+static void build_packet (int idx) +{[ ... ]
Quoted 1 line
+ if (
sc->is_v6) {[ ... ]
Quoted 18 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +18
-0+ }else { +struct iphdr iph = {}; +__u32 ip_off = off; + + iph.version =4 ; + iph.ihl = sc->set_ip_options ?6 :5 ; + iph.ttl =64 ; + iph.protocol = sc->ip_proto; + iph.saddr =htonl (sc->src_addr); + iph.daddr =htonl (sc->vip_addr); + iph.frag_off = sc->set_frag ?htons (IP_MF ) :0 ; + off +=sizeof (iph); + +if (sc->set_ip_options) { +/* NOP option padding (4 bytes = 1 word) */ +__u32 nop =htonl (0x01010101 ); + +memcpy (p + off, &nop,sizeof (nop));tools/testing/selftests/bpf/benchs/bench_xdp_lb.c line 501
Quoted 10 lines
tools/testing/selftests/bpf/benchs/bench_xdp_lb.c new file +10
-0+ off +=sizeof (nop); + } + +build_l4 (sc, p, &off); + +memcpy (p + off,"bench___payload!" ,16 ); + off +=16 ; + + iph.tot_len =htons (off - ip_off); + iph.check =ip_checksum (&iph,sizeof (iph));Does this calculation exclude the appended IP options from the checksum?
When
sc->set_ip_optionsis true, the header length is set to 6 words (24 bytes) and 4 bytes of NOP options are copied directly into the packet buffer. However,ip_checksum()is only passed sizeof(iph), which is exactly 20 bytes, and computes the checksum over the local iph struct alone.
Quoted 4 lines
While this logically malformed packet might not cause the current tests to fail because the BPF program validates
iph->ihl== 5 before verifying the IP checksum, could this mask bugs or produce false test failures if the BPF program is later modified to validate L3 checksums?
the current BPF program drops on ihl != 5 and never validates checksums.
PATCH 7/7 selftests/bpf: Add XDP load-balancer benchmark run script
In reply to Puranjay MohanAdd a convenience script that runs all 24 XDP load-balancer scenarios and formats the results as a table with median, stddev, and p99 columns.
./benchs/run_bench_xdp_lb.sh
--- .../selftests/bpf/benchs/run_bench_xdp_lb.sh | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh
tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh new file +79 -0
@@ -0,0 +1,79 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +source ./benchs/run_common.sh + +set -eufo pipefail + +WARMUP=${WARMUP:-3} + +RUN="sudo ./bench -q -w${WARMUP} -a xdp-lb --machine-readable" + +SEP=" +----------------------------------+----------+---------+----------+" +HDR=" | %-32s | %8s | %7s | %8s |\n" +ROW=" | %-32s | %8s | %7s | %8s |\n" + +function group_header() +{ + printf "%s\n" "$SEP" + printf "$HDR" "$1" "p50" "stddev" "p99" + printf "%s\n" "$SEP" +} + +function rval() +{ + echo "$1" | sed -nE "s/.*$2=([^ ]+).*/\1/p" +} + +function run_scenario() +{ + local sc="$1" + shift + local output rline + + output=$($RUN --scenario "$sc" "$@" 2>&1) || true + rline=$(echo "$output" | grep '^RESULT ' || true) + + if [ -z "$rline" ]; then + printf "$ROW" "$sc" "ERR" "-" "-" + return + fi + + printf "$ROW" "$sc" \ + "$(rval "$rline" median)" \ + "$(rval "$rline" stddev)" \ + "$(rval "$rline" p99)" +} + +header "XDP load-balancer benchmark" + +group_header "Single-flow baseline" +for sc in tcp-v4-lru-hit tcp-v4-ch \ + tcp-v6-lru-hit tcp-v6-ch \ + udp-v4-lru-hit udp-v6-lru-hit \ + tcp-v4v6-lru-hit; do + run_scenario "$sc" +done + +group_header "Diverse flows (4K src addrs)" +for sc in tcp-v4-lru-diverse tcp-v4-ch-diverse \ + tcp-v6-lru-diverse tcp-v6-ch-diverse \ + udp-v4-lru-diverse; do + run_scenario "$sc" +done + +group_header "TCP flags" +run_scenario tcp-v4-syn +run_scenario tcp-v4-rst-miss + +group_header "LRU stress" +run_scenario tcp-v4-lru-miss +run_scenario udp-v4-lru-miss +run_scenario tcp-v4-lru-warmup + +group_header "Early exits" +for sc in pass-v4-no-vip pass-v6-no-vip pass-v4-icmp pass-non-ip drop-v4-frag drop-v4-options \ + drop-v6-frag; do + run_scenario "$sc" +done +printf "%s\n" "$SEP"
Signature · 1 line
2.52.0
[email protected]
AI reviewRe: [PATCH bpf-next 7/7] selftests/bpf: Add XDP load-balancer benchmark run script
Sashiko AI review found 1 potential issue(s): - [Medium] Any extra arguments passed to the script are silently ignored because `"$@"` is used incorrectly inside the `run_scenario` bash function. --
Sashiko AI review found 1 potential issue(s):
- [Medium] Any extra arguments passed to the script are silently ignored because "$@" is used incorrectly inside the run_scenario bash function. --
commit eb02fb82c543ffe6f8607af5eadc8ba489f2fb53
Author: Puranjay Mohan <[email protected]>
selftests/bpf: Add XDP load-balancer benchmark run script
This commit adds a convenience script to run 24 XDP load-balancer scenarios. It formats the benchmark results as a table displaying the median, standard deviation, and 99th percentile.
Quoted 5 lines
tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh new file +0
-0
[ ... ]
tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh line 34
Quoted 7 lines
tools/testing/selftests/bpf/benchs/run_bench_xdp_lb.sh new file +7
-0+function run_scenario() +{ + local sc="$1" + shift + local output rline + + output=$($RUN --scenario "$sc" "$@" 2>&1) || true
Does this silently ignore extra arguments passed to the script?
In bash, "$@" inside a function evaluates to the arguments passed to that specific function, not the global script arguments. Since run_scenario is called with exactly one argument ($sc) throughout the script, shift will consume it and leave "$@" empty.
Should the script's global arguments be passed explicitly to run_scenario when it is called, or should the script store its arguments in a global array?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Alexei Starovoitov
AppliedQuoted 110 lines · Puranjay Mohan
Changelog:
RFC: https://lore.kernel.org/all/[email protected]/
Changes in v1:
- Replacebpf_get_cpu_time_counter()withbpf_ktime_get_ns()
- Replacebpf_repeat()with plain for loop andmay_goto
- Refactorcollect_measurements()to reusebench_force_done()
- Remove histogram, verbose calibration output, and per-scenario status prints
- Trim run script table to p50/stddev/p99
- Set env.quiet when --machine-readable is passed
- Add || true to run script benchmark invocation for set -e safety
- Add bpf-nop benchmark as timing overhead baseline (patch 3)
- Use named struct for LRU inner map to fix build on older toolchainsThis series adds an XDP load-balancer benchmark (based on Katran) to the BPF selftest bench framework.
Motivation
----------Existing BPF bench tests measure individual operations (map lookups, kprobes, ring buffers) in isolation. Production BPF programs combine parsing, map lookups, branching, and packet rewriting in a single call chain. The performance characteristics of such programs depend on the interaction of these operations -- register pressure, spills, inlining decisions, branch layout -- which isolated micro-benchmarks do not capture.
This benchmark implements a simplified L4 load-balancer modeled after katran [1]. The BPF program reproduces katran's core datapath:
L3/L4 parsing -> VIP hash lookup -> per-CPU LRU connection table with consistent-hash fallback -> real server selection -> per-VIP and per-real stats -> IPIP/IP6IP6 encapsulation
The BPF code exercises hash maps, array-of-maps (per-CPU LRU), percpu arrays, jhash,
bpf_xdp_adjust_head(),bpf_ktime_get_ns(), andbpf_get_smp_processor_id()in a single pipeline.This is intended as the first in a series of BPF workload benchmarks covering other use cases (
sched_ext, etc.).Design
------A userspace loop calling
bpf_prog_test_run_opts(repeat=1)would measure syscall overhead, not BPF program cost -- the ~4 ns early-exit paths would be buried under kernel entry/exit. Using repeat=N is also unsuitable: the kernel re-runs the same packet without resetting state between iterations, so the second iteration of an encap scenario would process an already-encapsulated packet.Instead, timing is measured inside the BPF program using
bpf_ktime_get_ns().BENCH_BPF_LOOP()brackets N iterations with timestamp reads using a plain for loop withmay_goto, runs a caller-supplied reset block between iterations to undo side effects (e.g. strip encapsulation), and records the elapsed time per batch. One extra untimed iteration runs afterward for output validation.Auto-calibration picks a batch size targeting ~10 ms per invocation. A proportionality sanity check verifies that 2N iterations take ~2x as long as N.
24 scenarios cover the code-path matrix:
- Protocol: TCP, UDP
- Address family: IPv4, IPv6, cross-AF (IPv4-in-IPv6)
- LRU state: hit, miss (16M flow space), diverse (4K flows), cold
- Consistent-hash: direct (LRU bypass)
- TCP flags: SYN (skip LRU, force CH), RST (skip LRU insert)
- Early exits: unknown VIP, non-IP, ICMP, fragments, IP optionsEach scenario validates correctness before benchmarking by comparing the output packet byte-for-byte against a pre-built expected packet and checking BPF map counters.
Sample single-scenario output:
$ sudo ./bench xdp-lb --scenario tcp-v4-lru-hit Setting up benchmark 'xdp-lb'... Benchmark 'xdp-lb' started. tcp-v4-lru-hit: median 74.51 ns/op, stddev 0.11, p99 74.81 (202 samples)Sample run script output:
$ ./benchs/run_bench_xdp_lb.shXDP load-balancer benchmark
===========================+----------------------------------+----------+---------+----------+ | Single-flow baseline | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-hit | 74.30 | 0.08 | 74.48 | | tcp-v4-ch | 101.73 | 0.11 | 102.01 | | tcp-v6-lru-hit | 76.77 | 0.14 | 77.04 | | tcp-v6-ch | 121.40 | 0.10 | 121.65 | | udp-v4-lru-hit | 107.42 | 0.22 | 107.90 | | udp-v6-lru-hit | 110.21 | 0.12 | 110.45 | | tcp-v4v6-lru-hit | 74.82 | 0.35 | 75.43 | +----------------------------------+----------+---------+----------+ | Diverse flows (4K src addrs) | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-diverse | 86.63 | 0.37 | 89.04 | | tcp-v4-ch-diverse | 104.09 | 0.19 | 105.67 | | tcp-v6-lru-diverse | 89.34 | 0.42 | 90.70 | | tcp-v6-ch-diverse | 122.20 | 0.21 | 123.78 | | udp-v4-lru-diverse | 119.37 | 0.58 | 123.10 | +----------------------------------+----------+---------+----------+ | TCP flags | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+
Quoted 8 lines
| tcp-v4-syn | 165.52 | 15.68 | 198.34 | | tcp-v4-rst-miss | 161.34 | 2.69 | 172.64 | +----------------------------------+----------+---------+----------+ | LRU stress | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-miss | 440.39 | 35.75 | 550.62 | | udp-v4-lru-miss | 571.88 | 57.38 | 680.61 | | tcp-v4-lru-warmup | 317.75 | 9.55 | 356.20 |
Applied, but these huge stddev numbers are concerning. Since the results are all over the place, how can we trust them?
Also please follow with a fix for:
./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
[udp-v4-lru-miss] COUNTER FAIL: LRU misses=0, expected 1
Validation FAILED - aborting benchmark
./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
Benchmark 'xdp-lb' started.
udp-v4-lru-miss: median 282.49 ns/op, stddev 21.12, p99 340.05 (223 samples)
./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
[udp-v4-lru-miss] COUNTER FAIL: LRU misses=0, expected 1
Validation FAILED - aborting benchmark
./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
Benchmark 'xdp-lb' started.
udp-v4-lru-miss: median 289.42 ns/op, stddev 13.07, p99 355.29 (223 samples)
It fails way too often.
Quoted 121 lines · Alexei Starovoitov
Quoted 110 lines · Puranjay Mohan
Changelog:
RFC: https://lore.kernel.org/all/[email protected]/
Changes in v1:
- Replacebpf_get_cpu_time_counter()withbpf_ktime_get_ns()
- Replacebpf_repeat()with plain for loop andmay_goto
- Refactorcollect_measurements()to reusebench_force_done()
- Remove histogram, verbose calibration output, and per-scenario status prints
- Trim run script table to p50/stddev/p99
- Set env.quiet when --machine-readable is passed
- Add || true to run script benchmark invocation for set -e safety
- Add bpf-nop benchmark as timing overhead baseline (patch 3)
- Use named struct for LRU inner map to fix build on older toolchainsThis series adds an XDP load-balancer benchmark (based on Katran) to the BPF selftest bench framework.
Motivation
----------Existing BPF bench tests measure individual operations (map lookups, kprobes, ring buffers) in isolation. Production BPF programs combine parsing, map lookups, branching, and packet rewriting in a single call chain. The performance characteristics of such programs depend on the interaction of these operations -- register pressure, spills, inlining decisions, branch layout -- which isolated micro-benchmarks do not capture.
This benchmark implements a simplified L4 load-balancer modeled after katran [1]. The BPF program reproduces katran's core datapath:
L3/L4 parsing -> VIP hash lookup -> per-CPU LRU connection table with consistent-hash fallback -> real server selection -> per-VIP and per-real stats -> IPIP/IP6IP6 encapsulation
The BPF code exercises hash maps, array-of-maps (per-CPU LRU), percpu arrays, jhash,
bpf_xdp_adjust_head(),bpf_ktime_get_ns(), andbpf_get_smp_processor_id()in a single pipeline.This is intended as the first in a series of BPF workload benchmarks covering other use cases (
sched_ext, etc.).Design
------A userspace loop calling
bpf_prog_test_run_opts(repeat=1)would measure syscall overhead, not BPF program cost -- the ~4 ns early-exit paths would be buried under kernel entry/exit. Using repeat=N is also unsuitable: the kernel re-runs the same packet without resetting state between iterations, so the second iteration of an encap scenario would process an already-encapsulated packet.Instead, timing is measured inside the BPF program using
bpf_ktime_get_ns().BENCH_BPF_LOOP()brackets N iterations with timestamp reads using a plain for loop withmay_goto, runs a caller-supplied reset block between iterations to undo side effects (e.g. strip encapsulation), and records the elapsed time per batch. One extra untimed iteration runs afterward for output validation.Auto-calibration picks a batch size targeting ~10 ms per invocation. A proportionality sanity check verifies that 2N iterations take ~2x as long as N.
24 scenarios cover the code-path matrix:
- Protocol: TCP, UDP
- Address family: IPv4, IPv6, cross-AF (IPv4-in-IPv6)
- LRU state: hit, miss (16M flow space), diverse (4K flows), cold
- Consistent-hash: direct (LRU bypass)
- TCP flags: SYN (skip LRU, force CH), RST (skip LRU insert)
- Early exits: unknown VIP, non-IP, ICMP, fragments, IP optionsEach scenario validates correctness before benchmarking by comparing the output packet byte-for-byte against a pre-built expected packet and checking BPF map counters.
Sample single-scenario output:
$ sudo ./bench xdp-lb --scenario tcp-v4-lru-hit Setting up benchmark 'xdp-lb'... Benchmark 'xdp-lb' started. tcp-v4-lru-hit: median 74.51 ns/op, stddev 0.11, p99 74.81 (202 samples)Sample run script output:
$ ./benchs/run_bench_xdp_lb.shXDP load-balancer benchmark
===========================+----------------------------------+----------+---------+----------+ | Single-flow baseline | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-hit | 74.30 | 0.08 | 74.48 | | tcp-v4-ch | 101.73 | 0.11 | 102.01 | | tcp-v6-lru-hit | 76.77 | 0.14 | 77.04 | | tcp-v6-ch | 121.40 | 0.10 | 121.65 | | udp-v4-lru-hit | 107.42 | 0.22 | 107.90 | | udp-v6-lru-hit | 110.21 | 0.12 | 110.45 | | tcp-v4v6-lru-hit | 74.82 | 0.35 | 75.43 | +----------------------------------+----------+---------+----------+ | Diverse flows (4K src addrs) | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-diverse | 86.63 | 0.37 | 89.04 | | tcp-v4-ch-diverse | 104.09 | 0.19 | 105.67 | | tcp-v6-lru-diverse | 89.34 | 0.42 | 90.70 | | tcp-v6-ch-diverse | 122.20 | 0.21 | 123.78 | | udp-v4-lru-diverse | 119.37 | 0.58 | 123.10 | +----------------------------------+----------+---------+----------+ | TCP flags | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+Quoted 8 lines
| tcp-v4-syn | 165.52 | 15.68 | 198.34 | | tcp-v4-rst-miss | 161.34 | 2.69 | 172.64 | +----------------------------------+----------+---------+----------+ | LRU stress | p50 | stddev | p99 | +----------------------------------+----------+---------+----------+ | tcp-v4-lru-miss | 440.39 | 35.75 | 550.62 | | udp-v4-lru-miss | 571.88 | 57.38 | 680.61 | | tcp-v4-lru-warmup | 317.75 | 9.55 | 356.20 |
Quoted 2 lines
Applied, but these huge stddev numbers are concerning. Since the results are all over the place, how can we trust them?
These scenarios have a lot of moving parts and therefore the stddev is high, I will see if I can improve these by running for longer or improve the scenarios itself. The only way to trust these in the current state would be to only take trust improvements in these benchmarks when it is much bigger than stddev.
Quoted 17 lines
Also please follow with a fix for:
./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
[udp-v4-lru-miss] COUNTER FAIL: LRU misses=0, expected 1
Validation FAILED - aborting benchmark./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
Benchmark 'xdp-lb' started.
udp-v4-lru-miss: median 282.49 ns/op, stddev 21.12, p99 340.05 (223 samples)./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
[udp-v4-lru-miss] COUNTER FAIL: LRU misses=0, expected 1
Validation FAILED - aborting benchmark
Quoted 6 lines
./bench xdp-lb --scenario udp-v4-lru-miss
Setting up benchmark 'xdp-lb'...
Benchmark 'xdp-lb' started.
udp-v4-lru-miss: median 289.42 ns/op, stddev 13.07, p99 355.29 (223 samples)It fails way too often.
I will send follow up patches to fix this and the stddev issue.
Thanks
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 13 lines · On Mon, 27 Apr 2026 16:22:57 -0700 you wrote:
Changelog:
RFC: https://lore.kernel.org/all/[email protected]/
Changes in v1:
- Replacebpf_get_cpu_time_counter()withbpf_ktime_get_ns()
- Replacebpf_repeat()with plain for loop andmay_goto
- Refactorcollect_measurements()to reusebench_force_done()
- Remove histogram, verbose calibration output, and per-scenario status prints
- Trim run script table to p50/stddev/p99
- Set env.quiet when --machine-readable is passed
- Add || true to run script benchmark invocation for set -e safety
- Add bpf-nop benchmark as timing overhead baseline (patch 3)
- Use named struct for LRU inner map to fix build on older toolchains
Quoted 1 line
[...]
Here is the summary with links:
- [bpf-next,1/7] selftests/bpf: Add bench_force_done() for early benchmark completion
https://git.kernel.org/bpf/bpf-next/c/2b6f0a1e4c9e - [bpf-next,2/7] selftests/bpf: Add BPF batch-timing library https://git.kernel.org/bpf/bpf-next/c/08158c111d7d - [bpf-next,3/7] selftests/bpf: Add bpf-nop benchmark for timing overhead baseline https://git.kernel.org/bpf/bpf-next/c/dcf11479c2a8 - [bpf-next,4/7] selftests/bpf: Add XDP load-balancer common definitions https://git.kernel.org/bpf/bpf-next/c/6b4003a7b333 - [bpf-next,5/7] selftests/bpf: Add XDP load-balancer BPF program https://git.kernel.org/bpf/bpf-next/c/4b4f2229104c - [bpf-next,6/7] selftests/bpf: Add XDP load-balancer benchmark driver https://git.kernel.org/bpf/bpf-next/c/a4b5ba8187cb - [bpf-next,7/7] selftests/bpf: Add XDP load-balancer benchmark run script https://git.kernel.org/bpf/bpf-next/c/51312b6360a9
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html