bench: fast in-kernel triggering benchmarks
Subsystems (5)
Andrii Nakryiko
maintainerv2 cover bench: fast in-kernel triggering benchmarks
Remove "legacy" triggering benchmarks which rely on syscalls (and thus syscall overhead is a noticeable part of benchmark, unfortunately). Replace them with faster versions that rely on triggering BPF programs in-kernel through another simple "driver" BPF program. See patch #2 with comparison results.
raw_tp/tp/fmodret benchmarks required adding a simple kfunc in kernel to be able to trigger a simple tracepoint from BPF program (plus it is also allowed to be replaced by fmod_ret programs). This limits raw_tp/tp/fmodret benchmarks to new kernels only, but it keeps bench tool itself very portable and most of other benchmarks will still work on wide variety of kernels without the need to worry about building and deploying custom kernel module. See patches #5 and #6 for details.
v1->v2:
- move new TP closer to BPF test run code;
- rename/move kfunc and register it for fmod_rets (Alexei);
- limit --trig-batch-iters param to [1, 1000] (Alexei).
Andrii Nakryiko (6):
selftests/bpf: rename and clean up userspace-triggered benchmarks selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks selftests/bpf: remove syscall-driven benchs, keep syscall-count only selftests/bpf: lazy-load trigger bench BPF programs
bpf: add bpf_modify_return_test_tp() kfunc triggering tracepoint selftests/bpf: add batched tp/raw_tp/fmodret tests
include/trace/events/bpf_test_run.h | 17 + kernel/bpf/helpers.c | 1 + net/bpf/test_run.c | 8 + tools/testing/selftests/bpf/bench.c | 33 +- .../selftests/bpf/benchs/bench_trigger.c | 391 +++++++++--------- .../selftests/bpf/benchs/run_bench_trigger.sh | 22 +- .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- .../selftests/bpf/progs/trigger_bench.c | 68 ++- 8 files changed, 314 insertions(+), 228 deletions(-)
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v2 1/6 selftests/bpf: rename and clean up userspace-triggered benchmarks
Rename uprobe-base to more precise usermode-count (it will match other baseline-like benchmarks, kernel-count and syscall-count). Also use BENCH_TRIG_USERMODE() macro to define all usermode-based triggering benchmarks, which include usermode-count and uprobe/uretprobe benchmarks.
--- tools/testing/selftests/bpf/bench.c | 14 ++- .../selftests/bpf/benchs/bench_trigger.c | 106 ++++++------------ .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- 3 files changed, 49 insertions(+), 73 deletions(-)
tools/testing/selftests/bpf/bench.c +12 -2
@@ -491,7 +491,12 @@ extern const struct bench bench_rename_kretprobe;extern const struct bench bench_rename_rawtp;extern const struct bench bench_rename_fentry;extern const struct bench bench_rename_fexit; + +/* pure counting benchmarks to establish theoretical lmits */ +extern const struct bench bench_trig_usermode_count;extern const struct bench bench_trig_base; + +/* kernel-side syscall-triggered benchmarks */ extern const struct bench bench_trig_tp;extern const struct bench bench_trig_rawtp;extern const struct bench bench_trig_kprobe;
@@ -502,13 +507,15 @@ extern const struct bench bench_trig_fentry;extern const struct bench bench_trig_fexit;extern const struct bench bench_trig_fentry_sleep;extern const struct bench bench_trig_fmodret; -extern const struct bench bench_trig_uprobe_base; + +/* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop;extern const struct bench bench_trig_uretprobe_nop;extern const struct bench bench_trig_uprobe_push;extern const struct bench bench_trig_uretprobe_push;extern const struct bench bench_trig_uprobe_ret;extern const struct bench bench_trig_uretprobe_ret; +extern const struct bench bench_rb_libbpf;extern const struct bench bench_rb_custom;extern const struct bench bench_pb_libbpf;
@@ -539,7 +546,10 @@ static const struct bench *benchs[] = { &bench_rename_rawtp, &bench_rename_fentry, &bench_rename_fexit, +/* pure counting benchmarks for establishing theoretical limits */ + &bench_trig_usermode_count, &bench_trig_base, +/* syscall-driven triggering benchmarks */ &bench_trig_tp, &bench_trig_rawtp, &bench_trig_kprobe,
@@ -550,7 +560,7 @@ static const struct bench *benchs[] = { &bench_trig_fexit, &bench_trig_fentry_sleep, &bench_trig_fmodret, - &bench_trig_uprobe_base, +/* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop, &bench_trig_uprobe_push,
tools/testing/selftests/bpf/benchs/bench_trigger.c +36 -70
153 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh +1 -1
@@ -2,7 +2,7 @@ set -eufo pipefail -for i in base {uprobe,uretprobe}-{nop,push,ret} +for i in usermode-count base {uprobe,uretprobe}-{nop,push,ret} do summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) printf "%-15s: %s\n" $i "$summary"
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v2 2/6 selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks
In reply to Andrii NakryikoExisting kprobe/fentry triggering benchmarks have 1-to-1 mapping between one syscall execution and BPF program run. While we use a fast get_pgid() syscall, syscall overhead can still be non-trivial.
This patch adds kprobe/fentry set of benchmarks significantly amortizing the cost of syscall vs actual BPF triggering overhead. We do this by employing BPF_PROG_TEST_RUN command to trigger "driver" raw_tp program which does a tight parameterized loop calling cheap BPF helper (bpf_get_numa_node_id()), to which kprobe/fentry programs are attached for benchmarking.
This way 1 bpf() syscall causes N executions of BPF program being benchmarked. N defaults to 100, but can be adjusted with --trig-batch-iters CLI argument.
For comparison we also implement a new baseline program that instead of triggering another BPF program just does N atomic per-CPU counter increments, establishing the limit for all other types of program within this batched benchmarking setup.
Taking the final set of benchmarks added in this patch set (including tp/raw_tp/fmodret, added in later patch), and keeping for now "legacy" syscall-driven benchmarks, we can capture all triggering benchmarks in one place for comparison, before we remove the legacy ones (and rename xxx-batched into just xxx).
$ benchs/run_bench_trigger.sh usermode-count : 79.500 ± 0.024M/s kernel-count : 49.949 ± 0.081M/s syscall-count : 9.009 ± 0.007M/s
fentry-batch : 31.002 ± 0.015M/s fexit-batch : 20.372 ± 0.028M/s fmodret-batch : 21.651 ± 0.659M/s rawtp-batch : 36.775 ± 0.264M/s tp-batch : 19.411 ± 0.248M/s kprobe-batch : 12.949 ± 0.220M/s kprobe-multi-batch : 15.400 ± 0.007M/s kretprobe-batch : 5.559 ± 0.011M/s kretprobe-multi-batch: 5.861 ± 0.003M/s
fentry-legacy : 8.329 ± 0.004M/s fexit-legacy : 6.239 ± 0.003M/s fmodret-legacy : 6.595 ± 0.001M/s rawtp-legacy : 8.305 ± 0.004M/s tp-legacy : 6.382 ± 0.001M/s kprobe-legacy : 5.528 ± 0.003M/s kprobe-multi-legacy : 5.864 ± 0.022M/s kretprobe-legacy : 3.081 ± 0.001M/s kretprobe-multi-legacy: 3.193 ± 0.001M/s
Note how xxx-batch variants are measured with significantly higher throughput, even though it's exactly the same in-kernel overhead. As such, results can be compared only between benchmarks of the same kind (syscall vs batched):
fentry-legacy : 8.329 ± 0.004M/s fentry-batch : 31.002 ± 0.015M/s
kprobe-multi-legacy : 5.864 ± 0.022M/s kprobe-multi-batch : 15.400 ± 0.007M/s
Note also that syscall-count is setting a theoretical limit for syscall-triggered benchmarks, while kernel-count is setting similar limits for batch variants. usermode-count is a happy and unachievable case of user space counting without doing any syscalls, and is mostly the measure of CPU speed for such a trivial benchmark.
As was mentioned, tp/raw_tp/fmodret require kernel-side kfunc to produce similar benchmark, which we address in a separate patch.
Note that run_bench_trigger.sh allows to override a list of benchmarks to run, which is very useful for performance work.
--- tools/testing/selftests/bpf/bench.c | 21 ++- .../selftests/bpf/benchs/bench_trigger.c | 133 +++++++++++++++++- .../selftests/bpf/benchs/run_bench_trigger.sh | 24 +++- .../selftests/bpf/progs/trigger_bench.c | 67 ++++++++- 4 files changed, 238 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/bench.c +20 -1
@@ -280,6 +280,7 @@ extern struct argp bench_strncmp_argp;extern struct argp bench_hashmap_lookup_argp;extern struct argp bench_local_storage_create_argp;extern struct argp bench_htab_mem_argp; +extern struct argp bench_trigger_batch_argp;static const struct argp_child bench_parsers[] = { { &bench_ringbufs_argp,0 ,"Ring buffers benchmark" ,0 },
@@ -292,6 +293,7 @@ static const struct argp_child bench_parsers[] = { { &bench_hashmap_lookup_argp, 0, "Hashmap lookup benchmark", 0 }, { &bench_local_storage_create_argp, 0, "local-storage-create benchmark", 0 }, { &bench_htab_mem_argp, 0, "hash map memory benchmark", 0 }, + { &bench_trigger_batch_argp, 0, "BPF triggering benchmark", 0 }, {}, };
@@ -508,6 +510,15 @@ extern const struct bench bench_trig_fexit;extern const struct bench bench_trig_fentry_sleep;extern const struct bench bench_trig_fmodret; +/* batched, staying mostly in-kernel benchmarks */ +extern const struct bench bench_trig_kernel_count; +extern const struct bench bench_trig_kprobe_batch; +extern const struct bench bench_trig_kretprobe_batch; +extern const struct bench bench_trig_kprobe_multi_batch; +extern const struct bench bench_trig_kretprobe_multi_batch; +extern const struct bench bench_trig_fentry_batch; +extern const struct bench bench_trig_fexit_batch; +/* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop;extern const struct bench bench_trig_uretprobe_nop;
@@ -548,7 +559,7 @@ static const struct bench *benchs[] = { &bench_rename_fexit,/* pure counting benchmarks for establishing theoretical limits */ &bench_trig_usermode_count, - &bench_trig_base, + &bench_trig_kernel_count,/* syscall-driven triggering benchmarks */ &bench_trig_tp, &bench_trig_rawtp,
@@ -560,6 +571,13 @@ static const struct bench *benchs[] = { &bench_trig_fexit, &bench_trig_fentry_sleep, &bench_trig_fmodret, +/* batched, staying mostly in-kernel triggers */ + &bench_trig_kprobe_batch, + &bench_trig_kretprobe_batch, + &bench_trig_kprobe_multi_batch, + &bench_trig_kretprobe_multi_batch, + &bench_trig_fentry_batch, + &bench_trig_fexit_batch,/* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop,
@@ -567,6 +585,7 @@ static const struct bench *benchs[] = { &bench_trig_uretprobe_push, &bench_trig_uprobe_ret, &bench_trig_uretprobe_ret, +/* ringbuf/perfbuf benchmarks */ &bench_rb_libbpf, &bench_rb_custom, &bench_pb_libbpf,
tools/testing/selftests/bpf/benchs/bench_trigger.c +132 -1
180 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_trigger.sh +20 -4
@@ -2,8 +2,24 @@ set -eufo pipefail -for i in base tp rawtp kprobe fentry fmodret -do - summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) - printf "%-10s: %s\n" $i "$summary" +def_tests=( \ + usermode-count kernel-count syscall-count \ + fentry-batch fexit-batch \ + kprobe-batch kprobe-multi-batch \ + kretprobe-batch kretprobe-multi-batch \ + fentry fexit fmodret \ + rawtp tp \ + kprobe kprobe-multi kretprobe kretprobe-multi \ +) + +tests=("$@") +if [ ${#tests[@]} -eq 0 ]; then + tests=("${def_tests[@]}") +fi + +p=${PROD_CNT:-1} + +for t in "${tests[@]}"; do + summary=$(sudo ./bench -w2 -d5 -a -p$p trig-$t | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) + printf "%-21s: %s\n" $t "$summary" done
tools/testing/selftests/bpf/progs/trigger_bench.c +66 -1
75 lines not on this page · raw message
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v2 3/6 selftests/bpf: remove syscall-driven benchs, keep syscall-count only
In reply to Andrii NakryikoRemove "legacy" benchmarks triggered by syscalls in favor of newly added in-kernel/batched benchmarks. Drop -batched suffix now as well. Next patch will restore "feature parity" by adding back tp/raw_tp/fmodret benchmarks based on in-kernel kfunc approach.
--- tools/testing/selftests/bpf/bench.c | 32 +-- .../selftests/bpf/benchs/bench_trigger.c | 213 +++--------------- .../selftests/bpf/benchs/run_bench_trigger.sh | 11 +- .../selftests/bpf/benchs/run_bench_uprobes.sh | 2 +- .../selftests/bpf/progs/trigger_bench.c | 83 +------ 5 files changed, 42 insertions(+), 299 deletions(-)
tools/testing/selftests/bpf/bench.c +5 -27
@@ -496,28 +496,16 @@ extern const struct bench bench_rename_fexit;/* pure counting benchmarks to establish theoretical lmits */ extern const struct bench bench_trig_usermode_count; -extern const struct bench bench_trig_base; +extern const struct bench bench_trig_syscall_count; +extern const struct bench bench_trig_kernel_count; -/* kernel-side syscall-triggered benchmarks */ -extern const struct bench bench_trig_tp; -extern const struct bench bench_trig_rawtp; +/* batched, staying mostly in-kernel benchmarks */ extern const struct bench bench_trig_kprobe;extern const struct bench bench_trig_kretprobe;extern const struct bench bench_trig_kprobe_multi;extern const struct bench bench_trig_kretprobe_multi;extern const struct bench bench_trig_fentry;extern const struct bench bench_trig_fexit; -extern const struct bench bench_trig_fentry_sleep; -extern const struct bench bench_trig_fmodret; - -/* batched, staying mostly in-kernel benchmarks */ -extern const struct bench bench_trig_kernel_count; -extern const struct bench bench_trig_kprobe_batch; -extern const struct bench bench_trig_kretprobe_batch; -extern const struct bench bench_trig_kprobe_multi_batch; -extern const struct bench bench_trig_kretprobe_multi_batch; -extern const struct bench bench_trig_fentry_batch; -extern const struct bench bench_trig_fexit_batch;/* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop;
@@ -560,24 +548,14 @@ static const struct bench *benchs[] = {/* pure counting benchmarks for establishing theoretical limits */ &bench_trig_usermode_count, &bench_trig_kernel_count, -/* syscall-driven triggering benchmarks */ - &bench_trig_tp, - &bench_trig_rawtp, + &bench_trig_syscall_count, +/* batched, staying mostly in-kernel triggers */ &bench_trig_kprobe, &bench_trig_kretprobe, &bench_trig_kprobe_multi, &bench_trig_kretprobe_multi, &bench_trig_fentry, &bench_trig_fexit, - &bench_trig_fentry_sleep, - &bench_trig_fmodret, -/* batched, staying mostly in-kernel triggers */ - &bench_trig_kprobe_batch, - &bench_trig_kretprobe_batch, - &bench_trig_kprobe_multi_batch, - &bench_trig_kretprobe_multi_batch, - &bench_trig_fentry_batch, - &bench_trig_fexit_batch,/* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop,
tools/testing/selftests/bpf/benchs/bench_trigger.c +26 -187
257 lines not on this page · raw message
tools/testing/selftests/bpf/benchs/run_bench_trigger.sh +4 -7
@@ -4,12 +4,9 @@ set -eufo pipefail def_tests=( \ usermode-count kernel-count syscall-count \ - fentry-batch fexit-batch \ - kprobe-batch kprobe-multi-batch \ - kretprobe-batch kretprobe-multi-batch \ - fentry fexit fmodret \ - rawtp tp \ - kprobe kprobe-multi kretprobe kretprobe-multi \ + fentry fexit \ + kprobe kprobe-multi \ + kretprobe kretprobe-multi \ ) tests=("$@")
@@ -21,5 +18,5 @@ p=${PROD_CNT:-1} for t in "${tests[@]}"; do summary=$(sudo ./bench -w2 -d5 -a -p$p trig-$t | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) - printf "%-21s: %s\n" $t "$summary" + printf "%-15s: %s\n" $t "$summary" done
tools/testing/selftests/bpf/benchs/run_bench_uprobes.sh +1 -1
@@ -2,7 +2,7 @@ set -eufo pipefail -for i in usermode-count base {uprobe,uretprobe}-{nop,push,ret} +for i in usermode-count syscall-count {uprobe,uretprobe}-{nop,push,ret} do summary=$(sudo ./bench -w2 -d5 -a trig-$i | tail -n1 | cut -d'(' -f1 | cut -d' ' -f3-) printf "%-15s: %s\n" $i "$summary"
tools/testing/selftests/bpf/progs/trigger_bench.c +6 -77
125 lines not on this page · raw message
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v2 4/6 selftests/bpf: lazy-load trigger bench BPF programs
In reply to Andrii NakryikoInstead of front-loading all possible benchmarking BPF programs for trigger benchmarks, explicitly specify which BPF programs are used by specific benchmark and load only it.
This allows to be more flexible in supporting older kernels, where some program types might not be possible to load (e.g., those that rely on newly added kfunc).
--- .../selftests/bpf/benchs/bench_trigger.c | 36 +++++++++++++++++-- .../selftests/bpf/progs/trigger_bench.c | 18 +++++----- 2 files changed, 42 insertions(+), 12 deletions(-)
tools/testing/selftests/bpf/benchs/bench_trigger.c +33 -3
@@ -133,8 +133,6 @@ static void trigger_measure(struct bench_res *res)static void setup_ctx (void ) { -int err; -setup_libbpf (); ctx.skel =trigger_bench__open ();
@@ -143,7 +141,15 @@ static void setup_ctx(void)exit (1 ); } +/* default "driver" BPF program */ +bpf_program__set_autoload (ctx.skel->progs.trigger_driver, true); + ctx.skel->rodata->batch_iters = args.batch_iters; +} + +static void load_ctx (void ) +{ +int err; err =trigger_bench__load (ctx.skel);if (err) {
@@ -172,6 +178,9 @@ static void trigger_syscall_count_setup(void)static void trigger_kernel_count_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver, false); +bpf_program__set_autoload (ctx.skel->progs.trigger_count, true); +load_ctx ();/* override driver program */ ctx.driver_prog_fd =bpf_program__fd (ctx.skel->progs.trigger_count); }
@@ -179,36 +188,48 @@ static void trigger_kernel_count_setup(void)static void trigger_kprobe_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_kprobe, true); +load_ctx ();attach_bpf (ctx.skel->progs.bench_trigger_kprobe); }static void trigger_kretprobe_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_kretprobe, true); +load_ctx ();attach_bpf (ctx.skel->progs.bench_trigger_kretprobe); }static void trigger_kprobe_multi_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_kprobe_multi, true); +load_ctx ();attach_bpf (ctx.skel->progs.bench_trigger_kprobe_multi); }static void trigger_kretprobe_multi_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_kretprobe_multi, true); +load_ctx ();attach_bpf (ctx.skel->progs.bench_trigger_kretprobe_multi); }static void trigger_fentry_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_fentry, true); +load_ctx ();attach_bpf (ctx.skel->progs.bench_trigger_fentry); }static void trigger_fexit_setup (void ) {setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_fexit, true); +load_ctx ();attach_bpf (ctx.skel->progs.bench_trigger_fexit); }
@@ -279,15 +300,24 @@ static void usetup(bool use_retprobe, void *target_addr) {size_t uprobe_offset;struct bpf_link *link; +int err;setup_libbpf (); - ctx.skel =trigger_bench__open_and_load (); + ctx.skel =trigger_bench__open ();if (!ctx.skel) {fprintf (stderr,"failed to open skeleton\n" );exit (1 ); } +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_uprobe, true); + + err =trigger_bench__load (ctx.skel); +if (err) { +fprintf (stderr,"failed to load skeleton\n" ); +exit (1 ); + } + uprobe_offset =get_uprobe_offset (target_addr); link = bpf_program__attach_uprobe(ctx.skel->progs.bench_trigger_uprobe, use_retprobe,
tools/testing/selftests/bpf/progs/trigger_bench.c +9 -9
@@ -25,7 +25,7 @@ static __always_inline void inc_counter(void)__sync_add_and_fetch (&hits[cpu &CPU_MASK ].value,1 ); } -SEC("uprobe") +SEC("?uprobe")int bench_trigger_uprobe (void *ctx) {inc_counter ();
@@ -34,7 +34,7 @@ int bench_trigger_uprobe(void *ctx)const volatile int batch_iters =0 ; -SEC("raw_tp") +SEC("?raw_tp")int trigger_count (void *ctx) {int i;
@@ -45,7 +45,7 @@ int trigger_count(void *ctx)return 0 ; } -SEC("raw_tp") +SEC("?raw_tp")int trigger_driver (void *ctx) {int i;
@@ -56,42 +56,42 @@ int trigger_driver(void *ctx)return 0 ; } -SEC("kprobe/bpf_get_numa_node_id") +SEC("?kprobe/bpf_get_numa_node_id")int bench_trigger_kprobe (void *ctx) {inc_counter ();return 0 ; } -SEC("kretprobe/bpf_get_numa_node_id") +SEC("?kretprobe/bpf_get_numa_node_id")int bench_trigger_kretprobe (void *ctx) {inc_counter ();return 0 ; } -SEC("kprobe.multi/bpf_get_numa_node_id") +SEC("?kprobe.multi/bpf_get_numa_node_id")int bench_trigger_kprobe_multi (void *ctx) {inc_counter ();return 0 ; } -SEC("kretprobe.multi/bpf_get_numa_node_id") +SEC("?kretprobe.multi/bpf_get_numa_node_id")int bench_trigger_kretprobe_multi (void *ctx) {inc_counter ();return 0 ; } -SEC("fentry/bpf_get_numa_node_id") +SEC("?fentry/bpf_get_numa_node_id")int bench_trigger_fentry (void *ctx) {inc_counter ();return 0 ; } -SEC("fexit/bpf_get_numa_node_id") +SEC("?fexit/bpf_get_numa_node_id")int bench_trigger_fexit (void *ctx) {inc_counter ();
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v2 5/6 bpf: add bpf_modify_return_test_tp() kfunc triggering tracepoint
In reply to Andrii NakryikoAdd a simple bpf_modify_return_test_tp() kfunc, available to all program types, that is useful for various testing and benchmarking scenarios, as it allows to trigger most tracing BPF program types from BPF side, allowing to do complex testing and benchmarking scenarios.
It is also attachable to for fmod_ret programs, making it a good and simple way to trigger fmod_ret program under test/benchmark.
--- include/trace/events/bpf_test_run.h | 17 +++++++++++++++++ kernel/bpf/helpers.c | 1 + net/bpf/test_run.c | 8 ++++++++ 3 files changed, 26 insertions(+)
include/trace/events/bpf_test_run.h +17 -0
@@ -7,6 +7,23 @@#include <linux/tracepoint.h> +TRACE_EVENT (bpf_trigger_tp, + + TP_PROTO(int nonce), + +TP_ARGS (nonce), + +TP_STRUCT__entry ( +__field (int, nonce) + ), + +TP_fast_assign ( + __entry->nonce = nonce; + ), + +TP_printk ("nonce %d" , __entry->nonce) +); + DECLARE_EVENT_CLASS(bpf_test_finish, TP_PROTO(int *err),
kernel/bpf/helpers.c +1 -0
@@ -2625,6 +2625,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_null) BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) +BTF_ID_FLAGS(func, bpf_modify_return_test_tp)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
net/bpf/test_run.c +8 -0
@@ -575,6 +575,13 @@ __bpf_kfunc int bpf_modify_return_test2(int a, int *b, short c, int d,return a + *b + c + d + (long )e + f + g; } +__bpf_kfunc int bpf_modify_return_test_tp(int nonce) +{ +trace_bpf_trigger_tp (nonce); + +return nonce; +} + int noinline bpf_fentry_shadow_test(int a) {return a +1 ;
@@ -622,6 +629,7 @@ __bpf_kfunc_end_defs(); BTF_KFUNCS_START(bpf_test_modify_return_ids) BTF_ID_FLAGS(func, bpf_modify_return_test) BTF_ID_FLAGS(func, bpf_modify_return_test2) +BTF_ID_FLAGS(func, bpf_modify_return_test_tp) BTF_ID_FLAGS(func, bpf_fentry_test1, KF_SLEEPABLE) BTF_KFUNCS_END(bpf_test_modify_return_ids)
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v2 6/6 selftests/bpf: add batched tp/raw_tp/fmodret tests
In reply to Andrii NakryikoUtilize bpf_modify_return_test_tp() kfunc to have a fast way to trigger tp/raw_tp/fmodret programs from another BPF program, which gives us comparable batched benchmarks to (batched) kprobe/fentry benchmarks.
We don't switch kprobe/fentry batched benchmarks to this kfunc to make bench tool usable on older kernels as well.
--- tools/testing/selftests/bpf/bench.c | 6 +++ .../selftests/bpf/benchs/bench_trigger.c | 39 +++++++++++++++++++ .../selftests/bpf/benchs/run_bench_trigger.sh | 3 +- .../selftests/bpf/progs/trigger_bench.c | 34 ++++++++++++++++ 4 files changed, 81 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/bench.c +6 -0
@@ -506,6 +506,9 @@ extern const struct bench bench_trig_kprobe_multi;extern const struct bench bench_trig_kretprobe_multi;extern const struct bench bench_trig_fentry;extern const struct bench bench_trig_fexit; +extern const struct bench bench_trig_fmodret; +extern const struct bench bench_trig_tp; +extern const struct bench bench_trig_rawtp;/* uprobe/uretprobe benchmarks */ extern const struct bench bench_trig_uprobe_nop;
@@ -556,6 +559,9 @@ static const struct bench *benchs[] = { &bench_trig_kretprobe_multi, &bench_trig_fentry, &bench_trig_fexit, + &bench_trig_fmodret, + &bench_trig_tp, + &bench_trig_rawtp,/* uprobes */ &bench_trig_uprobe_nop, &bench_trig_uretprobe_nop,
tools/testing/selftests/bpf/benchs/bench_trigger.c +39 -0
@@ -233,6 +233,42 @@ static void trigger_fexit_setup(void)attach_bpf (ctx.skel->progs.bench_trigger_fexit); } +static void trigger_fmodret_setup (void ) +{ +setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver, false); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver_kfunc, true); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_fmodret, true); +load_ctx (); +/* override driver program */ + ctx.driver_prog_fd =bpf_program__fd (ctx.skel->progs.trigger_driver_kfunc); +attach_bpf (ctx.skel->progs.bench_trigger_fmodret); +} + +static void trigger_tp_setup (void ) +{ +setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver, false); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver_kfunc, true); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_tp, true); +load_ctx (); +/* override driver program */ + ctx.driver_prog_fd =bpf_program__fd (ctx.skel->progs.trigger_driver_kfunc); +attach_bpf (ctx.skel->progs.bench_trigger_tp); +} + +static void trigger_rawtp_setup (void ) +{ +setup_ctx (); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver, false); +bpf_program__set_autoload (ctx.skel->progs.trigger_driver_kfunc, true); +bpf_program__set_autoload (ctx.skel->progs.bench_trigger_rawtp, true); +load_ctx (); +/* override driver program */ + ctx.driver_prog_fd =bpf_program__fd (ctx.skel->progs.trigger_driver_kfunc); +attach_bpf (ctx.skel->progs.bench_trigger_rawtp); +} +/* make sure call is not inlined and not avoided by compiler, so __weak and * inline asm volatile in the body of the function *
@@ -395,6 +431,9 @@ BENCH_TRIG_KERNEL(kprobe_multi, "kprobe-multi");BENCH_TRIG_KERNEL (kretprobe_multi,"kretprobe-multi" );BENCH_TRIG_KERNEL (fentry,"fentry" );BENCH_TRIG_KERNEL (fexit,"fexit" ); +BENCH_TRIG_KERNEL (fmodret,"fmodret" ); +BENCH_TRIG_KERNEL (tp,"tp" ); +BENCH_TRIG_KERNEL (rawtp,"rawtp" );/* uprobe benchmarks */ #define BENCH_TRIG_USERMODE (KIND ,PRODUCER ,NAME ) \
tools/testing/selftests/bpf/benchs/run_bench_trigger.sh +2 -1
@@ -4,7 +4,8 @@ set -eufo pipefail def_tests=( \ usermode-count kernel-count syscall-count \ - fentry fexit \ + fentry fexit fmodret \ + rawtp tp \ kprobe kprobe-multi \ kretprobe kretprobe-multi \ )
tools/testing/selftests/bpf/progs/trigger_bench.c +34 -0
@@ -56,6 +56,19 @@ int trigger_driver(void *ctx)return 0 ; } +extern int bpf_modify_return_test_tp(int nonce) __ksym __weak; + +SEC("?raw_tp") +int trigger_driver_kfunc (void *ctx) +{ +int i; + +for (i =0 ; i < batch_iters; i++) + (void )bpf_modify_return_test_tp (0 );/* attach point for benchmarking */ + +return 0 ; +} + SEC("?kprobe/bpf_get_numa_node_id")int bench_trigger_kprobe (void *ctx) {
@@ -97,3 +110,24 @@ int bench_trigger_fexit(void *ctx)inc_counter ();return 0 ; } + +SEC("?fmod_ret/bpf_modify_return_test_tp") +int bench_trigger_fmodret (void *ctx) +{ +inc_counter (); +return -22 ; +} + +SEC("?tp/bpf_test_run/bpf_trigger_tp") +int bench_trigger_tp (void *ctx) +{ +inc_counter (); +return 0 ; +} + +SEC("?raw_tp/bpf_trigger_tp") +int bench_trigger_rawtp (void *ctx) +{ +inc_counter (); +return 0 ; +}
Signature · 1 line
2.43.0
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:
Quoted 13 lines · On Tue, 26 Mar 2024 09:21:45 -0700 you wrote:
Remove "legacy" triggering benchmarks which rely on syscalls (and thus syscall overhead is a noticeable part of benchmark, unfortunately). Replace them with faster versions that rely on triggering BPF programs in-kernel through another simple "driver" BPF program. See patch #2 with comparison results.
raw_tp/tp/fmodret benchmarks required adding a simple kfunc in kernel to be able to trigger a simple tracepoint from BPF program (plus it is also allowed to be replaced by
fmod_retprograms). This limits raw_tp/tp/fmodret benchmarks to new kernels only, but it keeps bench tool itself very portable and most of other benchmarks will still work on wide variety of kernels without the need to worry about building and deploying custom kernel module. See patches #5 and #6 for details.
Quoted 1 line
[...]
Here is the summary with links:
- [v2,bpf-next,1/6] selftests/bpf: rename and clean up userspace-triggered benchmarks
https://git.kernel.org/bpf/bpf-next/c/8f54386ea7ec - [v2,bpf-next,2/6] selftests/bpf: add batched, mostly in-kernel BPF triggering benchmarks https://git.kernel.org/bpf/bpf-next/c/e6c97e34ad7e - [v2,bpf-next,3/6] selftests/bpf: remove syscall-driven benchs, keep syscall-count only https://git.kernel.org/bpf/bpf-next/c/850860ee6712 - [v2,bpf-next,4/6] selftests/bpf: lazy-load trigger bench BPF programs https://git.kernel.org/bpf/bpf-next/c/f328a70ab28b - [v2,bpf-next,5/6] bpf: add bpf_modify_return_test_tp() kfunc triggering tracepoint https://git.kernel.org/bpf/bpf-next/c/c55e25089eba - [v2,bpf-next,6/6] selftests/bpf: add batched tp/raw_tp/fmodret tests https://git.kernel.org/bpf/bpf-next/c/b1d5f62f825d
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html