bpf: add cpu time counter kfuncs
Subsystems (7)
v11 cover bpf: add cpu time counter kfuncs
This patchset adds 2 kfuncs to provide a way to precisely measure the time spent running some code. The first patch provides a way to get cpu cycles counter which is used to feed CLOCK_MONOTONIC_RAW. On x86 architecture it is effectively rdtsc_ordered() function. The second patch adds a kfunc to convert cpu cycles to nanoseconds using shift/mult constants discovered by kernel. The main use-case for this kfunc is to convert deltas of timestamp counter values into nanoseconds. It is not supposed to get CLOCK_MONOTONIC_RAW values as offset part is skipped. JIT version is done for x86 for now, on other architectures it falls back to get CLOCK_MONOTONIC_RAW values.
The reason to have these functions is to avoid overhead added by a bpf_ktime_get_ns() call in case of benchmarking, when two timestamps are taken to get delta value. With both functions being JITed, the overhead is minimal and the result has better precision. New functions can be used to benchmark BPF code directly in the program, or can be used in kprobe/uprobe to store timestamp counter in the session coockie and then in kretprobe/uretprobe the delta can be calculated and converted into nanoseconds.
Selftests are also added to check whether the JIT implementation is correct and to show the simplest usage example.
Change log:
v10 -> v11:
* add missing IS_ENABLED(CONFIG_BPF_SYSCALL)
* reword "cycles" -> "counter"
v9 -> v10:
* rework fallback implementation to avoid using vDSO data from kernel space.
* add comment about using "LFENCE; RDTSC" instead of "RDTSCP"
* guard x86 JIT implementation to be sure that TSC is enabled and stable
* v9 link:
https://lore.kernel.org/bpf/[email protected]/
v8 -> v9:
* rewording of commit messages, no code changes
* move change log from each patch into cover letter
v7 -> v8:
* rename kfuncs again to bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns()
* use cyc2ns_read_begin()/cyc2ns_read_end() to get mult and shift constants in bpf_cpu_time_counter_to_ns()
v6 -> v7:
* change boot_cpu_has() to cpu_feature_enabled() (Borislav)
* return constant clock_mode in __arch_get_hw_counter() call v5 -> v6:
* added cover letter
* add comment about dropping S64_MAX manipulation in jitted implementation of rdtsc_oredered (Alexey)
* add comment about using 'lfence;rdtsc' variant (Alexey)
* change the check in fixup_kfunc_call() (Eduard)
* make __arch_get_hw_counter() call more aligned with vDSO implementation (Yonghong)
v4 -> v5:
* use #if instead of #ifdef with IS_ENABLED
v3 -> v4:
* change name of the helper to bpf_get_cpu_cycles (Andrii)
* Hide the helper behind CONFIG_GENERIC_GETTIMEOFDAY to avoid exposing it on architectures which do not have vDSO functions and data
* reduce the scope of check of inlined functions in verifier to only 2, which are actually inlined.
* change helper name to bpf_cpu_cycles_to_ns.
* hide it behind CONFIG_GENERIC_GETTIMEOFDAY to avoid exposing on unsupported architectures.
v2 -> v3:
* change name of the helper to bpf_get_cpu_cycles_counter to explicitly mention what counter it provides (Andrii)
* move kfunc definition to bpf.h to use it in JIT.
* introduce another kfunc to convert cycles into nanoseconds as more meaningful time units for generic tracing use case (Andrii)
v1 -> v2:
* Fix incorrect function return value type to u64
* Introduce bpf_jit_inlines_kfunc_call() and use it in mark_fastcall_pattern_for_call() to avoid clobbering in case of running programs with no JIT (Eduard)
* Avoid rewriting instruction and check function pointer directly in JIT (Alexei)
* Change includes to fix compile issues on non x86 architectures
Vadim Fedorenko (4):
bpf: add bpf_get_cpu_time_counter kfunc
bpf: add bpf_cpu_time_counter_to_ns helper selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit selftests/bpf: add usage example for cpu time counter kfuncs
arch/x86/net/bpf_jit_comp.c | 73 ++++++++++++ arch/x86/net/bpf_jit_comp32.c | 58 ++++++++++ include/linux/bpf.h | 4 + include/linux/filter.h | 1 + kernel/bpf/core.c | 11 ++ kernel/bpf/helpers.c | 12 ++ kernel/bpf/verifier.c | 41 ++++++- .../bpf/prog_tests/test_cpu_cycles.c | 35 ++++++ .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/test_cpu_cycles.c | 25 +++++ .../selftests/bpf/progs/verifier_cpu_cycles.c | 104 ++++++++++++++++++ 11 files changed, 360 insertions(+), 6 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c
Signature · 1 line
2.47.1
PATCH v11 1/4 bpf: add bpf_get_cpu_time_counter kfunc
New kfunc to return ARCH-specific timecounter. The main reason to implement this kfunc is to avoid extra overhead of benchmark measurements, which are usually done by a pair of bpf_ktime_get_ns() at the beginnig and at the end of the code block under benchmark. When fully JITed this function doesn't implement conversion to the monotonic clock and saves some CPU cycles by receiving timecounter values in single-digit amount of instructions. The delta values can be translated into nanoseconds using kfunc introduced in the next patch. For x86 BPF JIT converts this kfunc into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to get CLOCK_MONOTONIC_RAW value in ns.
JIT version of the function uses "LFENCE; RDTSC" variant because it doesn't care about cookie value returned by "RDTSCP" and it doesn't want to trash RCX value. LFENCE option provides the same ordering guarantee as RDTSCP variant.
The simplest use-case is added in 4th patch, where we calculate the time spent by bpf_get_ns_current_pid_tgid() kfunc. More complex example is to use session cookie to store timecounter value at kprobe/uprobe using kprobe.session/uprobe.session, and calculate the difference at kretprobe/uretprobe.
--- arch/x86/net/bpf_jit_comp.c | 47 +++++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 33 ++++++++++++++++++++++++ include/linux/bpf.h | 3 +++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 ++++++++ kernel/bpf/helpers.c | 6 +++++ kernel/bpf/verifier.c | 41 +++++++++++++++++++++++++----- 7 files changed, 136 insertions(+), 6 deletions(-)
arch/x86/net/bpf_jit_comp.c +47 -0
@@ -15,6 +15,7 @@#include <asm/ftrace.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/timer.h> #include <asm/text-patching.h> #include <asm/unwind.h> #include <asm/cfi.h>
@@ -2254,6 +2255,40 @@ st: if (is_imm8(insn->off))case BPF_JMP |BPF_CALL : {u8 *ip = image + addrs[i -1 ]; +if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && +IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) { +/* The default implementation of this kfunc uses +* ktime_get_raw_ns() which effectively is implemented as +* `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip +* masking part because we assume it's not needed in BPF +* use case (two measurements close in time). +* Original code for rdtsc_ordered() uses sequence: +* 'rdtsc; nop; nop; nop' to patch it into +* 'lfence; rdtsc' or 'rdtscp' depending on CPU features. +* JIT uses 'lfence; rdtsc' variant because BPF program +* doesn't care about cookie provided by rdtscp in RCX. +* Save RDX because RDTSC will use EDX:EAX to return u64 +*/ +emit_mov_reg (&prog, true,AUX_REG ,BPF_REG_3 ); +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +EMIT_LFENCE (); +EMIT2 (0x0F ,0x31 ); + +/* shl RDX, 32 */ +maybe_emit_1mod (&prog,BPF_REG_3 , true); +EMIT3 (0xC1 ,add_1reg (0xE0 ,BPF_REG_3 ),32 ); +/* or RAX, RDX */ +maybe_emit_mod (&prog,BPF_REG_0 ,BPF_REG_3 , true); +EMIT2 (0x09 ,add_2reg (0xC0 ,BPF_REG_0 ,BPF_REG_3 )); +/* restore RDX from R11 */ +emit_mov_reg (&prog, true,BPF_REG_3 ,AUX_REG ); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);
@@ -3865,3 +3900,15 @@ bool bpf_jit_supports_timed_may_goto(void) {return true; } + +/* x86-64 JIT can inline kfunc */ +bool bpf_jit_inlines_kfunc_call (s32 imm) +{ +if (!IS_ENABLED (CONFIG_BPF_SYSCALL )) +return false; +if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) +return true; +return false; +}
arch/x86/net/bpf_jit_comp32.c +33 -0
@@ -16,6 +16,7 @@#include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/asm-prototypes.h> +#include <asm/timer.h> #include <linux/bpf.h> /*
@@ -2094,6 +2095,27 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL ) {int err; +if (IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) { +/* The default implementation of this kfunc uses +* ktime_get_raw_ns() which effectively is implemented as +* `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip +* masking part because we assume it's not needed in BPF +* use case (two measurements close in time). +* Original code for rdtsc_ordered() uses sequence: +* 'rdtsc; nop; nop; nop' to patch it into +* 'lfence; rdtsc' or 'rdtscp' depending on CPU features. +* JIT uses 'lfence; rdtsc' variant because BPF program +* doesn't care about cookie provided by rdtscp in ECX. +*/ +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +EMIT3 (0x0F ,0xAE ,0xE8 ); +EMIT2 (0x0F ,0x31 ); +break ; + } + err =emit_kfunc_call (bpf_prog, image + addrs[i], insn, &prog);
@@ -2621,3 +2643,14 @@ bool bpf_jit_supports_kfunc_call(void) {return true; } + +bool bpf_jit_inlines_kfunc_call (s32 imm) +{ +if (!IS_ENABLED (CONFIG_BPF_SYSCALL )) +return false; +if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) +return true; +return false; +}
include/linux/bpf.h +3 -0
@@ -3387,6 +3387,9 @@ void bpf_user_rnd_init_once(void);u64 bpf_user_rnd_u32 (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5);u64 bpf_get_raw_cpu_id (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5); +/* Inlined kfuncs */ +u64 bpf_get_cpu_time_counter (void ); +#if defined(CONFIG_NET ) bool bpf_sock_common_is_valid_access(int off, int size, enum bpf_access_type type,
include/linux/filter.h +1 -0
@@ -1128,6 +1128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);void bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void );bool bpf_jit_inlines_helper_call (s32 imm); +bool bpf_jit_inlines_kfunc_call (s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );
kernel/bpf/core.c +11 -0
@@ -3035,6 +3035,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm)return false; } +/* Return true if the JIT inlines the call to the kfunc corresponding to +* the imm. +* +* The verifier will not patch the insn->imm for the call to the helper if +* this returns true. +*/ +bool __weak bpf_jit_inlines_kfunc_call (s32 imm) +{ +return false; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {
kernel/bpf/helpers.c +6 -0
@@ -3193,6 +3193,11 @@ __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)local_irq_restore (*flags__irq_flag); } +__bpf_kfunc u64 bpf_get_cpu_time_counter(void) +{ +return ktime_get_raw_fast_ns (); +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)
@@ -3293,6 +3298,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/verifier.c +35 -6
71 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v11 1/4] bpf: add bpf_get_cpu_time_counter kfunc
Quoted 51 lines · Vadim Fedorenko
New kfunc to return ARCH-specific timecounter. The main reason to implement this kfunc is to avoid extra overhead of benchmark measurements, which are usually done by a pair of
bpf_ktime_get_ns()at the beginnig and at the end of the code block under benchmark. When fully JITed this function doesn't implement conversion to the monotonic clock and saves some CPU cycles by receiving timecounter values in single-digit amount of instructions. The delta values can be translated into nanoseconds using kfunc introduced in the next patch. For x86 BPF JIT converts this kfunc into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to getCLOCK_MONOTONIC_RAWvalue in ns.JIT version of the function uses "LFENCE; RDTSC" variant because it doesn't care about cookie value returned by "RDTSCP" and it doesn't want to trash RCX value. LFENCE option provides the same ordering guarantee as RDTSCP variant.
The simplest use-case is added in 4th patch, where we calculate the time spent by
bpf_get_ns_current_pid_tgid()kfunc. More complex example is to use session cookie to store timecounter value at kprobe/uprobe using kprobe.session/uprobe.session, and calculate the difference at kretprobe/uretprobe.Acked-by Eduard Zingerman <[email protected]>Acked-by Andrii Nakryiko <[email protected]>Acked-by Yonghong Song <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 47 +++++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 33 ++++++++++++++++++++++++ include/linux/bpf.h | 3 +++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 ++++++++ kernel/bpf/helpers.c | 6 +++++ kernel/bpf/verifier.c | 41 +++++++++++++++++++++++++----- 7 files changed, 136 insertions(+), 6 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -15,6 +15,7 @@#include <asm/ftrace.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/timer.h> #include <asm/text-patching.h> #include <asm/unwind.h> #include <asm/cfi.h>
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2259
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +2
-0@@ -2254,6 +2255,40 @@ st: if (is_imm8(insn->off))case BPF_JMP |BPF_CALL : {u8 *ip = image + addrs[i -1 ]; + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + IS_ENABLED(CONFIG_BPF_SYSCALL) &&
why?
It's true that JIT can be compiled in even when there is no sys_bpf, but why gate this?
Quoted 36 lines
arch/x86/net/bpf_jit_comp.c +32
-0+ imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +/* The default implementation of this kfunc uses +* ktime_get_raw_ns() which effectively is implemented as +* `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip +* masking part because we assume it's not needed in BPF +* use case (two measurements close in time). +* Original code for rdtsc_ordered() uses sequence: +* 'rdtsc; nop; nop; nop' to patch it into +* 'lfence; rdtsc' or 'rdtscp' depending on CPU features. +* JIT uses 'lfence; rdtsc' variant because BPF program +* doesn't care about cookie provided by rdtscp in RCX. +* Save RDX because RDTSC will use EDX:EAX to return u64 +*/ +emit_mov_reg (&prog, true,AUX_REG ,BPF_REG_3 ); +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +EMIT_LFENCE (); +EMIT2 (0x0F ,0x31 ); + +/* shl RDX, 32 */ +maybe_emit_1mod (&prog,BPF_REG_3 , true); +EMIT3 (0xC1 ,add_1reg (0xE0 ,BPF_REG_3 ),32 ); +/* or RAX, RDX */ +maybe_emit_mod (&prog,BPF_REG_0 ,BPF_REG_3 , true); +EMIT2 (0x09 ,add_2reg (0xC0 ,BPF_REG_0 ,BPF_REG_3 )); +/* restore RDX from R11 */ +emit_mov_reg (&prog, true,BPF_REG_3 ,AUX_REG ); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);
arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_timed_may_goto(void) · line 3908
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +6
-0@@ -3865,3 +3900,15 @@ bool bpf_jit_supports_timed_may_goto(void) {return true; } + +/* x86-64 JIT can inline kfunc */ +bool bpf_jit_inlines_kfunc_call (s32 imm) +{ +if (!IS_ENABLED (CONFIG_BPF_SYSCALL )) +return false;
This is certainly unnecessary.
Only the verifier calls this bpf_jit_inlines_kfunc_call() helper.
arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_timed_may_goto(void) · line 3911
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +3
-0+if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ())
The duplication of the check is ugly.
Call this helper from an earlier bit ?
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
bpf_jit_inlines_kfunc_call(imm32))
?
Quoted 16 lines
arch/x86/net/bpf_jit_comp.c +3
-0+return true; +return false; +}arch/x86/net/bpf_jit_comp32.c +1
-0@@ -16,6 +16,7 @@#include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/asm-prototypes.h> +#include <asm/timer.h> #include <linux/bpf.h> /*
arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2098
Quoted 4 lines
arch/x86/net/bpf_jit_comp32.c +1
-0@@ -2094,6 +2095,27 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL ) {int err; + if (IS_ENABLED(CONFIG_BPF_SYSCALL) &&
same.
Quoted 26 lines
arch/x86/net/bpf_jit_comp32.c +20
-0+ imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +/* The default implementation of this kfunc uses +* ktime_get_raw_ns() which effectively is implemented as +* `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip +* masking part because we assume it's not needed in BPF +* use case (two measurements close in time). +* Original code for rdtsc_ordered() uses sequence: +* 'rdtsc; nop; nop; nop' to patch it into +* 'lfence; rdtsc' or 'rdtscp' depending on CPU features. +* JIT uses 'lfence; rdtsc' variant because BPF program +* doesn't care about cookie provided by rdtscp in ECX. +*/ +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +EMIT3 (0x0F ,0xAE ,0xE8 ); +EMIT2 (0x0F ,0x31 ); +break ; + } + err =emit_kfunc_call (bpf_prog, image + addrs[i], insn, &prog);@@ -2621,3 +2643,14 @@ bool bpf_jit_supports_kfunc_call(void) {return true;
arch/x86/net/bpf_jit_comp32.c bool bpf_jit_supports_kfunc_call(void) · line 2654
Quoted 10 lines
arch/x86/net/bpf_jit_comp32.c +9
-0} + +bool bpf_jit_inlines_kfunc_call (s32 imm) +{ +if (!IS_ENABLED (CONFIG_BPF_SYSCALL )) +return false; +if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) +return true;
same issue.
Quoted 55 lines
arch/x86/net/bpf_jit_comp32.c +2
-0+return false; +}include/linux/bpf.h +3
-0@@ -3387,6 +3387,9 @@ void bpf_user_rnd_init_once(void);u64 bpf_user_rnd_u32 (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5);u64 bpf_get_raw_cpu_id (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5); +/* Inlined kfuncs */ +u64 bpf_get_cpu_time_counter (void ); +#if defined(CONFIG_NET ) bool bpf_sock_common_is_valid_access(int off, int size, enum bpf_access_type type,include/linux/filter.h +1
-0@@ -1128,6 +1128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);void bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void );bool bpf_jit_inlines_helper_call (s32 imm); +bool bpf_jit_inlines_kfunc_call (s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );kernel/bpf/core.c +11
-0@@ -3035,6 +3035,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm)return false; } +/* Return true if the JIT inlines the call to the kfunc corresponding to +* the imm. +* +* The verifier will not patch the insn->imm for the call to the helper if +* this returns true. +*/ +bool __weak bpf_jit_inlines_kfunc_call (s32 imm) +{ +return false; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {kernel/bpf/helpers.c +0
-0
kernel/bpf/helpers.c __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag) · line 3198
Quoted 6 lines
kernel/bpf/helpers.c +3
-0@@ -3193,6 +3193,11 @@ __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)local_irq_restore (*flags__irq_flag); } +__bpf_kfunc u64 bpf_get_cpu_time_counter(void) +{ +return ktime_get_raw_fast_ns ();
Why 'raw' ?
Is it faster than 'mono' ?
This needs a comment at least.
Quoted 21 lines
kernel/bpf/helpers.c +3
-0+} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)@@ -3293,6 +3298,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {kernel/bpf/verifier.c +0
-0@@ -17035,6 +17035,24 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } }
kernel/bpf/verifier.c static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) · line 17047
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* True if fixup_kfunc_call() replaces calls to kfunc number 'imm', +* replacement patch is presumed to follow bpf_fastcall contract +* (see mark_fastcall_pattern_for_call() below). +*/ +static bool verifier_inlines_kfunc_call (struct bpf_verifier_env *env,s32 imm) +{ +const struct bpf_kfunc_desc *desc =find_kfunc_desc (env->prog, imm,0 ); + +if (!env->prog->jit_requested) +return false;
This is a regression.
Why disable verifier inlining when jit is off?
Quoted 32 lines
kernel/bpf/verifier.c +13
-1+ +if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) +return true; + +return false; +} +struct call_summary {u8 num_params;bool is_void;@@ -17077,7 +17095,10 @@ static bool get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call/* error would be reported later */ return false; cs->num_params =btf_type_vlen (meta.func_proto); - cs->fastcall = meta.kfunc_flags &KF_FASTCALL ; + cs->fastcall = meta.kfunc_flags &KF_FASTCALL && + (verifier_inlines_kfunc_call (env, call->imm) || + (meta.btf == btf_vmlinux && +bpf_jit_inlines_kfunc_call (call->imm))); cs->is_void =btf_type_is_void (btf_type_by_id (meta.btf, meta.func_proto->type));return true; }@@ -21223,6 +21244,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; +s32 imm = insn->imm;if (!insn->imm) {verbose (env,"invalid kernel function call not eliminated in verifier pass\n" );
kernel/bpf/verifier.c static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 21273
Quoted 7 lines
kernel/bpf/verifier.c +3
-1@@ -21246,7 +21268,18 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn->imm =BPF_CALL_IMM (desc->addr);if (insn->off)return 0 ; - if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl] || +if (verifier_inlines_kfunc_call (env, imm)) { +if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) {
I really don't like this copy paste.
Next trivial function that inlines as r0=r1
would need to add itself in two places for no good reason.
pw-bot: cr
Re: [PATCH bpf-next v11 1/4] bpf: add bpf_get_cpu_time_counter kfunc
Quoted 55 lines · Alexei Starovoitov
Quoted 54 lines · Vadim Fedorenko
New kfunc to return ARCH-specific timecounter. The main reason to implement this kfunc is to avoid extra overhead of benchmark measurements, which are usually done by a pair of
bpf_ktime_get_ns()at the beginnig and at the end of the code block under benchmark. When fully JITed this function doesn't implement conversion to the monotonic clock and saves some CPU cycles by receiving timecounter values in single-digit amount of instructions. The delta values can be translated into nanoseconds using kfunc introduced in the next patch. For x86 BPF JIT converts this kfunc into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to getCLOCK_MONOTONIC_RAWvalue in ns.JIT version of the function uses "LFENCE; RDTSC" variant because it doesn't care about cookie value returned by "RDTSCP" and it doesn't want to trash RCX value. LFENCE option provides the same ordering guarantee as RDTSCP variant.
The simplest use-case is added in 4th patch, where we calculate the time spent by
bpf_get_ns_current_pid_tgid()kfunc. More complex example is to use session cookie to store timecounter value at kprobe/uprobe using kprobe.session/uprobe.session, and calculate the difference at kretprobe/uretprobe.Acked-by Eduard Zingerman <[email protected]>Acked-by Andrii Nakryiko <[email protected]>Acked-by Yonghong Song <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 47 +++++++++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 33 ++++++++++++++++++++++++ include/linux/bpf.h | 3 +++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 ++++++++ kernel/bpf/helpers.c | 6 +++++ kernel/bpf/verifier.c | 41 +++++++++++++++++++++++++----- 7 files changed, 136 insertions(+), 6 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -15,6 +15,7 @@#include <asm/ftrace.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> +#include <asm/timer.h> #include <asm/text-patching.h> #include <asm/unwind.h> #include <asm/cfi.h> @@ -2254,6 +2255,40 @@ st: if (is_imm8(insn->off))case BPF_JMP |BPF_CALL : {u8 *ip = image + addrs[i -1 ];
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2259
Quoted 2 lines
arch/x86/net/bpf_jit_comp.c +2
-0+ if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + IS_ENABLED(CONFIG_BPF_SYSCALL) &&why?
It's true that JIT can be compiled in even when there is no
sys_bpf, but why gate this?
Both bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() are defined in helpers.c which is compiled only when CONFIG_BPF_SYSCALL is enabled. Otherwise these symbols are absent and compilation fails. See kernel test bot report:
https://lore.kernel.org/bpf/[email protected]/
Quoted 53 lines
Quoted 36 lines
arch/x86/net/bpf_jit_comp.c +32
-0+ imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +/* The default implementation of this kfunc uses +* ktime_get_raw_ns() which effectively is implemented as +* `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip +* masking part because we assume it's not needed in BPF +* use case (two measurements close in time). +* Original code for rdtsc_ordered() uses sequence: +* 'rdtsc; nop; nop; nop' to patch it into +* 'lfence; rdtsc' or 'rdtscp' depending on CPU features. +* JIT uses 'lfence; rdtsc' variant because BPF program +* doesn't care about cookie provided by rdtscp in RCX. +* Save RDX because RDTSC will use EDX:EAX to return u64 +*/ +emit_mov_reg (&prog, true,AUX_REG ,BPF_REG_3 ); +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +EMIT_LFENCE (); +EMIT2 (0x0F ,0x31 ); + +/* shl RDX, 32 */ +maybe_emit_1mod (&prog,BPF_REG_3 , true); +EMIT3 (0xC1 ,add_1reg (0xE0 ,BPF_REG_3 ),32 ); +/* or RAX, RDX */ +maybe_emit_mod (&prog,BPF_REG_0 ,BPF_REG_3 , true); +EMIT2 (0x09 ,add_2reg (0xC0 ,BPF_REG_0 ,BPF_REG_3 )); +/* restore RDX from R11 */ +emit_mov_reg (&prog, true,BPF_REG_3 ,AUX_REG ); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_timed_may_goto(void) · line 3908
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +6
-0@@ -3865,3 +3900,15 @@ bool bpf_jit_supports_timed_may_goto(void) {return true; } + +/* x86-64 JIT can inline kfunc */ +bool bpf_jit_inlines_kfunc_call (s32 imm) +{ +if (!IS_ENABLED (CONFIG_BPF_SYSCALL )) +return false;This is certainly unnecessary.
Only the verifier calls thisbpf_jit_inlines_kfunc_call()helper.arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_timed_may_goto(void) · line 3911
Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +3
-0+if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ())
Quoted 5 lines
The duplication of the check is ugly.
Call this helper from an earlier bit ?
if (insn->src_reg==BPF_PSEUDO_KFUNC_CALL&&bpf_jit_inlines_kfunc_call(imm32))?
LGTM, I'll adjust it
Quoted 121 lines
Quoted 17 lines
arch/x86/net/bpf_jit_comp.c +3
-0+return true; +return false; +}arch/x86/net/bpf_jit_comp32.c +1
-0@@ -16,6 +16,7 @@#include <asm/set_memory.h> #include <asm/nospec-branch.h> #include <asm/asm-prototypes.h> +#include <asm/timer.h> #include <linux/bpf.h> /* arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2098
Quoted 4 lines
arch/x86/net/bpf_jit_comp32.c +1
-0@@ -2094,6 +2095,27 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL ) {int err; + if (IS_ENABLED(CONFIG_BPF_SYSCALL) &&same.
Quoted 26 lines
arch/x86/net/bpf_jit_comp32.c +20
-0+ imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +/* The default implementation of this kfunc uses +* ktime_get_raw_ns() which effectively is implemented as +* `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip +* masking part because we assume it's not needed in BPF +* use case (two measurements close in time). +* Original code for rdtsc_ordered() uses sequence: +* 'rdtsc; nop; nop; nop' to patch it into +* 'lfence; rdtsc' or 'rdtscp' depending on CPU features. +* JIT uses 'lfence; rdtsc' variant because BPF program +* doesn't care about cookie provided by rdtscp in ECX. +*/ +if (cpu_feature_enabled (X86_FEATURE_LFENCE_RDTSC )) +EMIT3 (0x0F ,0xAE ,0xE8 ); +EMIT2 (0x0F ,0x31 ); +break ; + } + err =emit_kfunc_call (bpf_prog, image + addrs[i], insn, &prog);@@ -2621,3 +2643,14 @@ bool bpf_jit_supports_kfunc_call(void) {return true;arch/x86/net/bpf_jit_comp32.c bool bpf_jit_supports_kfunc_call(void) · line 2654
Quoted 10 lines
arch/x86/net/bpf_jit_comp32.c +9
-0} + +bool bpf_jit_inlines_kfunc_call (s32 imm) +{ +if (!IS_ENABLED (CONFIG_BPF_SYSCALL )) +return false; +if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) +return true;same issue.
Quoted 58 lines
arch/x86/net/bpf_jit_comp32.c +2
-0+return false; +}include/linux/bpf.h +3
-0@@ -3387,6 +3387,9 @@ void bpf_user_rnd_init_once(void);u64 bpf_user_rnd_u32 (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5);u64 bpf_get_raw_cpu_id (u64 r1,u64 r2,u64 r3,u64 r4,u64 r5); +/* Inlined kfuncs */ +u64 bpf_get_cpu_time_counter (void ); +#if defined(CONFIG_NET ) bool bpf_sock_common_is_valid_access(int off, int size, enum bpf_access_type type,include/linux/filter.h +1
-0@@ -1128,6 +1128,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);void bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void );bool bpf_jit_inlines_helper_call (s32 imm); +bool bpf_jit_inlines_kfunc_call (s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );kernel/bpf/core.c +11
-0@@ -3035,6 +3035,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm)return false; } +/* Return true if the JIT inlines the call to the kfunc corresponding to +* the imm. +* +* The verifier will not patch the insn->imm for the call to the helper if +* this returns true. +*/ +bool __weak bpf_jit_inlines_kfunc_call (s32 imm) +{ +return false; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {kernel/bpf/helpers.c +0
-0@@ -3193,6 +3193,11 @@ __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag)local_irq_restore (*flags__irq_flag); }
Quoted 7 lines
kernel/bpf/helpers.c __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag) · line 3198
Quoted 3 lines
kernel/bpf/helpers.c +3
-0+__bpf_kfunc u64 bpf_get_cpu_time_counter(void) +{ +return ktime_get_raw_fast_ns ();Why 'raw' ?
Is it faster than 'mono' ?
This needs a comment at least.
'raw' is the closest analogue to what is implemented in JIT. The access
time is the same as for 'mono', but the slope of 'raw' is not affected
by NTP adjustments, and with stable tsc it can provide less jitter in short term measurements.
Quoted 65 lines
Quoted 22 lines
kernel/bpf/helpers.c +3
-0+} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)@@ -3293,6 +3298,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) +BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {kernel/bpf/verifier.c +0
-0@@ -17035,6 +17035,24 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } }kernel/bpf/verifier.c static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) · line 17047
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+/* True if fixup_kfunc_call() replaces calls to kfunc number 'imm', +* replacement patch is presumed to follow bpf_fastcall contract +* (see mark_fastcall_pattern_for_call() below). +*/ +static bool verifier_inlines_kfunc_call (struct bpf_verifier_env *env,s32 imm) +{ +const struct bpf_kfunc_desc *desc =find_kfunc_desc (env->prog, imm,0 ); + +if (!env->prog->jit_requested) +return false;This is a regression.
Why disable verifier inlining when jit is off?Quoted 29 lines
kernel/bpf/verifier.c +13
-1+ +if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) +return true; + +return false; +} +struct call_summary {u8 num_params;bool is_void;@@ -17077,7 +17095,10 @@ static bool get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call/* error would be reported later */ return false; cs->num_params =btf_type_vlen (meta.func_proto); - cs->fastcall = meta.kfunc_flags &KF_FASTCALL ; + cs->fastcall = meta.kfunc_flags &KF_FASTCALL && + (verifier_inlines_kfunc_call (env, call->imm) || + (meta.btf == btf_vmlinux && +bpf_jit_inlines_kfunc_call (call->imm))); cs->is_void =btf_type_is_void (btf_type_by_id (meta.btf, meta.func_proto->type));return true; }@@ -21223,6 +21244,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; +s32 imm = insn->imm;
Quoted 14 lines
kernel/bpf/verifier.c static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 21273
Quoted 10 lines
kernel/bpf/verifier.c +3
-1if (!insn->imm) {verbose (env,"invalid kernel function call not eliminated in verifier pass\n" );@@ -21246,7 +21268,18 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, insn->imm =BPF_CALL_IMM (desc->addr);if (insn->off)return 0 ; - if (desc->func_id == special_kfunc_list[KF_bpf_obj_new_impl] || +if (verifier_inlines_kfunc_call (env, imm)) { +if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) {I really don't like this copy paste.
Next trivial function that inlines as r0=r1
would need to add itself in two places for no good reason.
OK, I'll keep this part untouched
Quoted 2 lines
pw-bot: cr
Re: [PATCH bpf-next v11 1/4] bpf: add bpf_get_cpu_time_counter kfunc
Quoted 7 lines · Vadim Fedorenko
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2259
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +2
-0+ if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + IS_ENABLED(CONFIG_BPF_SYSCALL) &&why?
Quoted 8 lines
Quoted 2 lines
It's true that JIT can be compiled in even when there is no
sys_bpf, but why gate this?Both
bpf_get_cpu_time_counter()andbpf_cpu_time_counter_to_ns()are defined inhelpers.cwhich is compiled only whenCONFIG_BPF_SYSCALLis enabled. Otherwise these symbols are absent and compilation fails. See kernel test bot report:
https://lore.kernel.org/bpf/[email protected]/
put in core.c then?
Quoted 8 lines
Quoted 7 lines
kernel/bpf/helpers.c __bpf_kfunc void bpf_local_irq_restore(unsigned long *flags__irq_flag) · line 3198
Quoted 3 lines
kernel/bpf/helpers.c +3
-0+__bpf_kfunc u64 bpf_get_cpu_time_counter(void) +{ +return ktime_get_raw_fast_ns ();Why 'raw' ?
Is it faster than 'mono' ?
This needs a comment at least.
Quoted 4 lines
'raw' is the closest analogue to what is implemented in JIT. The access time is the same as for 'mono', but the slope of 'raw' is not affected by NTP adjustments, and with stable tsc it can provide less jitter in short term measurements.
fair enough. the comment is needed.
PATCH v11 2/4 bpf: add bpf_cpu_time_counter_to_ns helper
In reply to Vadim FedorenkoThe new helper should be used to convert deltas of values received by bpf_get_cpu_time_counter() into nanoseconds. It is not designed to do full conversion of time counter values to CLOCK_MONOTONIC_RAW nanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds.
This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases.
When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds.
--- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++++++++- arch/x86/net/bpf_jit_comp32.c | 27 ++++++++++++++++++++++++++- include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 6 ++++++ 4 files changed, 60 insertions(+), 2 deletions(-)
arch/x86/net/bpf_jit_comp.c +27 -1
@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/clocksource.h> #include <linux/memory.h> #include <linux/sort.h> #include <asm/extable.h>
@@ -2289,6 +2290,30 @@ st: if (is_imm8(insn->off))break ; } +if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && +IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) { +struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end (); +/* imul RAX, RDI, mult */ +maybe_emit_mod (&prog,BPF_REG_1 ,BPF_REG_0 , true); +EMIT2_off32 (0x69 ,add_2reg (0xC0 ,BPF_REG_1 ,BPF_REG_0 ), + mult); + +/* shr RAX, shift (which is less than 64) */ +maybe_emit_1mod (&prog,BPF_REG_0 , true); +EMIT3 (0xC1 ,add_1reg (0xE8 ,BPF_REG_0 ), shift); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);
@@ -3906,7 +3931,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;
arch/x86/net/bpf_jit_comp32.c +26 -1
@@ -12,6 +12,7 @@#include <linux/netdevice.h> #include <linux/filter.h> #include <linux/if_vlan.h> +#include <linux/clocksource.h> #include <asm/cacheflush.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h>
@@ -2115,6 +2116,29 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,EMIT2 (0x0F ,0x31 );break ; } +if (IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) { +struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end (); + +/* move parameter to BPF_REG_0 */ +emit_ia32_mov_r64 (true, bpf2ia32[BPF_REG_0 ], + bpf2ia32[BPF_REG_1 ], true, true, + &prog, bpf_prog->aux); +/* multiply parameter by mut */ +emit_ia32_mul_i64 (bpf2ia32[BPF_REG_0 ], + mult, true, &prog); +/* shift parameter by shift which is less than 64 */ +emit_ia32_rsh_i64 (bpf2ia32[BPF_REG_0 ], + shift, true, &prog); + } err = emit_kfunc_call(bpf_prog, image + addrs[i],
@@ -2648,7 +2672,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;
include/linux/bpf.h +1 -0
@@ -3389,6 +3389,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);/* Inlined kfuncs */ u64 bpf_get_cpu_time_counter (void ); +u64 bpf_cpu_time_counter_to_ns (u64 counter);#if defined(CONFIG_NET ) bool bpf_sock_common_is_valid_access(int off, int size,
kernel/bpf/helpers.c +6 -0
@@ -3198,6 +3198,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void)return ktime_get_raw_fast_ns (); } +__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 counter) +{ +return counter; +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)
@@ -3299,6 +3304,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_cpu_time_counter_to_ns, KF_FASTCALL)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = {
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v11 2/4] bpf: add bpf_cpu_time_counter_to_ns helper
Quoted 37 lines · Vadim Fedorenko
The new helper should be used to convert deltas of values received by
bpf_get_cpu_time_counter()into nanoseconds. It is not designed to do full conversion of time counter values toCLOCK_MONOTONIC_RAWnanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds.This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases.
When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds.
Reviewed-by Eduard Zingerman <[email protected]>Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++++++++- arch/x86/net/bpf_jit_comp32.c | 27 ++++++++++++++++++++++++++- include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 6 ++++++ 4 files changed, 60 insertions(+), 2 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/clocksource.h> #include <linux/memory.h> #include <linux/sort.h> #include <asm/extable.h>
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2297
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +5
-0@@ -2289,6 +2290,30 @@ st: if (is_imm8(insn->off))break ; } +if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && +IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) {
And now this condition copy pasted 3 times ?!
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2304
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +7
-0+struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end ();
This needs a big comment explaining why this math will be stable after JIT and for the lifetime of the prog.
Quoted 42 lines
arch/x86/net/bpf_jit_comp.c +14
-1+/* imul RAX, RDI, mult */ +maybe_emit_mod (&prog,BPF_REG_1 ,BPF_REG_0 , true); +EMIT2_off32 (0x69 ,add_2reg (0xC0 ,BPF_REG_1 ,BPF_REG_0 ), + mult); + +/* shr RAX, shift (which is less than 64) */ +maybe_emit_1mod (&prog,BPF_REG_0 , true); +EMIT3 (0xC1 ,add_1reg (0xE8 ,BPF_REG_0 ), shift); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);@@ -3906,7 +3931,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;arch/x86/net/bpf_jit_comp32.c +2
-0@@ -12,6 +12,7 @@#include <linux/netdevice.h> #include <linux/filter.h> #include <linux/if_vlan.h> +#include <linux/clocksource.h> #include <asm/cacheflush.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> @@ -2115,6 +2116,29 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,EMIT2 (0x0F ,0x31 );break ; } + if (IS_ENABLED(CONFIG_BPF_SYSCALL) &&
arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2129
Quoted 10 lines
arch/x86/net/bpf_jit_comp32.c +10
-0+ imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end ();
same here.
arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2137
Quoted 8 lines
arch/x86/net/bpf_jit_comp32.c +8
-0+ +/* move parameter to BPF_REG_0 */ +emit_ia32_mov_r64 (true, bpf2ia32[BPF_REG_0 ], + bpf2ia32[BPF_REG_1 ], true, true, + &prog, bpf_prog->aux); +/* multiply parameter by mut */ +emit_ia32_mul_i64 (bpf2ia32[BPF_REG_0 ], + mult, true, &prog);
How did you test this?
It's far from obvious that this will match what mul_u64_u32_shr() does. And on a quick look I really doubt.
The trouble of adding support for 32-bit JIT doesn't seem worth it.
Quoted 56 lines
arch/x86/net/bpf_jit_comp32.c +6
-1+/* shift parameter by shift which is less than 64 */ +emit_ia32_rsh_i64 (bpf2ia32[BPF_REG_0 ], + shift, true, &prog); + } err = emit_kfunc_call(bpf_prog, image + addrs[i],@@ -2648,7 +2672,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;include/linux/bpf.h +1
-0@@ -3389,6 +3389,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);/* Inlined kfuncs */ u64 bpf_get_cpu_time_counter (void ); +u64 bpf_cpu_time_counter_to_ns (u64 counter);#if defined(CONFIG_NET ) bool bpf_sock_common_is_valid_access(int off, int size,kernel/bpf/helpers.c +6
-1@@ -3198,6 +3198,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void)return ktime_get_raw_fast_ns (); } +__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 counter) +{ +return counter; +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)@@ -3299,6 +3304,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_cpu_time_counter_to_ns, KF_FASTCALL)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { --2.47.1
Re: [PATCH bpf-next v11 2/4] bpf: add bpf_cpu_time_counter_to_ns helper
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Vadim Fedorenko
The new helper should be used to convert deltas of values received by
bpf_get_cpu_time_counter()into nanoseconds. It is not designed to do full conversion of time counter values toCLOCK_MONOTONIC_RAWnanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds.This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases.
When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds.
Reviewed-by Eduard Zingerman <[email protected]>Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++++++++- arch/x86/net/bpf_jit_comp32.c | 27 ++++++++++++++++++++++++++- include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 6 ++++++ 4 files changed, 60 insertions(+), 2 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/clocksource.h> #include <linux/memory.h> #include <linux/sort.h> #include <asm/extable.h> @@ -2289,6 +2290,30 @@ st: if (is_imm8(insn->off))break ; }
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2297
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && +IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) {And now this condition copy pasted 3 times ?!
Yeah, I'll factor it out
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2304
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +7
-0+struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end ();This needs a big comment explaining why this math will be stable after JIT and for the lifetime of the prog.
It's more or less the same comment as for the JIT of bpf_get_cpu_time_counter(). I'll add it.
Quoted 64 lines
Quoted 42 lines
arch/x86/net/bpf_jit_comp.c +14
-1+/* imul RAX, RDI, mult */ +maybe_emit_mod (&prog,BPF_REG_1 ,BPF_REG_0 , true); +EMIT2_off32 (0x69 ,add_2reg (0xC0 ,BPF_REG_1 ,BPF_REG_0 ), + mult); + +/* shr RAX, shift (which is less than 64) */ +maybe_emit_1mod (&prog,BPF_REG_0 , true); +EMIT3 (0xC1 ,add_1reg (0xE8 ,BPF_REG_0 ), shift); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);@@ -3906,7 +3931,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;arch/x86/net/bpf_jit_comp32.c +2
-0@@ -12,6 +12,7 @@#include <linux/netdevice.h> #include <linux/filter.h> #include <linux/if_vlan.h> +#include <linux/clocksource.h> #include <asm/cacheflush.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> @@ -2115,6 +2116,29 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,EMIT2 (0x0F ,0x31 );break ; } + if (IS_ENABLED(CONFIG_BPF_SYSCALL) &&arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2129
Quoted 10 lines
arch/x86/net/bpf_jit_comp32.c +10
-0+ imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end ();same here.
arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2137
Quoted 8 lines
arch/x86/net/bpf_jit_comp32.c +8
-0+ +/* move parameter to BPF_REG_0 */ +emit_ia32_mov_r64 (true, bpf2ia32[BPF_REG_0 ], + bpf2ia32[BPF_REG_1 ], true, true, + &prog, bpf_prog->aux); +/* multiply parameter by mut */ +emit_ia32_mul_i64 (bpf2ia32[BPF_REG_0 ], + mult, true, &prog);
Quoted 3 lines
How did you test this?
It's far from obvious that this will match whatmul_u64_u32_shr()does. And on a quick look I really doubt.
Well, I can re-write it op-by-op from mul_u64_u32_shr(), but it's more or less the same given that mult and shift are not too big, which is common for TSC coefficients.
Quoted 2 lines
The trouble of adding support for 32-bit JIT doesn't seem worth it.
Do you mean it's better to drop this JIT implementation?
Quoted 57 lines
Quoted 57 lines
arch/x86/net/bpf_jit_comp32.c +6
-1+/* shift parameter by shift which is less than 64 */ +emit_ia32_rsh_i64 (bpf2ia32[BPF_REG_0 ], + shift, true, &prog); + } err = emit_kfunc_call(bpf_prog, image + addrs[i],@@ -2648,7 +2672,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;include/linux/bpf.h +1
-0@@ -3389,6 +3389,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);/* Inlined kfuncs */ u64 bpf_get_cpu_time_counter (void ); +u64 bpf_cpu_time_counter_to_ns (u64 counter);#if defined(CONFIG_NET ) bool bpf_sock_common_is_valid_access(int off, int size,kernel/bpf/helpers.c +6
-1@@ -3198,6 +3198,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void)return ktime_get_raw_fast_ns (); } +__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 counter) +{ +return counter; +} +__bpf_kfunc_end_defs ();BTF_KFUNCS_START (generic_btf_ids)@@ -3299,6 +3304,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_local_irq_save) BTF_ID_FLAGS(func, bpf_local_irq_restore) BTF_ID_FLAGS(func, bpf_get_cpu_time_counter, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_cpu_time_counter_to_ns, KF_FASTCALL)BTF_KFUNCS_END (common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { --2.47.1
Re: [PATCH bpf-next v11 2/4] bpf: add bpf_cpu_time_counter_to_ns helper
Quoted 136 lines · Vadim Fedorenko
Quoted 41 lines · Alexei Starovoitov
Quoted 40 lines · Vadim Fedorenko
The new helper should be used to convert deltas of values received by
bpf_get_cpu_time_counter()into nanoseconds. It is not designed to do full conversion of time counter values toCLOCK_MONOTONIC_RAWnanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds.This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases.
When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds.
Reviewed-by Eduard Zingerman <[email protected]>Acked-by Andrii Nakryiko <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 28 +++++++++++++++++++++++++++- arch/x86/net/bpf_jit_comp32.c | 27 ++++++++++++++++++++++++++- include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 6 ++++++ 4 files changed, 60 insertions(+), 2 deletions(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -9,6 +9,7 @@#include <linux/filter.h> #include <linux/if_vlan.h> #include <linux/bpf.h> +#include <linux/clocksource.h> #include <linux/memory.h> #include <linux/sort.h> #include <asm/extable.h> @@ -2289,6 +2290,30 @@ st: if (is_imm8(insn->off))break ; }Quoted 7 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2297
Quoted 5 lines
arch/x86/net/bpf_jit_comp.c +5
-0+if (insn->src_reg ==BPF_PSEUDO_KFUNC_CALL && +IS_ENABLED (CONFIG_BPF_SYSCALL ) && + imm32 ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns) && +cpu_feature_enabled (X86_FEATURE_TSC ) && +using_native_sched_clock () &&sched_clock_stable ()) {And now this condition copy pasted 3 times ?!
Yeah, I'll factor it out
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2304
Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +7
-0+struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end ();This needs a big comment explaining why this math will be stable after JIT and for the lifetime of the prog.
It's more or less the same comment as for the JIT of
bpf_get_cpu_time_counter(). I'll add it.Quoted 64 lines
Quoted 42 lines
arch/x86/net/bpf_jit_comp.c +14
-1+/* imul RAX, RDI, mult */ +maybe_emit_mod (&prog,BPF_REG_1 ,BPF_REG_0 , true); +EMIT2_off32 (0x69 ,add_2reg (0xC0 ,BPF_REG_1 ,BPF_REG_0 ), + mult); + +/* shr RAX, shift (which is less than 64) */ +maybe_emit_1mod (&prog,BPF_REG_0 , true); +EMIT3 (0xC1 ,add_1reg (0xE8 ,BPF_REG_0 ), shift); + +break ; + } + func = (u8 *) __bpf_call_base + imm32;if (src_reg ==BPF_PSEUDO_CALL && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (stack_depth);@@ -3906,7 +3931,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) {if (!IS_ENABLED (CONFIG_BPF_SYSCALL ))return false; -if (imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) && +if ((imm ==BPF_CALL_IMM (bpf_get_cpu_time_counter) || + imm ==BPF_CALL_IMM (bpf_cpu_time_counter_to_ns)) &&cpu_feature_enabled (X86_FEATURE_TSC ) &&using_native_sched_clock () &&sched_clock_stable ())return true;arch/x86/net/bpf_jit_comp32.c +2
-0@@ -12,6 +12,7 @@#include <linux/netdevice.h> #include <linux/filter.h> #include <linux/if_vlan.h> +#include <linux/clocksource.h> #include <asm/cacheflush.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h> @@ -2115,6 +2116,29 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image,EMIT2 (0x0F ,0x31 );break ; } + if (IS_ENABLED(CONFIG_BPF_SYSCALL) &&arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2129
Quoted 10 lines
arch/x86/net/bpf_jit_comp32.c +10
-0+ imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) && + cpu_feature_enabled(X86_FEATURE_TSC) && + using_native_sched_clock() && sched_clock_stable()) { +struct cyc2ns_data data; +u32 mult, shift; + +cyc2ns_read_begin (&data); + mult = data.cyc2ns_mul; + shift = data.cyc2ns_shift; +cyc2ns_read_end ();same here.
arch/x86/net/bpf_jit_comp32.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, · line 2137
Quoted 8 lines
arch/x86/net/bpf_jit_comp32.c +8
-0+ +/* move parameter to BPF_REG_0 */ +emit_ia32_mov_r64 (true, bpf2ia32[BPF_REG_0 ], + bpf2ia32[BPF_REG_1 ], true, true, + &prog, bpf_prog->aux); +/* multiply parameter by mut */ +emit_ia32_mul_i64 (bpf2ia32[BPF_REG_0 ], + mult, true, &prog);Quoted 3 lines
How did you test this?
It's far from obvious that this will match whatmul_u64_u32_shr()does. And on a quick look I really doubt.
Quoted 8 lines
Well, I can re-write it op-by-op from
mul_u64_u32_shr(), but it's more or less the same given that mult and shift are not too big, which is common for TSC coefficients.Quoted 2 lines
The trouble of adding support for 32-bit JIT doesn't seem worth it.
Do you mean it's better to drop this JIT implementation?
yes.
The additional complexity doesn't look justified.
PATCH v11 3/4 selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit
In reply to Vadim Fedorenkobpf_get_cpu_time_counter() is replaced with rdtsc instruction on x86_64. Add tests to check that JIT works as expected.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_cpu_cycles.c | 104 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -102,6 +102,7 @@#include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "verifier_cpu_cycles.skel.h" #include "irq.skel.h" #define MAX_ENTRIES 11
@@ -236,6 +237,7 @@ void test_verifier_bits_iter(void) { RUN(verifier_bits_iter); }void test_verifier_lsm (void ) {RUN (verifier_lsm); }void test_irq (void ) {RUN (irq); }void test_verifier_mtu (void ) {RUN (verifier_mtu); } +void test_verifier_cpu_cycles (void ) {RUN (verifier_cpu_cycles); }static int init_test_val_map (struct bpf_object *obj,char *map_name) {
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +104 -0
104 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH bpf-next v11 3/4] selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit
Quoted 38 lines · Vadim Fedorenko
bpf_get_cpu_time_counter()is replaced with rdtsc instruction onx86_64. Add tests to check that JIT works as expected.Acked-by Eduard Zingerman <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_cpu_cycles.c | 104 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.ctools/testing/selftests/bpf/prog_tests/verifier.c +2
-0@@ -102,6 +102,7 @@#include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "verifier_cpu_cycles.skel.h" #include "irq.skel.h" #define MAX_ENTRIES 11@@ -236,6 +237,7 @@ void test_verifier_bits_iter(void) { RUN(verifier_bits_iter); }void test_verifier_lsm (void ) {RUN (verifier_lsm); }void test_irq (void ) {RUN (irq); }void test_verifier_mtu (void ) {RUN (verifier_mtu); } +void test_verifier_cpu_cycles (void ) {RUN (verifier_cpu_cycles); }static int init_test_val_map (struct bpf_object *obj,char *map_name) {tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +0
-0
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c line 2
Quoted 2 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +2
-0@@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Inc. */
botched copy paste.
Re: [PATCH bpf-next v11 3/4] selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit
Quoted 31 lines · Alexei Starovoitov
Quoted 30 lines · Vadim Fedorenko
bpf_get_cpu_time_counter()is replaced with rdtsc instruction onx86_64. Add tests to check that JIT works as expected.Acked-by Eduard Zingerman <[email protected]>Signed-off-by Vadim Fedorenko <[email protected]>--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_cpu_cycles.c | 104 ++++++++++++++++++ 2 files changed, 106 insertions(+)create mode 100644
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.ctools/testing/selftests/bpf/prog_tests/verifier.c +2
-0@@ -102,6 +102,7 @@#include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "verifier_cpu_cycles.skel.h" #include "irq.skel.h" #define MAX_ENTRIES 11@@ -236,6 +237,7 @@ void test_verifier_bits_iter(void) { RUN(verifier_bits_iter); }void test_verifier_lsm (void ) {RUN (verifier_lsm); }void test_irq (void ) {RUN (irq); }void test_verifier_mtu (void ) {RUN (verifier_mtu); } +void test_verifier_cpu_cycles (void ) {RUN (verifier_cpu_cycles); }
Quoted 12 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c line 2
Quoted 10 lines
tools/testing/selftests/bpf/prog_tests/verifier.c +0
-0static int init_test_val_map (struct bpf_object *obj,char *map_name) {tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +2
-0@@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Inc. */ botched copy paste.
Ouch.
PATCH v11 4/4 selftests/bpf: add usage example for cpu time counter kfuncs
In reply to Vadim FedorenkoThe selftest provides an example of how to measure the latency of bpf kfunc/helper call using time stamp counter and how to convert measured value into nanoseconds.
--- .../bpf/prog_tests/test_cpu_cycles.c | 35 +++++++++++++++++++ .../selftests/bpf/progs/test_cpu_cycles.c | 25 +++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c
tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file +35 -0
@@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Inc. */ + +#include <test_progs.h> +#include "test_cpu_cycles.skel.h" + +static void cpu_cycles (void ) +{ +LIBBPF_OPTS (bpf_test_run_opts, opts); +struct test_cpu_cycles *skel; +int err, pfd; + + skel =test_cpu_cycles__open_and_load (); +if (!ASSERT_OK_PTR (skel,"test_cpu_cycles open and load" )) +return ; + + pfd =bpf_program__fd (skel->progs.bpf_cpu_cycles); +if (!ASSERT_GT (pfd,0 ,"test_cpu_cycles fd" )) + goto fail; + + err =bpf_prog_test_run_opts (pfd, &opts); +if (!ASSERT_OK (err,"test_cpu_cycles test run" )) + goto fail; + +ASSERT_NEQ (skel->bss->cycles,0 ,"test_cpu_cycles 0 cycles" ); +ASSERT_NEQ (skel->bss->ns,0 ,"test_cpu_cycles 0 ns" ); +fail: +test_cpu_cycles__destroy (skel); +} + +void test_cpu_cycles (void ) +{ +if (test__start_subtest ("cpu_cycles" )) +cpu_cycles (); +}
tools/testing/selftests/bpf/progs/test_cpu_cycles.c new file +25 -0
@@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Inc. */ + +#include "vmlinux.h" +#include <bpf/bpf_helpers.h> + +extern u64 bpf_cpu_time_counter_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_time_counter(void) __weak __ksym; + +__u64 cycles, ns; + +SEC("syscall") +int bpf_cpu_cycles (void ) +{ +struct bpf_pidns_info pidns; +__u64 start; + + start =bpf_get_cpu_time_counter (); +bpf_get_ns_current_pid_tgid (0 ,0 , &pidns,sizeof (struct bpf_pidns_info )); + cycles =bpf_get_cpu_time_counter () - start; + ns =bpf_cpu_time_counter_to_ns (cycles); +return 0 ; +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.47.1