Skip to contentSkip to conversation

bpf: add cpu time counter kfuncs

PATCH In review Puranjay Mohan 7 participants 19 messages Apr 18 – Apr 20

Subsystems (9)
x86 Kselftest BPF arm64 +5 more subsystems
lore
Share & tools
lore

Puranjay Mohan

author maintainer
#0 lore raw

v13 cover bpf: add cpu time counter kfuncs

This patchset adds 2 kfuncs to provide a way to precisely measure the time spent running some code. The first patch provides a way to get cpu cycles counter which is used to feed CLOCK_MONOTONIC_RAW. On x86 architecture it is effectively rdtsc_ordered() function. On arm64, it is JITed to a read of CNTVCT_EL0 with ordering or CNTVCTSS_EL0, depending on cpu support.

The second patch adds a kfunc to convert cpu cycles to nanoseconds using shift/mult constants discovered by kernel. The main use-case for this kfunc is to convert deltas of timestamp counter values into nanoseconds. It is not supposed to get CLOCK_MONOTONIC_RAW values as offset part is skipped. JIT version is done for x86 and arm64 for now, on other architectures it falls back to get CLOCK_MONOTONIC_RAW values.

The reason to have these functions is to avoid overhead added by a bpf_ktime_get_ns() call in case of benchmarking, when two timestamps are taken to get delta value. With both functions being JITed, the overhead is minimal and the result has better precision. New functions can be used to benchmark BPF code directly in the program, or can be used in kprobe/uprobe to store timestamp counter in the session cookie and then in kretprobe/uretprobe the delta can be calculated and converted into nanoseconds.

These will be used in under-development BPF benchmarks, the first of which is available at [1].

Pre-requriement for this patch is to adjust CONFIG_BPF_JIT to depend on CONFIG_BPF_SYSCALL as it simplify things and it's actually a long overdue patch.

Selftests are also added to check whether the JIT implementation is correct and to show the simplest usage example.

[1] https://github.com/puranjaymohan/bpf/commits/bpf_bench/

Change log:
v12: https://lore.kernel.org/bpf/[email protected]/
Changes in v13:
- Added support for arm64
- Rebased on latest bpf-next/master
- Changed the word helper to kfunc in the commit message to not confuse with bpf helpers.

v11 -> v12:
* drop x86_32 JIT implementation because previous implementation of mul_u64_u32_shr() was not fully correct and the amount of work to properly implement it in asm is not worth the result.
* add comment explaining stablility of shift and mult values
* add comment explaining the reasoning behind CLOCK_MONOTONIC_RAW usage in fallback mode
* drop KF_FASTCALL tag from bpf_get_cpu_time_counter() as it cannot be implemented as fastcall in the verifier
* re-implement verifier checks for possible JIT of kfuncs
* adjust selftests to use JIT code for x86_64 and verifier inlining for arm64
v10 -> v11:
* add missing IS_ENABLED(CONFIG_BPF_SYSCALL)
* reword "cycles" -> "counter"
v9 -> v10:
* rework fallback implementation to avoid using vDSO data from kernel space.
* add comment about using "LFENCE; RDTSC" instead of "RDTSCP"
* guard x86 JIT implementation to be sure that TSC is enabled and stable
* v9 link:
https://lore.kernel.org/bpf/[email protected]/
v8 -> v9:
* rewording of commit messages, no code changes
* move change log from each patch into cover letter
v7 -> v8:
* rename kfuncs again to bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns()
* use cyc2ns_read_begin()/cyc2ns_read_end() to get mult and shift constants in bpf_cpu_time_counter_to_ns()
v6 -> v7:
* change boot_cpu_has() to cpu_feature_enabled() (Borislav)
* return constant clock_mode in __arch_get_hw_counter() call v5 -> v6:
* added cover letter
* add comment about dropping S64_MAX manipulation in jitted implementation of rdtsc_oredered (Alexey)
* add comment about using 'lfence;rdtsc' variant (Alexey)
* change the check in fixup_kfunc_call() (Eduard)
* make __arch_get_hw_counter() call more aligned with vDSO implementation (Yonghong)
v4 -> v5:
* use #if instead of #ifdef with IS_ENABLED
v3 -> v4:
* change name of the helper to bpf_get_cpu_cycles (Andrii)
* Hide the helper behind CONFIG_GENERIC_GETTIMEOFDAY to avoid exposing it on architectures which do not have vDSO functions and data
* reduce the scope of check of inlined functions in verifier to only 2, which are actually inlined.
* change helper name to bpf_cpu_cycles_to_ns.
* hide it behind CONFIG_GENERIC_GETTIMEOFDAY to avoid exposing on unsupported architectures.
v2 -> v3:
* change name of the helper to bpf_get_cpu_cycles_counter to explicitly mention what counter it provides (Andrii)
* move kfunc definition to bpf.h to use it in JIT.
* introduce another kfunc to convert cycles into nanoseconds as more meaningful time units for generic tracing use case (Andrii)
v1 -> v2:
* Fix incorrect function return value type to u64
* Introduce bpf_jit_inlines_kfunc_call() and use it in mark_fastcall_pattern_for_call() to avoid clobbering in case of running programs with no JIT (Eduard)
* Avoid rewriting instruction and check function pointer directly in JIT (Alexei)
* Change includes to fix compile issues on non x86 architectures

Puranjay Mohan (1):
bpf, arm64: Add JIT support for cpu time counter kfuncs

Vadim Fedorenko (5):
bpf: adjust BPF JIT dependency to BPF_SYSCALL
bpf: add bpf_get_cpu_time_counter kfunc
bpf: add bpf_cpu_time_counter_to_ns kfunc selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit selftests/bpf: add usage example for cpu time counter kfuncs

 arch/arm64/include/asm/insn.h                 |   2 +
 arch/arm64/net/bpf_jit.h                      |   4 +
 arch/arm64/net/bpf_jit_comp.c                 |  54 ++++++
 arch/x86/net/bpf_jit_comp.c                   |  72 ++++++++
 arch/x86/net/bpf_jit_comp32.c                 |   2 +
 include/linux/bpf.h                           |   4 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/Kconfig                            |   2 +-
 kernel/bpf/core.c                             |  11 ++
 kernel/bpf/helpers.c                          |  17 ++
 kernel/bpf/verifier.c                         |  14 +-
 .../bpf/prog_tests/test_cpu_cycles.c          |  35 ++++
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/test_cpu_cycles.c     |  25 +++
 .../selftests/bpf/progs/verifier_cpu_cycles.c | 168 ++++++++++++++++++
 15 files changed, 408 insertions(+), 5 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c

base-commit: eb0d6d97c27c29cd7392c8fd74f46edf7dff7ec2

Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH v13 1/6 bpf: adjust BPF JIT dependency to BPF_SYSCALL

From: Vadim Fedorenko <[email protected]>

BPF JIT is moving towards optimizing kfuncs and it was long overdue to switch the dependency. Let's do it now to simplify other patches in the series.

---
 kernel/bpf/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/Kconfig +1 -1
index eb3de35734f0..b0f49be35675 100644
@@ -41,7 +41,7 @@ config BPF_SYSCALL
 
 config BPF_JIT
 	bool "Enable BPF Just In Time compiler"
-	depends on BPF
+	depends on BPF_SYSCALL
 	depends on HAVE_CBPF_JIT || HAVE_EBPF_JIT
 	select EXECMEM
 	help
Signature · 1 line
2.52.0
#2 lore raw

Re: [PATCH bpf-next v13 1/6] bpf: adjust BPF JIT dependency to BPF_SYSCALL

Sashiko AI review found 1 potential issue(s): - [High] Making `CONFIG_BPF_JIT` depend on `CONFIG_BPF_SYSCALL` silently disables JIT compilation for seccomp and classic BPF (cBPF) socket filters on…

Sashiko AI review found 1 potential issue(s):
- [High] Making CONFIG_BPF_JIT depend on CONFIG_BPF_SYSCALL silently disables JIT compilation for seccomp and classic BPF (cBPF) socket filters on systems that intentionally disable the bpf() syscall. --

commit b9f1b2db51ab2071f24b457fa58ed412ec5bc050
Author: Vadim Fedorenko <[email protected]>

bpf: adjust BPF JIT dependency to BPF_SYSCALL

This commit changes the Kconfig dependency for BPF_JIT from BPF to BPF_SYSCALL. The intention is to simplify other patches in the series as BPF JIT moves towards optimizing kfuncs.

kernel/bpf/Kconfig config BPF_SYSCALL · line 44

Quoted 10 lines
kernel/bpf/Kconfig +1 -1
index eb3de35734f09..b0f49be35675f 100644
@@ -41,7 +41,7 @@ config BPF_SYSCALL
 
 config BPF_JIT
 	bool "Enable BPF Just In Time compiler"
-	depends on BPF
+	depends on BPF_SYSCALL

Does this change cause a regression by silently disabling JIT compilation for seccomp and classic BPF socket filters on systems that intentionally disable the bpf() syscall?

The kernel's internal BPF engine (CONFIG_BPF) translates classic BPF to eBPF for internal users like seccomp and SO_ATTACH_FILTER. The JIT compiler (CONFIG_BPF_JIT) optimizes these internal programs independently of the bpf() syscall.

Systems can configure CONFIG_BPF_JIT=y and CONFIG_BPF_SYSCALL=n to prevent userspace from loading arbitrary eBPF programs via the syscall.

By tying BPF_JIT directly to BPF_SYSCALL, this change forces these environments to either run filters in the interpreter or expose the bpf() syscall.

Additionally, does this force architectures that only implement HAVE_CBPF_JIT to enable BPF_SYSCALL, pulling in unused eBPF verifier and map infrastructure just to use their classic JIT compiler?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author maintainer
#3 lore raw

PATCH v13 2/6 bpf: add bpf_get_cpu_time_counter kfunc

In reply to Puranjay Mohan

From: Vadim Fedorenko <[email protected]>

New kfunc to return ARCH-specific timecounter. The main reason to implement this kfunc is to avoid extra overhead of benchmark measurements, which are usually done by a pair of bpf_ktime_get_ns() at the beginnig and at the end of the code block under benchmark. When fully JITed this function doesn't implement conversion to the monotonic clock and saves some CPU cycles by receiving timecounter values in single-digit amount of instructions. The delta values can be translated into nanoseconds using kfunc introduced in the next patch. For x86_64 BPF JIT converts this kfunc into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to get CLOCK_MONOTONIC_RAW value in ns.

JIT version of the function uses "LFENCE; RDTSC" variant because it doesn't care about cookie value returned by "RDTSCP" and it doesn't want to trash RCX value. LFENCE option provides the same ordering guarantee as RDTSCP variant.

The simplest use-case is added in 5th patch, where we calculate the time spent by bpf_get_ns_current_pid_tgid() kfunc. More complex example is to use session cookie to store timecounter value at kprobe/uprobe using kprobe.session/uprobe.session, and calculate the difference at kretprobe/uretprobe.

---
 arch/x86/net/bpf_jit_comp.c   | 45 +++++++++++++++++++++++++++++++++++
 arch/x86/net/bpf_jit_comp32.c |  1 +
 include/linux/bpf.h           |  3 +++
 include/linux/filter.h        |  1 +
 kernel/bpf/core.c             | 11 +++++++++
 kernel/bpf/helpers.c          | 11 +++++++++
 kernel/bpf/verifier.c         |  4 +++-
 7 files changed, 75 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +45 -0
index ea9e707e8abf..7cda5589107b 100644
@@ -16,6 +16,7 @@
 #include <asm/ftrace.h>
 #include <asm/set_memory.h>
 #include <asm/nospec-branch.h>
+#include <asm/timer.h>
 #include <asm/text-patching.h>
 #include <asm/unwind.h>
 #include <asm/cfi.h>
@@ -2445,6 +2446,40 @@ st:			if (is_imm8(insn->off))
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			u8 *ip = image + addrs[i - 1];
+
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+			    bpf_jit_inlines_kfunc_call(imm32)) {
+				/* The default implementation of this kfunc uses
+				 * ktime_get_raw_ns() which effectively is implemented as
+				 * `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip
+				 * masking part because we assume it's not needed in BPF
+				 * use case (two measurements close in time).
+				 * Original code for rdtsc_ordered() uses sequence:
+				 * 'rdtsc; nop; nop; nop' to patch it into
+				 * 'lfence; rdtsc' or 'rdtscp' depending on CPU features.
+				 * JIT uses 'lfence; rdtsc' variant because BPF program
+				 * doesn't care about cookie provided by rdtscp in RCX.
+				 * Save RDX because RDTSC will use EDX:EAX to return u64
+				 */
+				emit_mov_reg(&prog, true, AUX_REG, BPF_REG_3);
+				if (cpu_feature_enabled(X86_FEATURE_LFENCE_RDTSC))
+					EMIT_LFENCE();
+				EMIT2(0x0F, 0x31);
+
+				/* shl RDX, 32 */
+				maybe_emit_1mod(&prog, BPF_REG_3, true);
+				EMIT3(0xC1, add_1reg(0xE0, BPF_REG_3), 32);
+				/* or RAX, RDX */
+				maybe_emit_mod(&prog, BPF_REG_0, BPF_REG_3, true);
+				EMIT2(0x09, add_2reg(0xC0, BPF_REG_0, BPF_REG_3));
+				/* restore RDX from R11 */
+				emit_mov_reg(&prog, true, BPF_REG_3, AUX_REG);
+
+				break;
+			}
+
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -4081,3 +4116,13 @@ bool bpf_jit_supports_fsession(void)
 {
 	return true;
 }
+
+/* x86-64 JIT can inline kfunc */
+bool bpf_jit_inlines_kfunc_call(s32 imm)
+{
+	if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+	    cpu_feature_enabled(X86_FEATURE_TSC) &&
+	    using_native_sched_clock() && sched_clock_stable())
+		return true;
+	return false;
+}
arch/x86/net/bpf_jit_comp32.c +1 -0
index 852baf2e4db4..ca208378c979 100644
@@ -16,6 +16,7 @@
 #include <asm/set_memory.h>
 #include <asm/nospec-branch.h>
 #include <asm/asm-prototypes.h>
+#include <asm/timer.h>
 #include <linux/bpf.h>
 
 /*
include/linux/bpf.h +3 -0
index b4b703c90ca9..74abf2b639fd 100644
@@ -3744,6 +3744,9 @@ void bpf_user_rnd_init_once(void);
 u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
 u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
 
+/* Inlined kfuncs */
+u64 bpf_get_cpu_time_counter(void);
+
 #if defined(CONFIG_NET)
 bool bpf_sock_common_is_valid_access(int off, int size,
 				     enum bpf_access_type type,
include/linux/filter.h +1 -0
index 1ec6d5ba64cc..4924c0bab1a0 100644
@@ -1159,6 +1159,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 void bpf_jit_compile(struct bpf_prog *prog);
 bool bpf_jit_needs_zext(void);
 bool bpf_jit_inlines_helper_call(s32 imm);
+bool bpf_jit_inlines_kfunc_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
kernel/bpf/core.c +11 -0
index 8b018ff48875..e06e750ee16f 100644
@@ -3201,6 +3201,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm)
 	return false;
 }
 
+/* Return true if the JIT inlines the call to the kfunc corresponding to
+ * the imm.
+ *
+ * The verifier will not patch the insn->imm for the call to the helper if
+ * this returns true.
+ */
+bool __weak bpf_jit_inlines_kfunc_call(s32 imm)
+{
+	return false;
+}
+
 /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */
 bool __weak bpf_jit_supports_subprog_tailcalls(void)
 {
kernel/bpf/helpers.c +11 -0
index 2bb60200c266..dfe280440120 100644
@@ -4664,6 +4664,16 @@ __bpf_kfunc int bpf_timer_cancel_async(struct bpf_timer *timer)
 	}
 }
 
+__bpf_kfunc u64 bpf_get_cpu_time_counter(void)
+{
+	/* CLOCK_MONOTONIC_RAW is the closest analogue to what is implemented
+	 * in JIT. The access time is the same as for CLOCK_MONOTONIC, but the
+	 * slope of 'raw' is not affected by NTP adjustments, and with stable
+	 * TSC it can provide less jitter in short term measurements.
+	 */
+	return ktime_get_raw_fast_ns();
+}
+
 __bpf_kfunc_end_defs();
 
 static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
@@ -4859,6 +4869,7 @@ BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS)
 BTF_ID_FLAGS(func, bpf_dynptr_from_file)
 BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
 BTF_ID_FLAGS(func, bpf_timer_cancel_async)
+BTF_ID_FLAGS(func, bpf_get_cpu_time_counter)
 BTF_KFUNCS_END(common_btf_ids)
 
 static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/verifier.c +3 -1
index 69d75515ed3f..b8d26e1bff48 100644
@@ -17035,7 +17035,9 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 			/* error would be reported later */
 			return false;
 		cs->num_params = btf_type_vlen(meta.func_proto);
-		cs->fastcall = meta.kfunc_flags & KF_FASTCALL;
+		cs->fastcall = (meta.kfunc_flags & KF_FASTCALL) ||
+			       (meta.btf == btf_vmlinux &&
+				bpf_jit_inlines_kfunc_call(call->imm));
 		cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type));
 		return true;
 	}
Signature · 1 line
2.52.0
#4 lore raw

Re: [PATCH bpf-next v13 2/6] bpf: add bpf_get_cpu_time_counter kfunc

Sashiko AI review found 3 potential issue(s): - [High] Type confusion in `bpf_jit_inlines_kfunc_call` breaks fastcall optimization and hides a TOCTOU register corruption vulnerability. - [High] The…

Sashiko AI review found 3 potential issue(s):
- [High] Type confusion in bpf_jit_inlines_kfunc_call breaks fastcall optimization and hides a TOCTOU register corruption vulnerability.
- [High] The bpf_get_cpu_time_counter() kfunc returns inconsistent physical units (CPU cycles vs. nanoseconds) depending on JIT execution.
- [Medium] Missing memory barrier for RDTSC allows speculative out-of-order execution on certain CPUs.
--

commit e2ed941037816a39f712a14ccb3383436319583d
Author: Vadim Fedorenko <[email protected]>

bpf: add bpf_get_cpu_time_counter kfunc

This commit adds a bpf_get_cpu_time_counter kfunc to return an ARCH-specific timecounter, bypassing the usual ktime_get_ns() overhead. It implements an inlined x86_64 JIT version using RDTSC, and falls back to ktime_get_raw_fast_ns() in C.

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
index ea9e707e8abff..7cda5589107bf 100644

[ ... ]

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2456

Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +8 -0
@@ -2445,6 +2446,40 @@ st:			if (is_imm8(insn->off))
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			u8 *ip = image + addrs[i - 1];
+
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    imm32 == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+			    bpf_jit_inlines_kfunc_call(imm32)) {
+				/* The default implementation of this kfunc uses
+				 * ktime_get_raw_ns() which effectively is implemented as
+				 * `(u64)rdtsc_ordered() & S64_MAX`. For JIT We skip

Does this JIT implementation return inconsistent time units compared to the C fallback?

The inline RDTSC emits hardware CPU cycles, but the fallback bpf_get_cpu_time_counter() calls ktime_get_raw_fast_ns(), which returns nanoseconds. If a BPF program calculates a delta using the fallback path and later converts it, won't the nanosecond value be multiplied again, resulting in incorrect metrics?

[ ... ]

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2469

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4 -0
+				emit_mov_reg(&prog, true, AUX_REG, BPF_REG_3);
+				if (cpu_feature_enabled(X86_FEATURE_LFENCE_RDTSC))
+					EMIT_LFENCE();
+				EMIT2(0x0F, 0x31);

The kernel's standard rdtsc_ordered() implementation ensures serialization by checking both X86_FEATURE_LFENCE_RDTSC and X86_FEATURE_MFENCE_RDTSC.

Does this code need to check for MFENCE as well? On older AMD CPUs that rely on MFENCE, omitting it could result in out-of-order execution across the measured code block.

[ ... ]

Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +0 -0
@@ -4081,3 +4116,13 @@ bool bpf_jit_supports_fsession(void)
 {
 	return true;
 }

arch/x86/net/bpf_jit_comp.c bool bpf_jit_supports_fsession(void) · line 4128

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+
+/* x86-64 JIT can inline kfunc */
+bool bpf_jit_inlines_kfunc_call(s32 imm)
+{
+	if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+	    cpu_feature_enabled(X86_FEATURE_TSC) &&
+	    using_native_sched_clock() && sched_clock_stable())
+		return true;
+	return false;
+}

Is it safe to evaluate sched_clock_stable() dynamically for fastcall optimizations?

If sched_clock_stable() returns true during verification, the verifier could strip the caller-saved register spills. If the clock later transitions to false before JIT compilation, the JIT will emit a standard C call instead of inlining, which could silently clobber BPF registers R1-R5.

[ ... ]

Quoted 4 lines
kernel/bpf/verifier.c +0 -0
index 69d75515ed3f7..b8d26e1bff481 100644

[ ... ]

kernel/bpf/verifier.c bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, · line 17043

Quoted 11 lines
kernel/bpf/verifier.c +3 -1
@@ -17035,7 +17035,9 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
 			/* error would be reported later */
 			return false;
 		cs->num_params = btf_type_vlen(meta.func_proto);
-		cs->fastcall = meta.kfunc_flags & KF_FASTCALL;
+		cs->fastcall = (meta.kfunc_flags & KF_FASTCALL) ||
+			       (meta.btf == btf_vmlinux &&
+				bpf_jit_inlines_kfunc_call(call->imm));
 		cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type));
 		return true;
 	}

In the verifier, call->imm holds the kfunc's BTF ID, but the x86 JIT implementation of bpf_jit_inlines_kfunc_call() compares it against BPF_CALL_IMM(bpf_get_cpu_time_counter), which is a relative memory offset.

Since a BTF ID will never match this memory offset, won't this always return false during verification, defeating the fastcall optimization?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author maintainer
#5 lore raw

PATCH v13 3/6 bpf: add bpf_cpu_time_counter_to_ns kfunc

In reply to Puranjay Mohan

From: Vadim Fedorenko <[email protected]>

The new kfunc should be used to convert deltas of values received by bpf_get_cpu_time_counter() into nanoseconds. It is not designed to do full conversion of time counter values to CLOCK_MONOTONIC_RAW nanoseconds and cannot guarantee monotonicity of 2 independent values, but rather to convert the difference of 2 close enough values of CPU timestamp counter into nanoseconds.

This function is JITted into just several instructions and adds as low overhead as possible and perfectly suits benchmark use-cases.

When the kfunc is not JITted it returns the value provided as argument because the kfunc in previous patch will return values in nanoseconds and can be optimized by verifier.

---
 arch/x86/net/bpf_jit_comp.c   | 29 ++++++++++++++++++++++++++++-
 arch/x86/net/bpf_jit_comp32.c |  1 +
 include/linux/bpf.h           |  1 +
 kernel/bpf/helpers.c          |  6 ++++++
 kernel/bpf/verifier.c         | 10 +++++++---
 5 files changed, 43 insertions(+), 4 deletions(-)
arch/x86/net/bpf_jit_comp.c +28 -1
index 7cda5589107b..a8956eb867ef 100644
@@ -10,6 +10,7 @@
 #include <linux/if_vlan.h>
 #include <linux/bitfield.h>
 #include <linux/bpf.h>
+#include <linux/clocksource.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
 #include <asm/extable.h>
@@ -2480,6 +2481,31 @@ st:			if (is_imm8(insn->off))
 				break;
 			}
 
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) &&
+			    bpf_jit_inlines_kfunc_call(imm32)) {
+				struct cyc2ns_data data;
+				u32 mult, shift;
+
+				/* stable TSC runs with fixed frequency and
+				 * transformation coefficients are also fixed
+				 */
+				cyc2ns_read_begin(&data);
+				mult = data.cyc2ns_mul;
+				shift = data.cyc2ns_shift;
+				cyc2ns_read_end();
+				/* imul RAX, RDI, mult */
+				maybe_emit_mod(&prog, BPF_REG_1, BPF_REG_0, true);
+				EMIT2_off32(0x69, add_2reg(0xC0, BPF_REG_1, BPF_REG_0),
+					    mult);
+
+				/* shr RAX, shift (which is less than 64) */
+				maybe_emit_1mod(&prog, BPF_REG_0, true);
+				EMIT3(0xC1, add_1reg(0xE8, BPF_REG_0), shift);
+
+				break;
+			}
+
 			func = (u8 *) __bpf_call_base + imm32;
 			if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -4120,7 +4146,8 @@ bool bpf_jit_supports_fsession(void)
 /* x86-64 JIT can inline kfunc */
 bool bpf_jit_inlines_kfunc_call(s32 imm)
 {
-	if (imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+	if ((imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) ||
+	     imm == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns)) &&
 	    cpu_feature_enabled(X86_FEATURE_TSC) &&
 	    using_native_sched_clock() && sched_clock_stable())
 		return true;
arch/x86/net/bpf_jit_comp32.c +1 -0
index ca208378c979..da61bc5585aa 100644
@@ -12,6 +12,7 @@
 #include <linux/netdevice.h>
 #include <linux/filter.h>
 #include <linux/if_vlan.h>
+#include <linux/clocksource.h>
 #include <asm/cacheflush.h>
 #include <asm/set_memory.h>
 #include <asm/nospec-branch.h>
include/linux/bpf.h +1 -0
index 74abf2b639fd..d523168b8998 100644
@@ -3746,6 +3746,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
 
 /* Inlined kfuncs */
 u64 bpf_get_cpu_time_counter(void);
+u64 bpf_cpu_time_counter_to_ns(u64 counter);
 
 #if defined(CONFIG_NET)
 bool bpf_sock_common_is_valid_access(int off, int size,
kernel/bpf/helpers.c +6 -0
index dfe280440120..bc7f5ccac761 100644
@@ -4674,6 +4674,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void)
 	return ktime_get_raw_fast_ns();
 }
 
+__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 counter)
+{
+	return counter;
+}
+
 __bpf_kfunc_end_defs();
 
 static void bpf_task_work_cancel_scheduled(struct irq_work *irq_work)
@@ -4870,6 +4875,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_from_file)
 BTF_ID_FLAGS(func, bpf_dynptr_file_discard)
 BTF_ID_FLAGS(func, bpf_timer_cancel_async)
 BTF_ID_FLAGS(func, bpf_get_cpu_time_counter)
+BTF_ID_FLAGS(func, bpf_cpu_time_counter_to_ns, KF_FASTCALL)
 BTF_KFUNCS_END(common_btf_ids)
 
 static const struct btf_kfunc_id_set common_kfunc_set = {
kernel/bpf/verifier.c +7 -3
index b8d26e1bff48..5341dc6d29ca 100644
@@ -11178,8 +11178,8 @@ enum special_kfunc_type {
 	KF_bpf_session_is_return,
 	KF_bpf_stream_vprintk,
 	KF_bpf_stream_print_stack,
+	KF_bpf_cpu_time_counter_to_ns,
 };
-
 BTF_ID_LIST(special_kfunc_list)
 BTF_ID(func, bpf_obj_new_impl)
 BTF_ID(func, bpf_obj_new)
@@ -11266,6 +11266,7 @@ BTF_ID(func, bpf_arena_reserve_pages)
 BTF_ID(func, bpf_session_is_return)
 BTF_ID(func, bpf_stream_vprintk)
 BTF_ID(func, bpf_stream_print_stack)
+BTF_ID(func, bpf_cpu_time_counter_to_ns)
 
 static bool is_bpf_obj_new_kfunc(u32 func_id)
 {
@@ -18629,7 +18630,6 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
 }
 
 
-
 static void free_states(struct bpf_verifier_env *env)
 {
 	struct bpf_verifier_state_list *sl;
@@ -19791,6 +19791,9 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 	if (!bpf_jit_supports_far_kfunc_call())
 		insn->imm = BPF_CALL_IMM(desc->addr);
+	/* if JIT will inline kfunc verifier shouldn't change the code */
+	if (bpf_jit_inlines_kfunc_call(insn->imm))
+		return 0;
 
 	if (is_bpf_obj_new_kfunc(desc->func_id) || is_bpf_percpu_obj_new_kfunc(desc->func_id)) {
 		struct btf_struct_meta *kptr_struct_meta = env->insn_aux_data[insn_idx].kptr_struct_meta;
@@ -19851,7 +19854,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		__fixup_collection_insert_kfunc(&env->insn_aux_data[insn_idx], struct_meta_reg,
 						node_offset_reg, insn, insn_buf, cnt);
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
-		   desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) {
+		   desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast] ||
+		   desc->func_id == special_kfunc_list[KF_bpf_cpu_time_counter_to_ns]) {
 		insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
 		*cnt = 1;
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_session_is_return] &&
Signature · 1 line
2.52.0
#6 lore raw

Re: [PATCH bpf-next v13 3/6] bpf: add bpf_cpu_time_counter_to_ns kfunc

This isn't a bug, but the include appears to be unused in bpf_jit_comp32.c. During review of v11 on the mailing list, Alexei Starovoitov noted that the 32-bit JIT implementation was questionable and…
Quoted 12 lines
arch/x86/net/bpf_jit_comp.c +1 -0
index 7cda5589107b..a8956eb867ef 100644
@@ -10,6 +10,7 @@
 #include <linux/if_vlan.h>
 #include <linux/bitfield.h>
 #include <linux/bpf.h>
+#include <linux/clocksource.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
 #include <asm/extable.h>

[ ... ]

arch/x86/net/bpf_jit_comp32.c line 18

Quoted 12 lines
arch/x86/net/bpf_jit_comp32.c +1 -0
index ca208378c979..da61bc5585aa 100644
@@ -12,6 +12,7 @@
 #include <linux/netdevice.h>
 #include <linux/filter.h>
 #include <linux/if_vlan.h>
+#include <linux/clocksource.h>
 #include <asm/cacheflush.h>
 #include <asm/set_memory.h>
 #include <asm/nospec-branch.h>

This isn't a bug, but the include appears to be unused in bpf_jit_comp32.c. During review of v11 on the mailing list, Alexei Starovoitov noted that the 32-bit JIT implementation was questionable and should be dropped entirely:

https://lore.kernel.org/bpf/CAADnVQLYT5SV+tS2ycLteBMYOc12C=X7iHZ=RjhyVzuY=6=8Uw@mail.gmail.com/

The 32-bit JIT implementation was indeed removed, but this header include remained. Would it make sense to remove the unused include from bpf_jit_comp32.c?

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24605755621

#7 lore raw

Re: [PATCH bpf-next v13 3/6] bpf: add bpf_cpu_time_counter_to_ns kfunc

In reply to Puranjay Mohan
Sashiko AI review found 3 potential issue(s): - [Critical] JIT-emitted IMUL instruction sign-extends the 32-bit cyc2ns_mul multiplier, corrupting the nanosecond conversion. - [High] 64-bit…

Sashiko AI review found 3 potential issue(s):
- [Critical] JIT-emitted IMUL instruction sign-extends the 32-bit cyc2ns_mul multiplier, corrupting the nanosecond conversion.
- [High] 64-bit truncation in JITted bpf_cpu_time_counter_to_ns causes severe time overflow for intervals larger than ~4.3 seconds.
- [High] API unit inconsistency and TOCTOU race on sched_clock_stable() cause silent unit corruption.
--

commit d48481a8aafecd7a1cb43ba6a7f2840fa2d497e3
Author: Vadim Fedorenko <[email protected]>
Subject: bpf: add bpf_cpu_time_counter_to_ns kfunc

Adds a bpf_cpu_time_counter_to_ns() kfunc to convert CPU cycle deltas into nanoseconds. The kfunc is JIT inlined on x86 for low overhead, leveraging cyc2ns multiplication and shift values.

Quoted 3 lines
arch/x86/net/bpf_jit_comp.c +0 -0

[ ... ]

Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +7 -0
@@ -2480,6 +2481,31 @@ st:			if (is_imm8(insn->off))
 				break;
 			}
 
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    imm32 == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) &&
+			    bpf_jit_inlines_kfunc_call(imm32)) {
+				struct cyc2ns_data data;
+				u32 mult, shift;
+
+				/* stable TSC runs with fixed frequency and

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2500

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +10 -0
+				 * transformation coefficients are also fixed
+				 */
+				cyc2ns_read_begin(&data);
+				mult = data.cyc2ns_mul;
+				shift = data.cyc2ns_shift;
+				cyc2ns_read_end();
+				/* imul RAX, RDI, mult */
+				maybe_emit_mod(&prog, BPF_REG_1, BPF_REG_0, true);
+				EMIT2_off32(0x69, add_2reg(0xC0, BPF_REG_1, BPF_REG_0),
+					    mult);

Can this IMUL instruction corrupt the multiplication? The IMUL r64, r/m64, imm32 instruction (opcode 0x69) sign-extends the 32-bit immediate to 64 bits. Since data.cyc2ns_mul is heavily scaled to retain precision, it is frequently >= 0x80000000.

When cyc2ns_mul >= 0x80000000, it would be sign-extended to a negative 64-bit value, drastically altering the product when multiplied with the unsigned cycle count. Would it be safer to zero-extend the multiplier by loading it into a scratch register first?

Will this 64-bit multiplication overflow for longer time intervals? The IMUL r64, r/m64, imm32 instruction computes a 64x64-bit multiplication but only stores the lower 64 bits of the product in the destination register.

Because cyc2ns_mul is scaled up for precision, the product could overflow the 64-bit register when the cycle count exceeds roughly 2^33 (around 4.3 seconds on a typical 2 GHz CPU). Could this be avoided by using the MUL r/m64 instruction to compute the full 128-bit product in RDX:RAX followed by SHRD?

Quoted 7 lines
arch/x86/net/bpf_jit_comp.c +7 -0
+
+				/* shr RAX, shift (which is less than 64) */
+				maybe_emit_1mod(&prog, BPF_REG_0, true);
+				EMIT3(0xC1, add_1reg(0xE8, BPF_REG_0), shift);
+
+				break;
+			}

[ ... ]

kernel/bpf/helpers.c __bpf_kfunc u64 bpf_get_cpu_time_counter(void) · line 4680

Quoted 11 lines
kernel/bpf/helpers.c +4 -0
@@ -4674,6 +4674,11 @@ __bpf_kfunc u64 bpf_get_cpu_time_counter(void)
 	return ktime_get_raw_fast_ns();
 }
 
+__bpf_kfunc u64 bpf_cpu_time_counter_to_ns(u64 counter)
+{
+	return counter;
+}

Does this introduce an API unit inconsistency? bpf_get_cpu_time_counter() returns raw cycles when JIT inlining is active on x86, but falls back to ktime_get_raw_fast_ns() (nanoseconds) when JIT is disabled or on other architectures. BPF programs exporting this value might output different units depending on the environment.

Quoted 10 lines
kernel/bpf/verifier.c +3 -0
@@ -19791,6 +19791,9 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 
 	if (!bpf_jit_supports_far_kfunc_call())
 		insn->imm = BPF_CALL_IMM(desc->addr);
+	/* if JIT will inline kfunc verifier shouldn't change the code */
+	if (bpf_jit_inlines_kfunc_call(insn->imm))
+		return 0;

[ ... ]

kernel/bpf/verifier.c int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · line 19860

Quoted 9 lines
kernel/bpf/verifier.c +2 -1
@@ -19851,7 +19854,8 @@ int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		__fixup_collection_insert_kfunc(&env->insn_aux_data[insn_idx], struct_meta_reg,
 						node_offset_reg, insn, insn_buf, cnt);
 	} else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
-		   desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) {
+		   desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast] ||
+		   desc->func_id == special_kfunc_list[KF_bpf_cpu_time_counter_to_ns]) {
 		insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1);
 		*cnt = 1;

Is there a race condition here involving sched_clock_stable()? bpf_jit_inlines_kfunc_call() reads sched_clock_stable() which can change dynamically.

If sched_clock_stable() returns false during verification, bpf_fixup_kfunc_call() will replace bpf_cpu_time_counter_to_ns with a MOV instruction. If it transitions to true before the JIT phase, bpf_get_cpu_time_counter() will be inlined to emit RDTSC. In that scenario, the program would receive raw cycles from the get kfunc but use a no-op for the conversion, silently treating raw cycles as nanoseconds.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author maintainer
#8 lore raw

PATCH v13 4/6 selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit

In reply to Puranjay Mohan

From: Vadim Fedorenko <[email protected]>

bpf_get_cpu_time_counter() is replaced with rdtsc instruction on x86_64. Add tests to check that JIT works as expected. When JIT is not supported, bpf_cpu_time_counter_to_ns() can be inlined by verifier.

---
 .../selftests/bpf/prog_tests/verifier.c       |   2 +
 .../selftests/bpf/progs/verifier_cpu_cycles.c | 120 ++++++++++++++++++
 2 files changed, 122 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
index a96b25ebff23..775fe69b0948 100644
@@ -116,6 +116,7 @@
 #include "verifier_bits_iter.skel.h"
 #include "verifier_lsm.skel.h"
 #include "verifier_jit_inline.skel.h"
+#include "verifier_cpu_cycles.skel.h"
 #include "irq.skel.h"
 #include "verifier_ctx_ptr_param.skel.h"
 
@@ -265,6 +266,7 @@ void test_irq(void)			      { RUN(irq); }
 void test_verifier_mtu(void)		      { RUN(verifier_mtu); }
 void test_verifier_jit_inline(void)               { RUN(verifier_jit_inline); }
 void test_verifier_ctx_ptr_param(void)       { RUN(verifier_ctx_ptr_param); }
+void test_verifier_cpu_cycles(void)	      { RUN(verifier_cpu_cycles); }
 
 static int init_test_val_map(struct bpf_object *obj, char *map_name)
 {
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +120 -0

120 lines not on this page · raw message

Signature · 1 line
2.52.0
#9 lore raw

Re: [PATCH bpf-next v13 4/6] selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit

Sashiko AI review found 3 potential issue(s): - [High] The selftests rigidly enforce a broken x86-64 JIT implementation for `bpf_cpu_time_counter_to_ns()`. - [Medium] Fragile tests dependent on…

Sashiko AI review found 3 potential issue(s):
- [High] The selftests rigidly enforce a broken x86-64 JIT implementation for bpf_cpu_time_counter_to_ns().
- [Medium] Fragile tests dependent on hardware features will cause CI flakiness.
- [Low] Mismatched and inaccurate comments due to copy-pasting.
--

commit 01407c2c12560c811a29d35f221844615da2962d
Author: Vadim Fedorenko <[email protected]>

selftests/bpf: add selftest to check bpf_get_cpu_time_counter jit

This commit adds selftests for the JIT implementations of bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() on x86_64 and arm64. It verifies the emitted assembly using __jited and __xlated macros.

Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +0 -0
new file mode 100644
index 0000000000000..26c02010ccf1f

[ ... ]

Quoted 4 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +4 -0
+SEC("syscall")
+__arch_x86_64
+/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */
+__jited("	endbr64")

tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c line 38

Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +10 -0
+__jited("	nopl	(%rax,%rax)")
+__jited("	nopl	(%rax)")
+__jited("	pushq	%rbp")
+__jited("	movq	%rsp, %rbp")
+__jited("	endbr64")
+/* save RDX in R11 as it will be overwritten */
+__jited("	movq	%rdx, %r11")
+/* lfence may not be executed depending on cpu features */
+__jited("	{{(lfence|)}}")
+__jited("	rdtsc")

Will this test deterministically fail in environments where sched_clock_stable() is false or the CPU lacks the TSC feature?

The test strictly asserts that rdtsc is emitted. However, the underlying JIT falls back to a standard function call if those hardware conditions are not met, which might cause test failures in some CI and virtualization environments.

[ ... ]

tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c line 27

Quoted 3 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +3 -0
+SEC("syscall")
+__arch_x86_64
+/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */

This isn't a bug, but does this comment reference the wrong function name due to a copy-paste error?

tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c line 98

Quoted 8 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +8 -0
+__jited("	endbr64")
+__jited("	nopl	(%rax,%rax)")
+__jited("	nopl	(%rax)")
+__jited("	pushq	%rbp")
+__jited("	movq	%rsp, %rbp")
+__jited("	endbr64")
+/* save RDX in R11 as it will be overwritten */
+__jited("	movabsq	$0x2a2a2a2a2a, %rdi")

This isn't a bug, but the comment mentions saving RDX in R11, while the following instruction does not touch either of those registers.

tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c line 100

Quoted 2 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +2 -0
+__jited("	imulq	${{.*}}, %rdi, %rax")
+__jited("	shrq	${{.*}}, %rax")

Does enforcing this specific JIT sequence codify broken math for large cycle counts?

The x86 imulq instruction computes a 64-bit product, truncating the upper bits. Since cycle counts are 64-bit and multipliers can be large, their product often requires 128 bits. The truncation would result in silent wraparound.

Additionally, using a 32-bit immediate with imulq sign-extends the value to 64 bits. If the multiplier has its most significant bit set, it will be treated as a large negative number, corrupting the result.

Finally, the emitted sequence does not add the offset. On arm64, the test expects a no-op since the counter returns absolute nanoseconds, but on x86-64 without the offset, absolute counter conversions will return incorrect values. Does this test assert a cross-architecture inconsistency?

Quoted 2 lines
tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file +2 -0
+__jited("	leave")
+__naked int bpf_cyc2ns_jit_x86(void)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author maintainer
#10 lore raw

PATCH v13 5/6 selftests/bpf: add usage example for cpu time counter kfuncs

In reply to Puranjay Mohan

From: Vadim Fedorenko <[email protected]>

The selftest provides an example of how to measure the latency of bpf kfunc/helper call using time stamp counter and how to convert measured value into nanoseconds.

---
 .../bpf/prog_tests/test_cpu_cycles.c          | 35 +++++++++++++++++++
 .../selftests/bpf/progs/test_cpu_cycles.c     | 25 +++++++++++++
 2 files changed, 60 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c
tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file +35 -0
new file mode 100644
index 000000000000..067307f0c4c2
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Inc. */
+
+#include <test_progs.h>
+#include "test_cpu_cycles.skel.h"
+
+static void cpu_cycles(void)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct test_cpu_cycles *skel;
+	int err, pfd;
+
+	skel = test_cpu_cycles__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "test_cpu_cycles open and load"))
+		return;
+
+	pfd = bpf_program__fd(skel->progs.bpf_cpu_cycles);
+	if (!ASSERT_GT(pfd, 0, "test_cpu_cycles fd"))
+		goto fail;
+
+	err = bpf_prog_test_run_opts(pfd, &opts);
+	if (!ASSERT_OK(err, "test_cpu_cycles test run"))
+		goto fail;
+
+	ASSERT_NEQ(skel->bss->cycles, 0, "test_cpu_cycles 0 cycles");
+	ASSERT_NEQ(skel->bss->ns, 0, "test_cpu_cycles 0 ns");
+fail:
+	test_cpu_cycles__destroy(skel);
+}
+
+void test_cpu_cycles(void)
+{
+	if (test__start_subtest("cpu_cycles"))
+		cpu_cycles();
+}
tools/testing/selftests/bpf/progs/test_cpu_cycles.c new file +25 -0
new file mode 100644
index 000000000000..3c428f3be831
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Meta Inc. */
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+
+extern u64 bpf_cpu_time_counter_to_ns(u64 cycles) __weak __ksym;
+extern u64 bpf_get_cpu_time_counter(void) __weak __ksym;
+
+__u64 cycles, ns;
+
+SEC("syscall")
+int bpf_cpu_cycles(void)
+{
+	struct bpf_pidns_info pidns;
+	__u64 start;
+
+	start = bpf_get_cpu_time_counter();
+	bpf_get_ns_current_pid_tgid(0, 0, &pidns, sizeof(struct bpf_pidns_info));
+	cycles = bpf_get_cpu_time_counter() - start;
+	ns = bpf_cpu_time_counter_to_ns(cycles);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
Signature · 1 line
2.52.0
#11 lore raw

Re: [PATCH bpf-next v13 5/6] selftests/bpf: add usage example for cpu time counter kfuncs

Sashiko AI review found 1 potential issue(s): - [Medium] Incorrect and redundant file descriptor validation in BPF skeleton test. --

Sashiko AI review found 1 potential issue(s):
- [Medium] Incorrect and redundant file descriptor validation in BPF skeleton test. --

commit 0a7806f6db918db501677b21fa0e15ae508d1ada
Author: Vadim Fedorenko <[email protected]>

selftests/bpf: add usage example for cpu time counter kfuncs

This commit adds a selftest example demonstrating how to measure the latency of BPF kfuncs or helpers using the time stamp counter and how to convert the measured values into nanoseconds.

Quoted 5 lines
tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file +0 -0
new file mode 100644
index 0000000000000..067307f0c4c29

[ ... ]

tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c line 19

Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file +13 -0
+static void cpu_cycles(void)
+{
+	LIBBPF_OPTS(bpf_test_run_opts, opts);
+	struct test_cpu_cycles *skel;
+	int err, pfd;
+
+	skel = test_cpu_cycles__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "test_cpu_cycles open and load"))
+		return;
+
+	pfd = bpf_program__fd(skel->progs.bpf_cpu_cycles);
+	if (!ASSERT_GT(pfd, 0, "test_cpu_cycles fd"))
+		goto fail;

Is it necessary to validate the file descriptor here? The BPF skeleton API guarantees that after a successful open_and_load(), all programs and maps are created with valid file descriptors, making this validation redundant.

Additionally, if a file descriptor check were needed, shouldn't it use ASSERT_GE rather than ASSERT_GT? 0 is technically a valid file descriptor, so if pfd evaluated to 0, this test would falsely fail.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
author maintainer
#12 lore raw

PATCH v13 6/6 bpf, arm64: Add JIT support for cpu time counter kfuncs

In reply to Puranjay Mohan

Add ARM64 JIT inlining for bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() kfuncs.

bpf_get_cpu_time_counter() is JIT-inlined as:

ISB // serialize instruction stream
MRS Xn, CNTVCT_EL0 // read architected timer counter

The ISB before the MRS is required for ordering, matching the kernel's arch_timer_read_cntvct_el0() implementation.

On newer CPUs it will be JITed to:

MRS Xn, CNTVCTSS_EL0 // self-synchronized (ISB not needed)

bpf_cpu_time_counter_to_ns() is JIT-inlined using mult/shift constants computed at JIT time from the architected timer frequency (CNTFRQ_EL0):

  MOV Xtmp, #mult           // load conversion multiplier
  MUL Xn, Xarg, Xtmp        // delta_ticks * mult
  LSR Xn, Xn, #shift        // >> shift = nanoseconds

On systems with a 1GHz counter (e.g., Neoverse-V2), mult=1 and shift=0, so the conversion collapses to a single MOV (identity).

---
 arch/arm64/include/asm/insn.h                 |  2 +
 arch/arm64/net/bpf_jit.h                      |  4 ++
 arch/arm64/net/bpf_jit_comp.c                 | 54 +++++++++++++++++++
 .../selftests/bpf/progs/verifier_cpu_cycles.c | 50 ++++++++++++++++-
 4 files changed, 109 insertions(+), 1 deletion(-)
arch/arm64/include/asm/insn.h +2 -0
index f463a654a2bb..bb235a39cef0 100644
@@ -139,6 +139,8 @@ enum aarch64_insn_system_register {
 	AARCH64_INSN_SYSREG_TPIDR_EL1	= 0x4684,
 	AARCH64_INSN_SYSREG_TPIDR_EL2	= 0x6682,
 	AARCH64_INSN_SYSREG_SP_EL0	= 0x4208,
+	AARCH64_INSN_SYSREG_CNTVCT_EL0		= 0x5F02,
+	AARCH64_INSN_SYSREG_CNTVCTSS_EL0	= 0x5F06,
 };
 
 enum aarch64_insn_variant {
arch/arm64/net/bpf_jit.h +4 -0
index d13de4222cfb..a525387439fe 100644
@@ -326,6 +326,10 @@
 	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL2)
 #define A64_MRS_SP_EL0(Rt) \
 	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_SP_EL0)
+#define A64_MRS_CNTVCT_EL0(Rt) \
+	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_CNTVCT_EL0)
+#define A64_MRS_CNTVCTSS_EL0(Rt) \
+	aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_CNTVCTSS_EL0)
 
 /* Barriers */
 #define A64_SB aarch64_insn_get_sb_value()
arch/arm64/net/bpf_jit_comp.c +54 -0

82 lines not on this page · raw message

tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c +49 -1

62 lines not on this page · raw message

Signature · 1 line
2.52.0
#13 lore raw

Re: [PATCH bpf-next v13 6/6] bpf, arm64: Add JIT support for cpu time counter kfuncs

Can mult be sign-extended here? Looking at the declaration of mult as u32, when it has bit 31 set (mult >= 0x80000000), the implicit conversion to s32 in emit_a64_mov_i (which takes 'const s32 val')…
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 0816c40fc7af..7da7507ab431 100644

[ ... ]

Quoted 35 lines
arch/arm64/net/bpf_jit_comp.c +27 -0
@@ -1571,10 +1573,54 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn
 	case BPF_JMP | BPF_CALL:
 	{
 		const u8 r0 = bpf2a64[BPF_REG_0];
+		const u8 r1 = bpf2a64[BPF_REG_1];
+		const s32 imm = insn->imm;
 		bool func_addr_fixed;
 		u64 func_addr;
 		u32 cpu_offset;
 
+		/* Inline kfunc bpf_get_cpu_time_counter() */
+		if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+		    imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+		    bpf_jit_inlines_kfunc_call(imm)) {
+			/*
+			 * With ECV (ARMv8.6+), CNTVCTSS_EL0 is self-
+			 * synchronizing — no ISB needed.  Without ECV,
+			 * an ISB is required before reading CNTVCT_EL0
+			 * to prevent speculative/out-of-order reads.
+			 *
+			 * Matches arch_timer_read_cntvct_el0().
+			 */
+			if (cpus_have_cap(ARM64_HAS_ECV)) {
+				emit(A64_MRS_CNTVCTSS_EL0(r0), ctx);
+			} else {
+				emit(A64_ISB, ctx);
+				emit(A64_MRS_CNTVCT_EL0(r0), ctx);
+			}
+			break;
+		}
+
+		/* Inline kfunc bpf_cpu_time_counter_to_ns() */
+		if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+		    imm == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) &&
+		    bpf_jit_inlines_kfunc_call(imm)) {

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1616

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+			u32 freq = arch_timer_get_cntfrq();
+
+			if (freq == NSEC_PER_SEC) {
+				/* 1 GHz counter: 1 tick = 1 ns, identity */
+				emit(A64_MOV(1, r0, r1), ctx);
+			} else {
+				u32 mult, shift;
+
+				clocks_calc_mult_shift(&mult, &shift, freq, NSEC_PER_SEC, 3600);
+				emit_a64_mov_i(1, tmp, mult, ctx);
                                                         ^^^^

Can mult be sign-extended here? Looking at the declaration of mult as u32, when it has bit 31 set (mult >= 0x80000000), the implicit conversion to s32 in emit_a64_mov_i (which takes 'const s32 val') makes the value negative.

In emit_a64_mov_i:

  u16 hi = val >> 16;      /* arithmetic shift when val is s32 negative */
  u16 lo = val & 0xffff;
  if (hi & 0x8000) {
    ...
    emit(A64_MOVN(is64, reg, (u16)~hi, 16), ctx);  /* is64=1 => 64-bit MOVN */
    ...
  }

For is64=1, MOVN sets the full 64-bit register to ~(imm << shift), so the upper 32 bits become 0xFFFFFFFF. For example, mult = 0xFA000000 would yield tmp = 0xFFFFFFFF_FA000000 instead of 0x00000000_FA000000. The subsequent 64-bit MUL would then multiply by this sign-extended value.

This can happen when clocks_calc_mult_shift produces mult >= 2^31. The algorithm in kernel/time/clocksource.c keeps sftacc at 32 whenever maxsec*from < 2^32, i.e., whenever arch_timer_get_cntfrq() returns a value less than about 1,193,046 Hz. In that regime mult can occupy the full u32 range, including values with bit 31 set (e.g., for freq == 1 MHz, mult = 0xFA000000).

While drivers/clocksource/arm_arch_timer.c only warns for arch_timer_rate < 1000000 and does not reject such rates, a platform configured with CNTFRQ_EL0 below about 1.2 MHz would execute this code path.

Other uses of emit_a64_mov_i(1, ...) in this file pass s32 offsets/imms where sign-extension is desired. For the u32 mult value zero-extension is required.

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1623

Quoted 7 lines
arch/arm64/net/bpf_jit_comp.c +7 -0
+				emit(A64_MUL(1, r0, r1, tmp), ctx);
+				if (shift)
+					emit(A64_LSR(1, r0, r0, shift), ctx);
+			}
+			break;
+		}
+

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/24605755621

#14 lore raw

Re: [PATCH bpf-next v13 6/6] bpf, arm64: Add JIT support for cpu time counter kfuncs

In reply to Puranjay Mohan
Sashiko AI review found 4 potential issue(s): - [Critical] Division by zero kernel panic during JIT compilation due to direct read of cntfrq_el0. - [High] Implicit sign-extension of the 32-bit…

Sashiko AI review found 4 potential issue(s):
- [Critical] Division by zero kernel panic during JIT compilation due to direct read of cntfrq_el0.
- [High] Implicit sign-extension of the 32-bit multiplier corrupts nanosecond conversions.
- [Medium] Unconditional JIT inlining of timer read bypasses hardware errata workarounds.
- [Medium] Silent 64-bit math overflow for duration measurements exceeding 1 hour. --

commit f282b55d9d2af467b0a51264b002bf14ff3648fc
Author: Puranjay Mohan <[email protected]>

bpf, arm64: Add JIT support for cpu time counter kfuncs

This commit adds ARM64 JIT inlining for the bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() kfuncs. It emits instructions to read the architected timer directly and perform the nanosecond conversion.

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0 -0
index 0816c40fc7af9..7da7507ab4315 100644
@@ -1571,10 +1573,54 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn

[ ... ]

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+		/* Inline kfunc bpf_get_cpu_time_counter() */
+		if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+		    imm == BPF_CALL_IMM(bpf_get_cpu_time_counter) &&
+		    bpf_jit_inlines_kfunc_call(imm)) {
+			/*
+			 * With ECV (ARMv8.6+), CNTVCTSS_EL0 is self-
+			 * synchronizing — no ISB needed.  Without ECV,
+			 * an ISB is required before reading CNTVCT_EL0
+			 * to prevent speculative/out-of-order reads.
+			 *

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1601

Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10 -0
+			 * Matches arch_timer_read_cntvct_el0().
+			 */
+			if (cpus_have_cap(ARM64_HAS_ECV)) {
+				emit(A64_MRS_CNTVCTSS_EL0(r0), ctx);
+			} else {
+				emit(A64_ISB, ctx);
+				emit(A64_MRS_CNTVCT_EL0(r0), ctx);
+			}
+			break;
+		}

Does this unconditional JIT inlining bypass the hardware errata workarounds?

The generic C implementation of bpf_get_cpu_time_counter() uses ktime_get_raw_fast_ns(), which safely respects the out-of-line workaround framework (CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND). By blindly emitting the MRS CNTVCT_EL0 (or CNTVCTSS_EL0) instructions, could BPF programs receive unstable and corrupted timestamps on affected hardware?

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1607

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+		/* Inline kfunc bpf_cpu_time_counter_to_ns() */
+		if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+		    imm == BPF_CALL_IMM(bpf_cpu_time_counter_to_ns) &&
+		    bpf_jit_inlines_kfunc_call(imm)) {
+			u32 freq = arch_timer_get_cntfrq();

Could this cause a division-by-zero kernel panic during JIT compilation?

On platforms with broken firmware, the cntfrq_el0 register is often left uninitialized (0), and the kernel relies on a device tree fallback. If arch_timer_get_cntfrq() returns 0, will clocks_calc_mult_shift() below execute a do_div(tmp, freq) and crash the system during BPF program load?

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1616

Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +8 -0
+			if (freq == NSEC_PER_SEC) {
+				/* 1 GHz counter: 1 tick = 1 ns, identity */
+				emit(A64_MOV(1, r0, r1), ctx);
+			} else {
+				u32 mult, shift;
+
+				clocks_calc_mult_shift(&mult, &shift, freq, NSEC_PER_SEC, 3600);
+				emit_a64_mov_i(1, tmp, mult, ctx);

Does emit_a64_mov_i() implicitly sign-extend the 32-bit multiplier?

The emit_a64_mov_i() function takes a signed s32 value as its argument. For systems with slower timer frequencies, clocks_calc_mult_shift() can compute a large unsigned 32-bit multiplier whose most significant bit is set. Will this cause the unsigned multiplier to be implicitly cast to a negative number and sign-extended to 64 bits, corrupting the subsequent multiplication?

arch/arm64/net/bpf_jit_comp.c static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn · line 1617

Quoted 1 line
arch/arm64/net/bpf_jit_comp.c +1 -0
+				emit(A64_MUL(1, r0, r1, tmp), ctx);

Will this simple 64-bit multiplication silently overflow for duration measurements exceeding one hour?

Since clocks_calc_mult_shift() is called with maxsec = 3600 (1 hour), the multiplier is calculated such that delta_ticks * mult fits within 64 bits up to that duration. For longer durations, could this simple 64-bit multiplication wrap around, returning falsely small or corrupted nanosecond durations, unlike the 128-bit math used in the generic kernel timekeeping?

Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +5 -0
+				if (shift)
+					emit(A64_LSR(1, r0, r0, shift), ctx);
+			}
+			break;
+		}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
reviewer
#15 lore raw

Re: [PATCH bpf-next v13 6/6] bpf, arm64: Add JIT support for cpu time counter kfuncs

In reply to Puranjay Mohan
Quoted 18 lines · Puranjay Mohan

Add ARM64 JIT inlining for bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() kfuncs.

bpf_get_cpu_time_counter() is JIT-inlined as:

ISB // serialize instruction stream
MRS Xn, CNTVCT_EL0 // read architected timer counter

The ISB before the MRS is required for ordering, matching the kernel's arch_timer_read_cntvct_el0() implementation.

On newer CPUs it will be JITed to:

MRS Xn, CNTVCTSS_EL0 // self-synchronized (ISB not needed)

bpf_cpu_time_counter_to_ns() is JIT-inlined using mult/shift constants computed at JIT time from the architected timer frequency (CNTFRQ_EL0):

Quoted 7 lines
   MOV Xtmp, #mult           // load conversion multiplier
   MUL Xn, Xarg, Xtmp        // delta_ticks * mult
   LSR Xn, Xn, #shift        // >> shift = nanoseconds

On systems with a 1GHz counter (e.g., Neoverse-V2), mult=1 and shift=0, so the conversion collapses to a single MOV (identity).

ISB triggers pipeline flush and instruction refetch. Since it is a really heavy operation, I think We should collect performance data to verify whether the jited instructions are actually faster than the original bpf_ktime_get_ns().

#16 lore raw

Re: [PATCH bpf-next v13 6/6] bpf, arm64: Add JIT support for cpu time counter kfuncs

Quoted 24 lines · Xu Kuohai
Quoted 22 lines · Puranjay Mohan

Add ARM64 JIT inlining for bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() kfuncs.

bpf_get_cpu_time_counter() is JIT-inlined as:

ISB // serialize instruction stream
MRS Xn, CNTVCT_EL0 // read architected timer counter

The ISB before the MRS is required for ordering, matching the kernel's arch_timer_read_cntvct_el0() implementation.

On newer CPUs it will be JITed to:

MRS Xn, CNTVCTSS_EL0 // self-synchronized (ISB not needed)

bpf_cpu_time_counter_to_ns() is JIT-inlined using mult/shift constants computed at JIT time from the architected timer frequency (CNTFRQ_EL0):

   MOV Xtmp, #mult           // load conversion multiplier
   MUL Xn, Xarg, Xtmp        // delta_ticks * mult
   LSR Xn, Xn, #shift        // >> shift = nanoseconds
Quoted 7 lines
Quoted 3 lines

On systems with a 1GHz counter (e.g., Neoverse-V2), mult=1 and shift=0, so the conversion collapses to a single MOV (identity).

ISB triggers pipeline flush and instruction refetch. Since it is a really heavy operation, I think We should collect performance data to verify whether the jited instructions are actually faster than the original bpf_ktime_get_ns().

Okay, I will get data, but shouldn't we still have it as future CPUs will have CNTVCTSS_EL0 and will not need ISB.

maintainer
#17 lore raw

Re: [PATCH bpf-next v13 6/6] bpf, arm64: Add JIT support for cpu time counter kfuncs

In reply to Puranjay Mohan

[+ Marc]

Quoted 10 lines · Puranjay Mohan

Add ARM64 JIT inlining for bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() kfuncs.

bpf_get_cpu_time_counter() is JIT-inlined as:

ISB // serialize instruction stream
MRS Xn, CNTVCT_EL0 // read architected timer counter

The ISB before the MRS is required for ordering, matching the kernel's arch_timer_read_cntvct_el0() implementation.

Careful here: this will _not_ order counter accesses against normal memory barriers (e.g. smp_mb(), acquire/release). If you need that, then you need something like arch_counter_enforce_ordering(), which we do have in __arch_counter_get_cntvct().

Furthermore, using the virtual counter may expose you to situations where a guest value for CNTVOFF is installed (e.g. during the VCPU run loop). Given all the contexts in which BPF can run, this worries me a little as you might end up seeing a non-monotonic view of time between BPF programs.

Quoted 7 lines

On newer CPUs it will be JITed to:

MRS Xn, CNTVCTSS_EL0 // self-synchronized (ISB not needed)

bpf_cpu_time_counter_to_ns() is JIT-inlined using mult/shift constants computed at JIT time from the architected timer frequency (CNTFRQ_EL0):

Quoted 6 lines
  MOV Xtmp, #mult           // load conversion multiplier
  MUL Xn, Xarg, Xtmp        // delta_ticks * mult
  LSR Xn, Xn, #shift        // >> shift = nanoseconds

On systems with a 1GHz counter (e.g., Neoverse-V2), mult=1 and shift=0, so the conversion collapses to a single MOV (identity).

Do you have any performance numbers to show that this is worthwhile compared to calling a helper wrapping __arch_counter_get_cntvct().

Quoted 7 lines
---
 arch/arm64/include/asm/insn.h                 |  2 +
 arch/arm64/net/bpf_jit.h                      |  4 ++
 arch/arm64/net/bpf_jit_comp.c                 | 54 +++++++++++++++++++
 .../selftests/bpf/progs/verifier_cpu_cycles.c | 50 ++++++++++++++++-
 4 files changed, 109 insertions(+), 1 deletion(-)

Aren't you conveniently ignoring CPU errata here? I suspect this needs to be predicated on the absence of those, in a similar way to how the vDSO deals with the 'vsdo_clockmode'.

Will

#18 lore raw

Re: [PATCH bpf-next v13 6/6] bpf, arm64: Add JIT support for cpu time counter kfuncs

Quoted 20 lines · Will Deacon

[+ Marc]

Quoted 10 lines · Puranjay Mohan

Add ARM64 JIT inlining for bpf_get_cpu_time_counter() and bpf_cpu_time_counter_to_ns() kfuncs.

bpf_get_cpu_time_counter() is JIT-inlined as:

ISB // serialize instruction stream
MRS Xn, CNTVCT_EL0 // read architected timer counter

The ISB before the MRS is required for ordering, matching the kernel's arch_timer_read_cntvct_el0() implementation.

Careful here: this will _not_ order counter accesses against normal memory barriers (e.g. smp_mb(), acquire/release). If you need that, then you need something like arch_counter_enforce_ordering(), which we do have in __arch_counter_get_cntvct().

Quoted 5 lines

Furthermore, using the virtual counter may expose you to situations where a guest value for CNTVOFF is installed (e.g. during the VCPU run loop). Given all the contexts in which BPF can run, this worries me a little as you might end up seeing a non-monotonic view of time between BPF programs.

Yup, and this will catch you out in subtle ways depending on whether the host runs at EL2 or not.

The kernel (mostly) avoids this pitfall by picking the physical counter for sched_clock() when running at EL2, so that it isn't directly affected by CNTVOFF_EL2.

I suspect that you'll need something similar if you insist on a globally monotonic view of the counter.

Quoted 7 lines
Quoted 7 lines

On newer CPUs it will be JITed to:

MRS Xn, CNTVCTSS_EL0 // self-synchronized (ISB not needed)

bpf_cpu_time_counter_to_ns() is JIT-inlined using mult/shift constants computed at JIT time from the architected timer frequency (CNTFRQ_EL0):

And what happens when CNTFRQ_EL0 is not populated, as it is the case on a lot of stupidly integrated HW? The kernel has a fallback, but you don't seem to make use of it here.

Quoted 19 lines
Quoted 7 lines
  MOV Xtmp, #mult           // load conversion multiplier
  MUL Xn, Xarg, Xtmp        // delta_ticks * mult
  LSR Xn, Xn, #shift        // >> shift = nanoseconds

On systems with a 1GHz counter (e.g., Neoverse-V2), mult=1 and shift=0, so the conversion collapses to a single MOV (identity).

Do you have any performance numbers to show that this is worthwhile compared to calling a helper wrapping __arch_counter_get_cntvct().

Quoted 7 lines
---
 arch/arm64/include/asm/insn.h                 |  2 +
 arch/arm64/net/bpf_jit.h                      |  4 ++
 arch/arm64/net/bpf_jit_comp.c                 | 54 +++++++++++++++++++
 .../selftests/bpf/progs/verifier_cpu_cycles.c | 50 ++++++++++++++++-
 4 files changed, 109 insertions(+), 1 deletion(-)
Quoted 3 lines

Aren't you conveniently ignoring CPU errata here? I suspect this needs to be predicated on the absence of those, in a similar way to how the vDSO deals with the 'vsdo_clockmode'.

And I'm afraid this still is the common case...

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.