tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
Subsystems (4)
Steven Rostedt
maintainerPATCH v5 1/1 tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: "Paul E. McKenney" <[email protected]>
The current use of guard(preempt_notrace)() within __DECLARE_TRACE() to protect invocation of __DO_TRACE_CALL() means that BPF programs attached to tracepoints are non-preemptible. This is unhelpful in real-time systems, whose users apparently wish to use BPF while also achieving low latencies. (Who knew?)
One option would be to use preemptible RCU, but this introduces many opportunities for infinite recursion, which many consider to be counterproductive, especially given the relatively small stacks provided by the Linux kernel. These opportunities could be shut down by sufficiently energetic duplication of code, but this sort of thing is considered impolite in some circles.
Therefore, use the shiny new SRCU-fast API, which provides somewhat faster readers than those of preemptible RCU, at least on Paul E. McKenney's laptop, where task_struct access is more expensive than access to per-CPU variables. And SRCU-fast provides way faster readers than does SRCU, courtesy of being able to avoid the read-side use of smp_mb(). Also, it is quite straightforward to create srcu_read_{,un}lock_fast_notrace() functions.
While in the area, SRCU now supports early boot call_srcu(). Therefore, remove the checks that used to avoid such use from rcu_free_old_probes() before this commit was applied:
e53244e2c893 ("tracepoint: Remove SRCU protection")
The current commit can be thought of as an approximate revert of that commit, with some compensating additions of preemption disabling. This preemption disabling uses guard(preempt_notrace)().
However, Yonghong Song points out that BPF assumes that non-sleepable BPF programs will remain on the same CPU, which means that migration must be disabled whenever preemption remains enabled. In addition, non-RT kernels have performance expectations that would be violated by allowing the BPF programs to be preempted.
Therefore, continue to disable preemption in non-RT kernels, and protect the BPF program with both SRCU and migration disabling for RT kernels, and even then only if preemption is not already disabled.
---
Changes since v4: https://lore.kernel.org/all/[email protected]/
- Added kerneldoc to trace_event_buffer_reserve{_syscall}
- Restructured the tracepoint.c code to not have #ifdef within functions
include/linux/trace_events.h | 24 ++++++++++++++ include/linux/tracepoint.h | 25 ++++++++++++-- include/trace/perf.h | 4 +-- include/trace/trace_events.h | 21 ++++++++++-- kernel/trace/trace_events.c | 61 +++++++++++++++++++++++++++++------ kernel/trace/trace_syscalls.c | 4 +-- kernel/tracepoint.c | 44 +++++++++++++++++++++++++ 7 files changed, 164 insertions(+), 19 deletions(-)
include/linux/trace_events.h +24 -0
@@ -222,6 +222,26 @@ static inline unsigned int tracing_gen_ctx_dec(void)return trace_ctx; } +/* +* When PREEMPT_RT is enabled, trace events are called with disabled +* migration. The trace events need to know if the tracepoint disabled +* migration or not so that what is recorded to the ring buffer shows +* the state of when the trace event triggered, and not the state caused +* by the trace event. +*/ +#ifdef CONFIG_PREEMPT_RT +static inline unsigned int tracing_gen_ctx_dec_cond (void ) +{ +unsigned int trace_ctx; + + trace_ctx =tracing_gen_ctx_dec (); +/* The migration counter starts at bit 4 */ +return trace_ctx - (1 <<4 ); +} +#else +# define tracing_gen_ctx_dec_cond () tracing_gen_ctx_dec() +#endif +struct trace_event_file ; struct ring_buffer_event *
@@ -313,6 +333,10 @@ void *trace_event_buffer_reserve(struct trace_event_buffer *fbuffer,struct trace_event_file *trace_file, unsigned long len); +void *trace_event_buffer_reserve_syscall (struct trace_event_buffer *fbuffer, +struct trace_event_file *trace_file, +unsigned long len); +void trace_event_buffer_commit (struct trace_event_buffer *fbuffer);enum {
include/linux/tracepoint.h +22 -3
@@ -100,6 +100,25 @@ void for_each_tracepoint_in_module(struct module *mod, }#endif /* CONFIG_MODULES */ +/* +* BPF programs can attach to the tracepoint callbacks. But if the +* callbacks are called with preemption disabled, the BPF programs +* can cause quite a bit of latency. When PREEMPT_RT is enabled, +* instead of disabling preemption, use srcu_fast_notrace() for +* synchronization. As BPF programs that are attached to tracepoints +* expect to stay on the same CPU, also disable migration. +*/ +#ifdef CONFIG_PREEMPT_RT +extern struct srcu_struct tracepoint_srcu; +# define tracepoint_sync () synchronize_srcu(&tracepoint_srcu); +# define tracepoint_guard () \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ + guard(migrate)() +#else +# define tracepoint_sync () synchronize_rcu(); +# define tracepoint_guard () guard(preempt_notrace)() +#endif +/* * tracepoint_synchronize_unregister must be called between the last tracepoint * probe unregistration and the end of module exit to make sure there is no
@@ -115,7 +134,7 @@ void for_each_tracepoint_in_module(struct module *mod,static inline void tracepoint_synchronize_unregister (void ) {synchronize_rcu_tasks_trace (); -synchronize_rcu (); +tracepoint_sync (); }static inline bool tracepoint_is_faultable (struct tracepoint *tp) {
@@ -275,13 +294,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) return static_branch_unlikely(&__tracepoint_##name.key);\ } -#define __DECLARE_TRACE (name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE (name, proto, args, cond, data_proto) \ __DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_proto)) \ static inline void __do_trace_##name(proto) \ { \ TRACEPOINT_CHECK(name) \ if (cond) { \ - guard(preempt_notrace)(); \ + tracepoint_guard(); \ __DO_TRACE_CALL(name, TP_ARGS(args)); \ } \ } \
include/trace/perf.h +2 -2
@@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ +guard (preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ }
@@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \ struct task_struct *__task __attribute__((unused)); \ \might_fault (); \ -preempt_disable_notrace (); \ +guard (preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ - preempt_enable_notrace(); \ }/*
include/trace/trace_events.h +19 -2
@@ -429,6 +429,22 @@ do_trace_event_raw_event_##call(void *__data, proto) \trace_event_buffer_commit (&fbuffer); \ } +/* +* When PREEMPT_RT is enabled, the tracepoint does not disable preemption +* but instead disables migration. The callbacks for the trace events +* need to have a consistent state so that it can reflect the proper +* preempt_disabled counter. +*/ +#ifdef CONFIG_PREEMPT_RT +/* disable preemption for RT so that the counters still match */ +# define trace_event_guard () guard(preempt_notrace)() +/* Have syscalls up the migrate disable counter to emulate non-syscalls */ +# define trace_syscall_event_guard () guard(migrate)() +#else +# define trace_event_guard () +# define trace_syscall_event_guard () +#endif +#undef DECLARE_EVENT_CLASS#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ __DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), PARAMS(tstruct), \
@@ -436,6 +452,7 @@ __DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), PARAMS(tstruct), \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ + trace_event_guard(); \ do_trace_event_raw_event_##call(__data, args); \ }
@@ -447,9 +464,9 @@ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ might_fault(); \ - preempt_disable_notrace(); \ + trace_syscall_event_guard(); \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ - preempt_enable_notrace(); \ }/*
kernel/trace/trace_events.c +51 -10
81 lines not on this page · raw message
kernel/trace/trace_syscalls.c +2 -2
@@ -909,7 +909,7 @@ static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id) size +=sizeof (*entry) +sizeof (unsigned long ) * sys_data->nb_args; - entry =trace_event_buffer_reserve (&fbuffer, trace_file, size); + entry =trace_event_buffer_reserve_syscall (&fbuffer, trace_file, size);if (!entry)return ;
@@ -955,7 +955,7 @@ static void ftrace_syscall_exit(void *data, struct pt_regs *regs, long ret)if (!sys_data)return ; - entry =trace_event_buffer_reserve (&fbuffer, trace_file,sizeof (*entry)); + entry =trace_event_buffer_reserve_syscall (&fbuffer, trace_file,sizeof (*entry));if (!entry)return ;
kernel/tracepoint.c +44 -0
81 lines not on this page · raw message
Signature · 1 line
2.51.0
Mathieu Desnoyers
reviewerQuoted 1 line · Steven Rostedt
From: "Paul E. McKenney" <[email protected]>
[...]
I disagree with many elements of the proposed approach.
On one end we have BPF wanting to hook on arbitrary tracepoints without adding significant latency to PREEMPT RT kernels.
One the other hand, we have high-speed tracers which execute very short critical sections to serialize trace data into ring buffers.
All of those users register to the tracepoint API.
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
So I'm wondering why all tracepoint users need to pay the migrate disable runtime overhead on preempt RT kernels for the sake of BPF ?
Using SRCU-fast to protect tracepoint callback iteration makes sense for preempt-rt, but I'd recommend moving the migrate disable guard within the bpf callback code rather than slowing down other tracers which execute within a short amount of time. Other tracers can then choose to disable preemption rather than migration if that's a better fit for their needs.
Quoted 13 lines
include/linux/trace_events.h +5
-0@@ -222,6 +222,26 @@ static inline unsigned int tracing_gen_ctx_dec(void)return trace_ctx; } +/* +* When PREEMPT_RT is enabled, trace events are called with disabled +* migration. The trace events need to know if the tracepoint disabled +* migration or not so that what is recorded to the ring buffer shows +* the state of when the trace event triggered, and not the state caused
include/linux/trace_events.h static inline unsigned int tracing_gen_ctx_dec(void) · line 239
Quoted 10 lines
include/linux/trace_events.h +10
-0+* by the trace event. +*/ +#ifdef CONFIG_PREEMPT_RT +static inline unsigned int tracing_gen_ctx_dec_cond (void ) +{ +unsigned int trace_ctx; + + trace_ctx =tracing_gen_ctx_dec (); +/* The migration counter starts at bit 4 */ +return trace_ctx - (1 <<4 );
We should turn this hardcoded "4" value into an enum label or a define. That define should be exposed by tracepoint.h. We should not hardcode expectations about the implementation of distinct APIs across the tracing subsystem.
[...]
Quoted 14 lines
include/linux/tracepoint.h +8
-0@@ -100,6 +100,25 @@ void for_each_tracepoint_in_module(struct module *mod, }#endif /* CONFIG_MODULES */ +/* +* BPF programs can attach to the tracepoint callbacks. But if the +* callbacks are called with preemption disabled, the BPF programs +* can cause quite a bit of latency. When PREEMPT_RT is enabled, +* instead of disabling preemption, use srcu_fast_notrace() for +* synchronization. As BPF programs that are attached to tracepoints +* expect to stay on the same CPU, also disable migration. +*/
include/linux/tracepoint.h void for_each_tracepoint_in_module(struct module *mod, · line 120
Quoted 10 lines
include/linux/tracepoint.h +10
-0+#ifdef CONFIG_PREEMPT_RT +extern struct srcu_struct tracepoint_srcu; +# define tracepoint_sync () synchronize_srcu(&tracepoint_srcu); +# define tracepoint_guard () \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ + guard(migrate)() +#else +# define tracepoint_sync () synchronize_rcu(); +# define tracepoint_guard () guard(preempt_notrace)() +#endif
Doing migrate disable on PREEMPT RT for BPF vs preempt disable in other tracers should come in a separate preparation patch. It belongs to the tracers, not to tracepoints.
[...]
\
Quoted 13 lines
include/trace/perf.h +1
-0@@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ +guard (preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ }
include/trace/perf.h perf_trace_##call(void *__data, proto) \ · line 90
Quoted 7 lines
include/trace/perf.h +1
-2@@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \ struct task_struct *__task __attribute__((unused)); \ \might_fault (); \ -preempt_disable_notrace (); \ +guard (preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ - preempt_enable_notrace();
Move this to a perf-specific preparation patch. \
Quoted 8 lines
include/trace/perf.h +0
-0}/* include/trace/trace_events.h +0
-0
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \ · line 436
Quoted 8 lines
include/trace/trace_events.h +5
-0@@ -429,6 +429,22 @@ do_trace_event_raw_event_##call(void *__data, proto) \trace_event_buffer_commit (&fbuffer); \ } +/* +* When PREEMPT_RT is enabled, the tracepoint does not disable preemption +* but instead disables migration. The callbacks for the trace events +* need to have a consistent state so that it can reflect the proper +* preempt_disabled counter.
Having those defines within trace_events.h is poking holes within any hope of abstraction we can have from the tracepoint.h API. This adds strong coupling between tracepoint and trace_event.h.
Rather than hardcoding preempt counter expectations across tracepoint and trace-events, we should expose a #define in tracepoint.h which will make the preempt counter nesting level available to other parts of the kernel such as trace_events.h. This way we keep everything in one place and we don't add cross-references about subtle preempt counter nesting level details.
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \ · line 446
Quoted 10 lines
include/trace/trace_events.h +10
-0+*/ +#ifdef CONFIG_PREEMPT_RT +/* disable preemption for RT so that the counters still match */ +# define trace_event_guard () guard(preempt_notrace)() +/* Have syscalls up the migrate disable counter to emulate non-syscalls */ +# define trace_syscall_event_guard () guard(migrate)() +#else +# define trace_event_guard () +# define trace_syscall_event_guard () +#endif
This should be moved to separate tracer-specific prep patches.
[...]
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \ · line 447
Quoted 1 line
include/trace/trace_events.h +1
-0+* The @trace_file is the desrciptor with information about the status
descriptor
[...]
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \
Quoted 4 lines
include/trace/trace_events.h +4
-0+* +* Returns a pointer to the data on the ring buffer or NULL if the +* event was not reserved (event was filtered, too big, or the buffer +* simply was disabled for write).
odd spaces here.
[...]
Quoted 21 lines
include/trace/trace_events.h +12
-0+#ifdef CONFIG_PREEMPT_RT +static void srcu_free_old_probes (struct rcu_head *head) +{ +kfree (container_of(head ,struct tp_probes ,rcu )); +} + +static void rcu_free_old_probes (struct rcu_head *head) +{ +call_srcu (&tracepoint_srcu, head, srcu_free_old_probes); +} +#else static void rcu_free_old_probes (struct rcu_head *head) {kfree (container_of(head ,struct tp_probes ,rcu )); } +#endif static inline void release_probes (struct tracepoint *tp,struct tracepoint_func *old) {
include/trace/trace_events.h static inline void release_probes(struct tracepoint *tp, struct tracepoint_func · line 154
Quoted 6 lines
include/trace/trace_events.h +3
-0@@ -112,6 +149,13 @@ static inline void release_probes(struct tracepoint *tp, struct tracepoint_funcstruct tp_probes *tp_probes =container_of (old, struct tp_probes, probes[0]); +/* +* Tracepoint probes are protected by either RCU or +* Tasks Trace RCU and also by SRCU. By calling the SRCU
I'm confused.
My understanding is that in !RT we have:
- RCU (!tracepoint_is_faultable(tp))
- RCU tasks trace (tracepoint_is_faultable(tp))
And for RT:
- SRCU-fast (!tracepoint_is_faultable(tp))
- RCU tasks trace (tracepoint_is_faultable(tp))
So I don't understand this comment, and also I don't understand why we need to chain the callbacks rather than just call the appropriate call_rcu based on the tracepoint "is_faultable" state.
What am I missing ?
include/trace/trace_events.h static inline void release_probes(struct tracepoint *tp, struct tracepoint_func · line 161
Quoted 7 lines
include/trace/trace_events.h +4
-0+* callback in the [Tasks Trace] RCU callback we cover +* both cases. So let us chain the SRCU and [Tasks Trace] +* RCU callbacks to wait for both grace periods. +*/ if (tracepoint_is_faultable (tp))call_rcu_tasks_trace (&tp_probes->rcu, rcu_free_old_probes);else
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Steven Rostedt
maintainerQuoted 9 lines · On Fri, 9 Jan 2026 09:40:17 -0500 Mathieu Desnoyers…
Quoted 1 line · Steven Rostedt
From: "Paul E. McKenney" <[email protected]>
[...]
I disagree with many elements of the proposed approach.
On one end we have BPF wanting to hook on arbitrary tracepoints without adding significant latency to PREEMPT RT kernels.
Quoted 7 lines
One the other hand, we have high-speed tracers which execute very short critical sections to serialize trace data into ring buffers.
All of those users register to the tracepoint API.
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
To be fair, every spin_lock() converted into a mutex in PREEMPT_RT now calls migrate_disable() instead of preempt_disable(). I'm just saying the overhead of migrate_disable() in PREEMPT_RT is not limited to tracepoints.
Quoted 3 lines
So I'm wondering why all tracepoint users need to pay the migrate disable runtime overhead on preempt RT kernels for the sake of BPF ?
We could argue that it is to keep the same paradigm as non RT. Where the code expects to stay on the same CPU. This is why we needed to add it to spin_lock() code. Only a few places in the kernel expect spin_lock() to pin the current task on the CPU, but because of those few cases, we needed to make all callers of spin_lock() call migrate disable :-/
Quoted 7 lines
Using SRCU-fast to protect tracepoint callback iteration makes sense for preempt-rt, but I'd recommend moving the migrate disable guard within the bpf callback code rather than slowing down other tracers which execute within a short amount of time. Other tracers can then choose to disable preemption rather than migration if that's a better fit for their needs.
This is a discussion with the BPF folks.
Quoted 21 lines
Quoted 21 lines
include/linux/trace_events.h +12
-0@@ -222,6 +222,26 @@ static inline unsigned int tracing_gen_ctx_dec(void)return trace_ctx; } +/* +* When PREEMPT_RT is enabled, trace events are called with disabled +* migration. The trace events need to know if the tracepoint disabled +* migration or not so that what is recorded to the ring buffer shows +* the state of when the trace event triggered, and not the state caused +* by the trace event. +*/ +#ifdef CONFIG_PREEMPT_RT +static inline unsigned int tracing_gen_ctx_dec_cond (void ) +{ +unsigned int trace_ctx; +
Quoted 8 lines
include/linux/trace_events.h static inline unsigned int tracing_gen_ctx_dec(void) · line 239
Quoted 3 lines
include/linux/trace_events.h +3
-0+ trace_ctx =tracing_gen_ctx_dec (); +/* The migration counter starts at bit 4 */ +return trace_ctx - (1 <<4 );We should turn this hardcoded "4" value into an enum label or a define. That define should be exposed by
tracepoint.h. We should not hardcode expectations about the implementation of distinct APIs across the tracing subsystem.
This is exposed to user space already, so that 4 will never change. And this is specifically for "trace events" which are what are attached to tracepoints. No other tracepoint caller needs to know about this "4". This value goes into the common_preempt_count of the event. libtraceevent already parses this.
I have no problem making this an enum (or define) and using it here and where it is set in trace.c:tracing_gen_ctx_irq_test(). But it belongs in trace_event.h not tracepoint.h.
I can call it TRACE_MIGRATION_SHIFT
#define TRACE_MIGRATION_SHIFT 4
Quoted 23 lines
[...]
Quoted 20 lines
include/linux/tracepoint.h +14
-0@@ -100,6 +100,25 @@ void for_each_tracepoint_in_module(struct module *mod, }#endif /* CONFIG_MODULES */ +/* +* BPF programs can attach to the tracepoint callbacks. But if the +* callbacks are called with preemption disabled, the BPF programs +* can cause quite a bit of latency. When PREEMPT_RT is enabled, +* instead of disabling preemption, use srcu_fast_notrace() for +* synchronization. As BPF programs that are attached to tracepoints +* expect to stay on the same CPU, also disable migration. +*/ +#ifdef CONFIG_PREEMPT_RT +extern struct srcu_struct tracepoint_srcu; +# define tracepoint_sync () synchronize_srcu(&tracepoint_srcu); +# define tracepoint_guard () \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ + guard(migrate)()
Quoted 8 lines
include/linux/tracepoint.h void for_each_tracepoint_in_module(struct module *mod, · line 120
Quoted 4 lines
include/linux/tracepoint.h +4
-0+#else +# define tracepoint_sync () synchronize_rcu(); +# define tracepoint_guard () guard(preempt_notrace)() +#endif Doing migrate disable on PREEMPT RT for BPF vs preempt disable in other tracers should come in a separate preparation patch. It belongs to the tracers, not to tracepoints.
That's fair.
Quoted 15 lines
[...]
\Quoted 12 lines
include/trace/perf.h +1
-0@@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ +guard (preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ }
Quoted 10 lines
include/trace/perf.h perf_trace_##call(void *__data, proto) \ · line 90
Quoted 8 lines
include/trace/perf.h +1
-2@@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \ struct task_struct *__task __attribute__((unused)); \ \might_fault (); \ -preempt_disable_notrace (); \ +guard (preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ - preempt_enable_notrace();Move this to a perf-specific preparation patch. \
Sure.
Quoted 11 lines
Quoted 11 lines
include/trace/perf.h +0
-0}/* include/trace/trace_events.h +0
-0@@ -429,6 +429,22 @@ do_trace_event_raw_event_##call(void *__data, proto) \trace_event_buffer_commit (&fbuffer); \ }
Quoted 9 lines
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \ · line 436
Quoted 5 lines
include/trace/trace_events.h +5
-0+/* +* When PREEMPT_RT is enabled, the tracepoint does not disable preemption +* but instead disables migration. The callbacks for the trace events +* need to have a consistent state so that it can reflect the proper +* preempt_disabled counter. Having those defines within
trace_events.his poking holes within any hope of abstraction we can have from thetracepoint.hAPI. This adds strong coupling between tracepoint andtrace_event.h.
OK, I see you are worried about the coupling between the behavior of tracepoint.h and trace_event.h.
Quoted 7 lines
Rather than hardcoding preempt counter expectations across tracepoint and trace-events, we should expose a #define in
tracepoint.hwhich will make the preempt counter nesting level available to other parts of the kernel such astrace_events.h. This way we keep everything in one place and we don't add cross-references about subtle preempt counter nesting level details.
OK, so how do we pass this information from tracepoint.h to the users? I hate to add another field to task_struct for this.
Quoted 13 lines
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \ · line 446
Quoted 11 lines
include/trace/trace_events.h +10
-0+*/ +#ifdef CONFIG_PREEMPT_RT +/* disable preemption for RT so that the counters still match */ +# define trace_event_guard () guard(preempt_notrace)() +/* Have syscalls up the migrate disable counter to emulate non-syscalls */ +# define trace_syscall_event_guard () guard(migrate)() +#else +# define trace_event_guard () +# define trace_syscall_event_guard () +#endif This should be moved to separate tracer-specific prep patches.
Quoted 5 lines
[...]
Quoted 1 line
+ * The @trace_file is the desrciptor with information about the status
descriptor
Oops.
Quoted 9 lines
[...]
Quoted 4 lines
+ *
+ * Returns a pointer to the data on the ring buffer or NULL if the
+ * event was not reserved (event was filtered, too big, or the buffer
+ * simply was disabled for write).odd spaces here.
You mean the indentation? I could add it more and also a colon:
* Returns: A pointer to the data on the ring buffer or NULL if the
* event was not reserved (event was filtered, too big, or the
* buffer simply was disabled for write).
Would that work better?
Quoted 40 lines
[...]
Quoted 21 lines
include/trace/trace_events.h +12
-0+#ifdef CONFIG_PREEMPT_RT +static void srcu_free_old_probes (struct rcu_head *head) +{ +kfree (container_of(head ,struct tp_probes ,rcu )); +} + +static void rcu_free_old_probes (struct rcu_head *head) +{ +call_srcu (&tracepoint_srcu, head, srcu_free_old_probes); +} +#else static void rcu_free_old_probes (struct rcu_head *head) {kfree (container_of(head ,struct tp_probes ,rcu )); } +#endif static inline void release_probes (struct tracepoint *tp,struct tracepoint_func *old) {include/trace/trace_events.h static inline void release_probes(struct tracepoint *tp, struct tracepoint_func · line 154
Quoted 6 lines
include/trace/trace_events.h +3
-0@@ -112,6 +149,13 @@ static inline void release_probes(struct tracepoint *tp, struct tracepoint_funcstruct tp_probes *tp_probes =container_of (old, struct tp_probes, probes[0]); +/* +* Tracepoint probes are protected by either RCU or +* Tasks Trace RCU and also by SRCU. By calling the SRCU I'm confused.
My understanding is that in !RT we have:
- RCU (!tracepoint_is_faultable(tp))
- RCU tasks trace (tracepoint_is_faultable(tp))And for RT:
Quoted 8 lines
- SRCU-fast (!tracepoint_is_faultable(tp))
- RCU tasks trace (tracepoint_is_faultable(tp))So I don't understand this comment, and also I don't understand why we need to chain the callbacks rather than just call the appropriate
call_rcubased on the tracepoint "is_faultable" state.What am I missing ?
Ah, you are right. I think this was the result of trying different ways of synchronization. The non-faultable version should be wrapped to either call normal RCU synchronization or SRCU synchronization.
I can send a new version, or do we want to wait if the BPF folks have a better idea about the "migrate disable" issue?
Thanks for the review.
-- Steve
Quoted 13 lines
include/trace/trace_events.h static inline void release_probes(struct tracepoint *tp, struct tracepoint_func · line 161
Quoted 8 lines
include/trace/trace_events.h +4
-0+* callback in the [Tasks Trace] RCU callback we cover +* both cases. So let us chain the SRCU and [Tasks Trace] +* RCU callbacks to wait for both grace periods. +*/ if (tracepoint_is_faultable (tp))call_rcu_tasks_trace (&tp_probes->rcu, rcu_free_old_probes);else Thanks,
Mathieu
Mathieu Desnoyers
reviewerQuoted 15 lines · Steven Rostedt
Quoted 12 lines · On Fri, 9 Jan 2026 09:40:17 -0500 Mathieu Desnoyers…
Quoted 1 line · Steven Rostedt
From: "Paul E. McKenney" <[email protected]>
[...]
I disagree with many elements of the proposed approach.
On one end we have BPF wanting to hook on arbitrary tracepoints without adding significant latency to PREEMPT RT kernels.
One the other hand, we have high-speed tracers which execute very short critical sections to serialize trace data into ring buffers.
Quoted 8 lines
Quoted 4 lines
All of those users register to the tracepoint API.
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
To be fair, every
spin_lock()converted into a mutex inPREEMPT_RTnow callsmigrate_disable()instead ofpreempt_disable(). I'm just saying the overhead ofmigrate_disable()inPREEMPT_RTis not limited to tracepoints.
That's a fair point. That being said, the kernel code which relies on spin locks for fast-paths tends to use raw spinlocks (e.g. scheduler), which AFAIU does not need migrate disable because those still disable preemption even on preempt RT.
Quoted 10 lines
Quoted 4 lines
So I'm wondering why all tracepoint users need to pay the migrate disable runtime overhead on preempt RT kernels for the sake of BPF ?
We could argue that it is to keep the same paradigm as non RT. Where the code expects to stay on the same CPU. This is why we needed to add it to
spin_lock()code. Only a few places in the kernel expectspin_lock()to pin the current task on the CPU, but because of those few cases, we needed to make all callers ofspin_lock()call migrate disable :-/
For tracepoints we have a handful of tracer users, so I don't consider the scope of the audit to be at the same scale as spinlocks. Changing each tracer user to either disable preemption or migration if they need it on preempt-RT is fairly straightforward. And if tracers don't need to stay on a single CPU, that's even better.
I'm also inclined to consider moving !RT kernels to srcu-fast rather than RCU to make RT and !RT more alike. The performance of srcu-fast read-side is excellent, at least on my EPYC test machine.
This would open the door to relax the tracer callback requirements on preempt-off/migrate-off if their callback implementation can deal with it.
Quoted 10 lines
Quoted 8 lines
Using SRCU-fast to protect tracepoint callback iteration makes sense for preempt-rt, but I'd recommend moving the migrate disable guard within the bpf callback code rather than slowing down other tracers which execute within a short amount of time. Other tracers can then choose to disable preemption rather than migration if that's a better fit for their needs.
This is a discussion with the BPF folks.
FWIW, the approach I'm proposing would be similar to what I've done for faultable syscall tracepoints.
[...]
Quoted 16 lines
Quoted 9 lines
include/linux/trace_events.h static inline unsigned int tracing_gen_ctx_dec(void) · line 239
Quoted 4 lines
include/linux/trace_events.h +4
-0+ + trace_ctx =tracing_gen_ctx_dec (); +/* The migration counter starts at bit 4 */ +return trace_ctx - (1 <<4 );We should turn this hardcoded "4" value into an enum label or a define. That define should be exposed by
tracepoint.h. We should not hardcode expectations about the implementation of distinct APIs across the tracing subsystem.This is exposed to user space already, so that 4 will never change. And this is specifically for "trace events" which are what are attached to tracepoints. No other tracepoint caller needs to know about this "4". This value goes into the
common_preempt_countof the event. libtraceevent already parses this.
Quoted 7 lines
I have no problem making this an enum (or define) and using it here and where it is set in trace.c:tracing_gen_ctx_irq_test(). But it belongs in
trace_event.hnottracepoint.h.I can call it
TRACE_MIGRATION_SHIFT#define
TRACE_MIGRATION_SHIFT4
I'm OK with that.
[...]
Quoted 32 lines
Quoted 11 lines
Quoted 11 lines
}/*include/trace/trace_events.h +0
-0@@ -429,6 +429,22 @@ do_trace_event_raw_event_##call(void *__data, proto) \trace_event_buffer_commit (&fbuffer); \ }Quoted 9 lines
include/trace/trace_events.h do_trace_event_raw_event_##call(void *__data, proto) \ · line 436
Quoted 5 lines
include/trace/trace_events.h +5
-0+/* +* When PREEMPT_RT is enabled, the tracepoint does not disable preemption +* but instead disables migration. The callbacks for the trace events +* need to have a consistent state so that it can reflect the proper +* preempt_disabled counter. Having those defines within
trace_events.his poking holes within any hope of abstraction we can have from thetracepoint.hAPI. This adds strong coupling between tracepoint andtrace_event.h.OK, I see you are worried about the coupling between the behavior of
tracepoint.handtrace_event.h.Quoted 7 lines
Rather than hardcoding preempt counter expectations across tracepoint and trace-events, we should expose a #define in
tracepoint.hwhich will make the preempt counter nesting level available to other parts of the kernel such astrace_events.h. This way we keep everything in one place and we don't add cross-references about subtle preempt counter nesting level details.
Quoted 2 lines
OK, so how do we pass this information from
tracepoint.hto the users? I hate to add another field totask_structfor this.
This could be as simple as adding something akin to
/*
* Level of preempt disable nesting between tracepoint caller code and
* tracer callback.
*/
#ifdef CONFIG_PREEMPT_RT
# define TRACEPOINT_PREEMPT_DISABLE_NESTING 0 #else # define TRACEPOINT_PREEMPT_DISABLE_NESTING 1 #endif
to tracepoint.h and then use that from trace_event.h.
Quoted 11 lines
Quoted 10 lines
[...]
Quoted 4 lines
+ *
+ * Returns a pointer to the data on the ring buffer or NULL if the
+ * event was not reserved (event was filtered, too big, or the buffer
+ * simply was disabled for write).odd spaces here.
Quoted 7 lines
You mean the indentation? I could add it more and also a colon:
* Returns: A pointer to the data on the ring buffer or NULL if the
* event was not reserved (event was filtered, too big, or the
* buffer simply was disabled for write).Would that work better?
Yes.
[...]
Quoted 10 lines
Quoted 6 lines
So I don't understand this comment, and also I don't understand why we need to chain the callbacks rather than just call the appropriate
call_rcubased on the tracepoint "is_faultable" state.What am I missing ?
Ah, you are right. I think this was the result of trying different ways of synchronization. The non-faultable version should be wrapped to either call normal RCU synchronization or SRCU synchronization.
Yes. Or we switch even !RT kernels non-faultable tracepoints to srcu-fast, considering its low overhead. This could come as a future step though.
Quoted 2 lines
I can send a new version, or do we want to wait if the BPF folks have a better idea about the "migrate disable" issue?
I'd recommend going ahead and move the migrate disable into the BPF callback similarly to what did for the faultable syscall tracepoint, and let the bpf folks comment on the resulting patch. It's not that much code, and showing the change will ground the discussion into something more tangible.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Quoted 8 lines · Mathieu Desnoyers
Quoted 7 lines · Steven Rostedt
Quoted 6 lines
Using SRCU-fast to protect tracepoint callback iteration makes sense for preempt-rt, but I'd recommend moving the migrate disable guard within the bpf callback code rather than slowing down other tracers which execute within a short amount of time. Other tracers can then choose to disable preemption rather than migration if that's a better fit for their needs.
Quoted 4 lines
Quoted 1 line
This is a discussion with the BPF folks.
FWIW, the approach I'm proposing would be similar to what I've done for faultable syscall tracepoints.
I just started reading this thread but we could limit the migrate_disable() to only the BPF callback part so it does not effect the whole tracepoint if there not a BFP program attached to it.
Sebastian
Quoted 10 lines · Mathieu Desnoyers
Quoted 1 line · Steven Rostedt
From: "Paul E. McKenney" <[email protected]>
[...]
I disagree with many elements of the proposed approach.
On one end we have BPF wanting to hook on arbitrary tracepoints without adding significant latency to PREEMPT RT kernels.
Quoted 7 lines
One the other hand, we have high-speed tracers which execute very short critical sections to serialize trace data into ring buffers.
All of those users register to the tracepoint API.
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
Looks like your complaint comes from lack of engagement in kernel development.migrate_disable _was_ not cheap.
Try to benchmark it now.
It's inlined. It's a fraction of extra overhead on top of preempt_disable.
Steven Rostedt
maintainerOn Fri, 9 Jan 2026 11:10:16 -0800
Alexei Starovoitov <[email protected]> wrote:
\> >
Quoted 5 lines
Quoted 2 lines
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
Looks like your complaint comes from lack of engagement in kernel development.
No need to make comments like that. The Linux kernel is an ocean of code. It's very hard to keep up on everything that is happening. I knew of work being done on migrate_disable but I didn't know what the impacts of that work was. Mathieu is still very much involved and engaged in kernel development.
Quoted 3 lines
migrate_disable_was_ not cheap.
Try to benchmark it now.
It's inlined. It's a fraction of extra overhead on top ofpreempt_disable.
It would be good to have a benchmark of the two. What about fast_srcu? Is that fast enough to replace the preempt_disable()? If so, then could we just make this the same for both RT and !RT?
-- Steve
Mathieu Desnoyers
reviewerQuoted 7 lines · Steven Rostedt
On Fri, 9 Jan 2026 11:10:16 -0800
Alexei Starovoitov <[email protected]> wrote:\> >
Quoted 3 lines
Quoted 2 lines
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
Quoted 8 lines
Quoted 2 lines
Looks like your complaint comes from lack of engagement in kernel development.
No need to make comments like that. The Linux kernel is an ocean of code. It's very hard to keep up on everything that is happening. I knew of work being done on
migrate_disablebut I didn't know what the impacts of that work was. Mathieu is still very much involved and engaged in kernel development.
Thanks Steven. I guess Alexei missed my recent involvement in other areas of the kernel.
As Steven pointed out, the kernel is vast, so I cannot keep up with the progress on every single topic. That being said, I very recently (about 1 month ago) tried using migrate disable for the RSS tracking improvements (hierarchical percpu counters), and found that the overhead of migrate disable was large compared to preempt disable. The generated assembler is also orders of magnitude larger (on x86-64).
Creating small placeholder functions which just call preempt/migrate disable and enable for a preempt RT build:
0000000000002a20 <test_preempt_disable>:
2a20: f3 0f 1e fa endbr64
2a24: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 2a2b <test_preempt_disable+0xb>
2a2b: e9 00 00 00 00 jmp 2a30 <test_preempt_disable+0x10>0000000000002a40 <test_preempt_enable>:
2a40: f3 0f 1e fa endbr64
2a44: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 2a4b <test_preempt_enable+0xb>
2a4b: 74 05 je 2a52 <test_preempt_enable+0x12>
2a4d: e9 00 00 00 00 jmp 2a52 <test_preempt_enable+0x12>
2a52: e8 00 00 00 00 call 2a57 <test_preempt_enable+0x17>
2a57: e9 00 00 00 00 jmp 2a5c <test_preempt_enable+0x1c>0000000000002920 <test_migrate_disable>:
2920: f3 0f 1e fa endbr64
2924: 65 48 8b 15 00 00 00 mov %gs:0x0(%rip),%rdx # 292c <test_migrate_disable+0xc>
292b: 00
292c: 0f b7 82 38 07 00 00 movzwl 0x738(%rdx),%eax
2933: 66 85 c0 test %ax,%ax
2936: 74 0f je 2947 <test_migrate_disable+0x27>
2938: 83 c0 01 add $0x1,%eax
293b: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
2942: e9 00 00 00 00 jmp 2947 <test_migrate_disable+0x27>
2947: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 294e <test_migrate_disable+0x2e>
294e: 65 48 8b 05 00 00 00 mov %gs:0x0(%rip),%rax # 2956 <test_migrate_disable+0x36>
2955: 00
2956: 83 80 00 00 00 00 01 addl $0x1,0x0(%rax)
295d: b8 01 00 00 00 mov $0x1,%eax
2962: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
2969: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 2970 <test_migrate_disable+0x50>
2970: 74 05 je 2977 <test_migrate_disable+0x57>
2972: e9 00 00 00 00 jmp 2977 <test_migrate_disable+0x57>
2977: e8 00 00 00 00 call 297c <test_migrate_disable+0x5c>
297c: e9 00 00 00 00 jmp 2981 <test_migrate_disable+0x61>00000000000029a0 <test_migrate_enable>:
29a0: f3 0f 1e fa endbr64
29a4: 65 48 8b 15 00 00 00 mov %gs:0x0(%rip),%rdx # 29ac <test_migrate_enable+0xc>
29ab: 00
29ac: 0f b7 82 38 07 00 00 movzwl 0x738(%rdx),%eax
29b3: 66 85 c0 test %ax,%ax
29b6: 74 0f je 29c7 <test_migrate_enable+0x27>
29b8: 83 c0 01 add $0x1,%eax
29bb: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
29c2: e9 00 00 00 00 jmp 29c7 <test_migrate_enable+0x27>
29c7: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 29ce <test_migrate_enable+0x2e>
29ce: 65 48 8b 05 00 00 00 mov %gs:0x0(%rip),%rax # 29d6 <test_migrate_enable+0x36>
29d5: 00
29d6: 83 80 00 00 00 00 01 addl $0x1,0x0(%rax)
29dd: b8 01 00 00 00 mov $0x1,%eax
29e2: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
29e9: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 29f0 <test_migrate_enable+0x50>
29f0: 74 05 je 29f7 <test_migrate_enable+0x57>
29f2: e9 00 00 00 00 jmp 29f7 <test_migrate_enable+0x57>
29f7: e8 00 00 00 00 call 29fc <test_migrate_enable+0x5c>
29fc: e9 00 00 00 00 jmp 2a01 <test_migrate_enable+0x61>Quoted 8 lines
Quoted 4 lines
migrate_disable_was_ not cheap.
Try to benchmark it now.
It's inlined. It's a fraction of extra overhead on top ofpreempt_disable.It would be good to have a benchmark of the two. What about
fast_srcu? Is that fast enough to replace thepreempt_disable()? If so, then could we just make this the same for both RT and !RT?
I've modified kernel/rcu/refscale.c to compare those:
AMD EPYC 9654 96-Core Processor, kernel baseline: v6.18.1 CONFIG_PREEMPT=y
# CONFIG_PREEMPT_LAZY is not set
# CONFIG_PREEMPT_RT is not set
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
* calls to migrate disable/enable pair within noinline functions: 17.0 ns
CONFIG_RCU_REF_SCALE_TEST=m
* migrate disable/enable pair: 22.0 ns
When I attempted using migrate disable, I configured refscale as a module, which gave me the appalling 22 ns overhead. It looks like the implementation of migrate disable/enable now differs depending on whether it's used from the core kernel or from a module. That's rather unexpected.
It seems to be done on purpose though (INSTANTIATE_EXPORTED_MIGRATE_DISABLE) to work around the fact that it is not possible to export the runqueues variable.
That's the kind of compilation context dependent overhead variability I'd rather avoid in the implementation of the tracepoint instrumentation API.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Steven Rostedt
maintainerQuoted 9 lines · On Fri, 9 Jan 2026 15:21:19 -0500 Mathieu Desnoyers…
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
* calls to migrate disable/enable pair within noinline functions: 17.0 ns
CONFIG_RCU_REF_SCALE_TEST=m
* migrate disable/enable pair: 22.0 ns
OUCH! So migrate disable/enable has a much larger overhead when executed in a module than in the kernel? This means all spin_locks() in modules converted to mutexes in PREEMPT_RT are taking this hit!
It looks like it has to allow access to the rq->nr_pinned. There's a hack to expose this part of the rq struct for in-kernel by the following:
kernel/sched/rq-offsets.c: DEFINE(RQ_nr_pinned, offsetof(struct rq, nr_pinned));
Then for the in-kernel code we have:
#define this_rq_raw() arch_raw_cpu_ptr(&runqueues)
#else
#define this_rq_raw() PERCPU_PTR(&runqueues)
#endif
#define this_rq_pinned() (*(unsigned int *)((void *)this_rq_raw() + RQ_nr_pinned))
Looking at the scheduler code, the rq->nr_pinned is referenced by a static function with:
static inline bool rq_has_pinned_tasks(struct rq *rq)
{
return rq->nr_pinned;
}Which is only referenced in hotplug code and a balance_push() path in load balancing. Does this variable really need to be in the runqueue struct?
Why not just make it a per-cpu variable. Maybe call it cpu_nr_pinned_tasks, and export that for all to use?
It will not only fix the discrepancy between the overhead of migrate_disable/enable in modules vs in-kernel. But it also removes the hack to expose a portion of the runqueue.
-- Steve
On 2026/1/10 05:02 Steven Rostedt <[email protected]> write:
Quoted 41 lines
Quoted 9 lines · On Fri, 9 Jan 2026 15:21:19 -0500 Mathieu Desnoyers…
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
* calls to migrate disable/enable pair within noinline functions: 17.0 ns
CONFIG_RCU_REF_SCALE_TEST=m
* migrate disable/enable pair: 22.0 nsOUCH! So migrate disable/enable has a much larger overhead when executed in a module than in the kernel? This means all
spin_locks()in modules converted to mutexes inPREEMPT_RTare taking this hit!It looks like it has to allow access to the
rq->nr_pinned. There's a hack to expose this part of the rq struct for in-kernel by the following:
kernel/sched/rq-offsets.c:DEFINE(RQ_nr_pinned, offsetof(struct rq, nr_pinned));Then for the in-kernel code we have:
#define
this_rq_raw()arch_raw_cpu_ptr(&runqueues)
#else
#definethis_rq_raw()PERCPU_PTR(&runqueues)
#endif
#definethis_rq_pinned()(*(unsigned int *)((void *)this_rq_raw() +RQ_nr_pinned))Looking at the scheduler code, the
rq->nr_pinnedis referenced by a static function with:static inline bool
rq_has_pinned_tasks(struct rq *rq){ return rq->nr_pinned; }Which is only referenced in hotplug code and a
balance_push()path in load balancing. Does this variable really need to be in the runqueue struct?
Quoted 6 lines
Why not just make it a per-cpu variable. Maybe call it
cpu_nr_pinned_tasks, and export that for all to use?It will not only fix the discrepancy between the overhead of migrate_disable/enable in modules vs in-kernel. But it also removes the hack to expose a portion of the runqueue.
I think it's a good idea to factor out the "nr_pinned" from struct rq. The current approach that we inline the migrate_disable is a little obscure. The initial propose of inline migrate_disable is to optimize the performance of bpf trampoline, so the modules are not considered.
As you said, rq_has_pinned_tasks() is the only place that use the nr_pinned, except the migrate_disable/migrate_enable. After more analysis, I think maybe we can do it this way:
DEFINE_PER_CPU_SHARED_ALIGNED(int, cpu_nr_pinned_tasks);
And change rq_has_pinned_tasks() to:
static inline bool rq_has_pinned_tasks(struct rq *rq)
{
return *per_cpu_ptr(&cpu_nr_pinned_tasks, rq->cpu);
}The "rq" in rq_has_pinned_tasks() may come from other CPU, so we can't use "return this_cpu_read(cpu_nr_pinned_tasks)" directly.
Thanks!
Menglong Dong
Quoted 4 lines
-- Steve
Quoted 10 lines · Steven Rostedt
Quoted 7 lines · On Fri, 9 Jan 2026 15:21:19 -0500 Mathieu Desnoyers…
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
* calls to migrate disable/enable pair within noinline functions: 17.0 ns
Quoted 6 lines
Quoted 2 lines
CONFIG_RCU_REF_SCALE_TEST=m
* migrate disable/enable pair: 22.0 nsOUCH! So migrate disable/enable has a much larger overhead when executed in a module than in the kernel? This means all
spin_locks()in modules converted to mutexes inPREEMPT_RTare taking this hit!
Not so, the migrate_disable() for PREEMPT_RT is still in core code -- kernel/locking/spinlock_rt.c is very much not build as a module.
Steven Rostedt
maintainerQuoted 6 lines · On Mon, 12 Jan 2026 16:31:28 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 3 lines
OUCH! So migrate disable/enable has a much larger overhead when executed in a module than in the kernel? This means all
spin_locks()in modules converted to mutexes inPREEMPT_RTare taking this hit!Not so, the
migrate_disable()forPREEMPT_RTis still in core code --kernel/locking/spinlock_rt.cis very much not build as a module.
True. But still, wouldn't it be cleaner to have that variable separate from the run queue and make the code a bit simpler?
As now it doesn't look like it will even bother tracing, as it appears that only BPF would need it. So this would just be a clean up.
-- Steve
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Mon, 12 Jan 2026 16:31:28 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 3 lines
OUCH! So migrate disable/enable has a much larger overhead when executed in a module than in the kernel? This means all
spin_locks()in modules converted to mutexes inPREEMPT_RTare taking this hit!
Quoted 5 lines
Quoted 2 lines
Not so, the
migrate_disable()forPREEMPT_RTis still in core code --kernel/locking/spinlock_rt.cis very much not build as a module.True. But still, wouldn't it be cleaner to have that variable separate from the run queue and make the code a bit simpler?
I still don't like exporting variables either way around. Exporting functions is much saner.
Quoted 7 lines · Mathieu Desnoyers
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
.. and you're arguing that 3ns vs 1ns difference is so important for your out-of-tree tracer that in-tree tracers need to do some workarounds?! wtf
Steven Rostedt
maintainerQuoted 7 lines · On Fri, 9 Jan 2026 13:54:34 -0800 Alexei Starovoitov…
Quoted 5 lines · Mathieu Desnoyers
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
Quoted 6 lines
Quoted 2 lines
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns.. and you're arguing that 3ns vs 1ns difference is so important for your out-of-tree tracer that in-tree tracers need to do some workarounds?! wtf
This has nothing to do with out of tree tracers. The overhead of the 22ns is for any tracepoint in an in-tree module. That's because the rq->nr_pinned isn't exported for modules to use.
-- Steve
Quoted 14 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Jan 2026 13:54:34 -0800 Alexei Starovoitov…
Quoted 7 lines · Mathieu Desnoyers
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
Quoted 7 lines
Quoted 3 lines
.. and you're arguing that 3ns vs 1ns difference is so important for your out-of-tree tracer that in-tree tracers need to do some workarounds?! wtf
This has nothing to do with out of tree tracers. The overhead of the 22ns is for any tracepoint in an in-tree module. That's because the
rq->nr_pinnedisn't exported for modules to use.
None of the driver's tracepoints are in the critical path. You perfectly know that Mathieu argued about not slowing down lttng.
Mathieu Desnoyers
reviewerQuoted 19 lines · Alexei Starovoitov
Quoted 18 lines · Steven Rostedt
Quoted 13 lines · On Fri, 9 Jan 2026 13:54:34 -0800 Alexei Starovoitov…
Quoted 7 lines · Mathieu Desnoyers
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns.. and you're arguing that 3ns vs 1ns difference is so important for your out-of-tree tracer that in-tree tracers need to do some workarounds?! wtf
Quoted 6 lines
Quoted 3 lines
This has nothing to do with out of tree tracers. The overhead of the 22ns is for any tracepoint in an in-tree module. That's because the
rq->nr_pinnedisn't exported for modules to use.None of the driver's tracepoints are in the critical path. You perfectly know that Mathieu argued about not slowing down lttng.
My argument is about not slowing down high-throughput tracers on preempt-rt kernels. This affects ftrace as well as lttng, so both in-tree and OOT tracers just alike.
Are you so sure that no tracepoint instrumentation whatsoever can be compiled into a module which is a critical path for some workload ? That's a bold statement.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Steven Rostedt
maintainerQuoted 19 lines · On Fri, 9 Jan 2026 14:18:36 -0800 Alexei Starovoitov…
Quoted 18 lines · Steven Rostedt
Quoted 13 lines · On Fri, 9 Jan 2026 13:54:34 -0800 Alexei Starovoitov…
Quoted 7 lines · Mathieu Desnoyers
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns.. and you're arguing that 3ns vs 1ns difference is so important for your out-of-tree tracer that in-tree tracers need to do some workarounds?! wtf
Quoted 6 lines
Quoted 3 lines
This has nothing to do with out of tree tracers. The overhead of the 22ns is for any tracepoint in an in-tree module. That's because the
rq->nr_pinnedisn't exported for modules to use.None of the driver's tracepoints are in the critical path. You perfectly know that Mathieu argued about not slowing down lttng.
How is this about lttng? Sure he cares about that, but even tracepoints that lttng uses doesn't get affected any more than ftrace or bpf. Because lttng is one of the callbacks. The migrate disable happens in the in-tree portion of the code.
So you are saying that all the tracepoints for xfs are not in a fastpath?
-- Steve
Steven Rostedt
maintainerQuoted 6 lines · On Fri, 9 Jan 2026 17:33:26 -0500 Steven Rostedt <[email protected]> wrote:
How is this about lttng? Sure he cares about that, but even tracepoints that lttng uses doesn't get affected any more than ftrace or bpf. Because lttng is one of the callbacks. The migrate disable happens in the in-tree portion of the code.
So you are saying that all the tracepoints for xfs are not in a fastpath?
Regardless of tracing. I now have my RT hat on. The spin_locks that are converted to mutex use migrate disable. The fact that migrate_disable in modules are close to 10x slower than the same code in-kernel is troubling to say the least. It means that modules in RT take a hit every time they take a spin_lock().
The migrate disable being slow for modules is no longer just a tracing issue. It's a PREEMPT_RT issue.
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 6 lines · On Fri, 9 Jan 2026 17:33:26 -0500 Steven Rostedt <[email protected]> wrote:
How is this about lttng? Sure he cares about that, but even tracepoints that lttng uses doesn't get affected any more than ftrace or bpf. Because lttng is one of the callbacks. The migrate disable happens in the in-tree portion of the code.
So you are saying that all the tracepoints for xfs are not in a fastpath?
Quoted 8 lines
Regardless of tracing. I now have my RT hat on. The
spin_locksthat are converted to mutex use migrate disable. The fact thatmigrate_disablein modules are close to 10x slower than the same code in-kernel is troubling to say the least. It means that modules in RT take a hit every time they take aspin_lock().The migrate disable being slow for modules is no longer just a tracing issue. It's a
PREEMPT_RTissue.
migrate_enable/disable() wasn't inlined for a long time. It bothered us enough, since sleepable bpf is the main user of it besides RT, so we made an effort to inline it.
RT, at the same time, doesn't inline rt_spin_lock() itself so inlining migrate_disable() or not is not 10x at all. Benchmark spin_lock on RT in-tree and in-module and I bet there won't be a big difference.
Steven Rostedt
maintainerQuoted 3 lines · On Fri, 9 Jan 2026 16:35:10 -0800 Alexei Starovoitov…
migrate_enable/disable() wasn't inlined for a long time. It bothered us enough, since sleepable bpf is the main user of it besides RT, so we made an effort to inline it.
It did bother us too. it went through lots of iterations to become more efficient over the years (it was really bad in the beginning while still in the rt-patch), and hopefully that will continue.
Quoted 5 lines
RT, at the same time, doesn't inline
rt_spin_lock()itself so inliningmigrate_disable()or not is not 10x at all. Benchmarkspin_lockon RT in-tree and in-module and I bet there won't be a big difference.
I'll put that on my todo list. But still, having migrate_disable a function for modules and 100% inlined for in-kernel code just because it needs access to a field in the run queue that doesn't need to be in the run queue seems like it should be fixed.
As for tracepoints, BPF is the only one that needs migrate disable. It's not needed for ftrace or perf (although perf uses preempt disable). It should be moved into the BPF callback code as perf has its preempt disable in its callback code.
If BPF doesn't care about the extra overhead of migrate_disable() for modules, then why should XFS suffer from that too?
-- Steve
Quoted 18 lines · Steven Rostedt
Quoted 3 lines · On Fri, 9 Jan 2026 16:35:10 -0800 Alexei Starovoitov…
migrate_enable/disable() wasn't inlined for a long time. It bothered us enough, since sleepable bpf is the main user of it besides RT, so we made an effort to inline it.
It did bother us too. it went through lots of iterations to become more efficient over the years (it was really bad in the beginning while still in the rt-patch), and hopefully that will continue.
Quoted 5 lines
RT, at the same time, doesn't inline
rt_spin_lock()itself so inliningmigrate_disable()or not is not 10x at all. Benchmarkspin_lockon RT in-tree and in-module and I bet there won't be a big difference.
Quoted 4 lines
I'll put that on my todo list. But still, having
migrate_disablea function for modules and 100% inlined for in-kernel code just because it needs access to a field in the run queue that doesn't need to be in the run queue seems like it should be fixed.
There was plenty of discussion with Peter regarding different ways to inline migrate_disable. What was landed was the best option at that point, but feel free to restart the discussion.
Quoted 8 lines
As for tracepoints, BPF is the only one that needs migrate disable. It's not needed for ftrace or perf (although perf uses preempt disable). It should be moved into the BPF callback code as perf has its preempt disable in its callback code.
If BPF doesn't care about the extra overhead of
migrate_disable()for modules, then why should XFS suffer from that too?
The diff has nothing to do with bpf needs and/or bpf internals. It's really about being a good citizen of PREEMP_RT.
bpf side already does migrate_disable,rcu_read_lock, srcu_fast/task_trace when necessary.
Most of the time we don't rely on any external preempt state or rcu/srcu. Removing guard(preempt_notrace)(); from tracepoint invocation would be just fine for bpf. Simple remove will trigger bug
on cant_sleep(), but that's a trivial fix.
Steven Rostedt
maintainerQuoted 8 lines · On Sun, 11 Jan 2026 12:04:51 -0800 Alexei Starovoitov…
The diff has nothing to do with bpf needs and/or bpf internals. It's really about being a good citizen of
PREEMP_RT.
bpf side already doesmigrate_disable,rcu_read_lock, srcu_fast/task_trace when necessary.
Most of the time we don't rely on any external preempt state or rcu/srcu. Removing guard(preempt_notrace)(); from tracepoint invocation would be just fine for bpf. Simple remove will trigger bug
oncant_sleep(), but that's a trivial fix.
Oh, so you are OK replacing the preempt_disable in the tracepoint callbacks with fast SRCU?
Then I guess we can simply do that. Would it be fine to do that for both RT and non-RT? That will simplify the code quite a bit.
-- Steve
Quoted 13 lines · Steven Rostedt
Quoted 8 lines · On Sun, 11 Jan 2026 12:04:51 -0800 Alexei Starovoitov…
The diff has nothing to do with bpf needs and/or bpf internals. It's really about being a good citizen of
PREEMP_RT.
bpf side already doesmigrate_disable,rcu_read_lock, srcu_fast/task_trace when necessary.
Most of the time we don't rely on any external preempt state or rcu/srcu. Removing guard(preempt_notrace)(); from tracepoint invocation would be just fine for bpf. Simple remove will trigger bug
oncant_sleep(), but that's a trivial fix.
Quoted 2 lines
Oh, so you are OK replacing the
preempt_disablein the tracepoint callbacks with fast SRCU?
yes, but..
Quoted 2 lines
Then I guess we can simply do that. Would it be fine to do that for both RT and non-RT? That will simplify the code quite a bit.
Agree. perf needs preempt_disable in their callbacks (as this patch does) and bpf side needs to add migrate_disable in __bpf_trace_run for now. Though syscall tracepoints are sleepable we don't take advantage of that on the bpf side. Eventually we will, and then rcu_lock inside __bpf_trace_run will become srcu_fast_lock.
The way to think about generic infrastructure like tracepoints is to minimize their overhead no matter what out-of-tree and in-tree users' assumptions are today, so why do we need preempt_disable or srcu_fast there?
I think today it's there because all callbacks (perf, ftrace, bpf) expect preemption to be disabled, but can we just remove it from tp side? and move preempt_disable to callbacks that actually need it?
I'm looking at release_probes(). It's fine as-is, no?
Steven Rostedt
maintainerQuoted 14 lines · On Sun, 11 Jan 2026 15:38:38 -0800 Alexei Starovoitov…
Quoted 2 lines
Oh, so you are OK replacing the
preempt_disablein the tracepoint callbacks with fast SRCU?yes, but..
Quoted 2 lines
Then I guess we can simply do that. Would it be fine to do that for both RT and non-RT? That will simplify the code quite a bit.
Agree. perf needs
preempt_disablein their callbacks (as this patch does) and bpf side needs to addmigrate_disablein__bpf_trace_runfor now. Though syscall tracepoints are sleepable we don't take advantage of that on the bpf side. Eventually we will, and thenrcu_lockinside__bpf_trace_runwill becomesrcu_fast_lock.
Quoted 4 lines
The way to think about generic infrastructure like tracepoints is to minimize their overhead no matter what out-of-tree and in-tree users' assumptions are today, so why do we need
preempt_disableorsrcu_fastthere?
Either preempt disable or srcu_fast is still needed.
Quoted 3 lines
I think today it's there because all callbacks (perf, ftrace, bpf) expect preemption to be disabled, but can we just remove it from tp side? and move
preempt_disableto callbacks that actually need it?
Yes if we are talking about switching from preempt_disable to src_fast. No if you mean to remove both as it still needs RCU protection. It's used for synchronizing changes in the tracepoint infrastructure itself. That __DO_TRACE_CALL() macro is the guts of the tracepoint callback code. It needs to handle races with additions and removals of callbacks there, as the callbacks also get data passed to them. If it gets out of sync, a callback could be called with another callback's data.
That's why it has:
it_func_ptr = \
rcu_dereference_raw((&__tracepoint_##name)->funcs);Quoted 2 lines
I'm looking at
release_probes(). It's fine as-is, no?
That's just freeing, and as you see, there's RCU synchronization required.
Updates to tracepoints require RCU protection. It started out with preempt_disable() for all tracepoints (which was synchronized with synchronized_sched() which later became just synchronize_rcu()).
The issue that came about was that both ftrace and perf had an assumption that its tracepoint callbacks always have preempt disabled when being called. To move to srcu_fast() that is no longer the case. And we need to do that for PREEMPT_RT if BPF is running very long callbacks to the tracepoints.
Ftrace has been fixed to not require it, but still needs to take into account if tracepoints disable preemption or not so that it can display the preempt_count() of when the tracepoint was called correctly.
Perf is trivial to fix as it can simply add a preempt_disable() in its handler.
But we were originally told that BPF had an issue because it had the assumption that tracepoint callbacks wouldn't migrate. That's where this patch came about.
Now if you are saying that BPF will handle migrate_disable() on its own and not require the tracepoint infrastructure to do it for it, then this is perfect. And I can then simplify this code, and just use srcu_fast for both RT and !RT.
-- Steve
Quoted 43 lines · Steven Rostedt
Quoted 14 lines · On Sun, 11 Jan 2026 15:38:38 -0800 Alexei Starovoitov…
Quoted 2 lines
Oh, so you are OK replacing the
preempt_disablein the tracepoint callbacks with fast SRCU?yes, but..
Quoted 2 lines
Then I guess we can simply do that. Would it be fine to do that for both RT and non-RT? That will simplify the code quite a bit.
Agree. perf needs
preempt_disablein their callbacks (as this patch does) and bpf side needs to addmigrate_disablein__bpf_trace_runfor now. Though syscall tracepoints are sleepable we don't take advantage of that on the bpf side. Eventually we will, and thenrcu_lockinside__bpf_trace_runwill becomesrcu_fast_lock.Quoted 4 lines
The way to think about generic infrastructure like tracepoints is to minimize their overhead no matter what out-of-tree and in-tree users' assumptions are today, so why do we need
preempt_disableorsrcu_fastthere?Either preempt disable or
srcu_fastis still needed.Quoted 3 lines
I think today it's there because all callbacks (perf, ftrace, bpf) expect preemption to be disabled, but can we just remove it from tp side? and move
preempt_disableto callbacks that actually need it?Yes if we are talking about switching from
preempt_disabletosrc_fast. No if you mean to remove both as it still needs RCU protection. It's used for synchronizing changes in the tracepoint infrastructure itself. That__DO_TRACE_CALL()macro is the guts of the tracepoint callback code. It needs to handle races with additions and removals of callbacks there, as the callbacks also get data passed to them. If it gets out of sync, a callback could be called with another callback's data.That's why it has:
it_func_ptr = \ rcu_dereference_raw((&__tracepoint_##name)->funcs);Quoted 1 line
Quoted 8 lines
Quoted 1 line
I'm looking at
release_probes(). It's fine as-is, no?That's just freeing, and as you see, there's RCU synchronization required.
Updates to tracepoints require RCU protection. It started out with
preempt_disable()for all tracepoints (which was synchronized withsynchronized_sched()which later became justsynchronize_rcu()).
I see.
Quoted 13 lines
The issue that came about was that both ftrace and perf had an assumption that its tracepoint callbacks always have preempt disabled when being called. To move to
srcu_fast()that is no longer the case. And we need to do that forPREEMPT_RTif BPF is running very long callbacks to the tracepoints.Ftrace has been fixed to not require it, but still needs to take into account if tracepoints disable preemption or not so that it can display the
preempt_count()of when the tracepoint was called correctly.Perf is trivial to fix as it can simply add a
preempt_disable()in its handler.
Quoted 8 lines
But we were originally told that BPF had an issue because it had the assumption that tracepoint callbacks wouldn't migrate. That's where this patch came about.
Now if you are saying that BPF will handle
migrate_disable()on its own and not require the tracepoint infrastructure to do it for it, then this is perfect. And I can then simplify this code, and just usesrcu_fastfor both RT and !RT.
Agree. Just add migrate_disable to __bpf_trace_run, or, better yet, use rcu_read_lock_dont_migrate() in there.
kernel/trace/bpf_trace.c +0 -0
@@ -2062,7 +2062,7 @@ void __bpf_trace_run(struct bpf_raw_tp_link
*link, u64 *args)
struct bpf_run_ctx *old_run_ctx;
struct bpf_trace_run_ctx run_ctx;- cant_sleep();
+ rcu_read_lock_dont_migrate();
if (unlikely(this_cpu_inc_return(*(prog->active)) != 1)) {
bpf_prog_inc_misses_counter(prog);
goto out;
@@ -2071,13 +2071,12 @@ void __bpf_trace_run(struct bpf_raw_tp_link
*link, u64 *args)
run_ctx.bpf_cookie = link->cookie;
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);- rcu_read_lock();
(void) bpf_prog_run(prog, args);- rcu_read_unlock();
bpf_reset_run_ctx(old_run_ctx);
out:
this_cpu_dec(*(prog->active));+ rcu_read_unlock_migrate();
}
Quoted 7 lines · Alexei Starovoitov
Quoted 4 lines
Now if you are saying that BPF will handle
migrate_disable()on its own and not require the tracepoint infrastructure to do it for it, then this is perfect. And I can then simplify this code, and just usesrcu_fastfor both RT and !RT.Agree. Just add
migrate_disableto__bpf_trace_run, or, better yet, usercu_read_lock_dont_migrate()in there.
Wonderful, thank you.
Is this "must remain on the same CPU and can be re-entrant" because BPF core code such memory allocator/ data structures use per-CPU data structures and must use the same through the whole invocation?
I did audit network related BPF code and their per-CPU usage usually had a local_bh_disable() in the relevant spots.
Sebastian
Quoted 7 lines · Sebastian Andrzej Siewior
Quoted 5 lines · Alexei Starovoitov
Quoted 4 lines
Now if you are saying that BPF will handle
migrate_disable()on its own and not require the tracepoint infrastructure to do it for it, then this is perfect. And I can then simplify this code, and just usesrcu_fastfor both RT and !RT.
Quoted 8 lines
Quoted 2 lines
Agree. Just add
migrate_disableto__bpf_trace_run, or, better yet, usercu_read_lock_dont_migrate()in there.Wonderful, thank you.
Is this "must remain on the same CPU and can be re-entrant" because BPF core code such memory allocator/ data structures use per-CPU data structures and must use the same through the whole invocation?
It's per-cpu maps.htab_percpu_map_lookup_elem() returns a pointer to per-cpu value which needs to be valid for the duration of the program. Not much else.
Yonghong Song
reviewerQuoted 16 lines · Alexei Starovoitov
Quoted 14 lines · Mathieu Desnoyers
Quoted 1 line · Steven Rostedt
From: "Paul E. McKenney" <[email protected]>
[...]
I disagree with many elements of the proposed approach.
On one end we have BPF wanting to hook on arbitrary tracepoints without adding significant latency to PREEMPT RT kernels.
One the other hand, we have high-speed tracers which execute very short critical sections to serialize trace data into ring buffers.
All of those users register to the tracepoint API.
Quoted 8 lines
Quoted 2 lines
We also have to consider that migrate disable is *not* cheap at all compared to preempt disable.
Looks like your complaint comes from lack of engagement in kernel development.
migrate_disable_was_ not cheap.
Try to benchmark it now.
It's inlined. It's a fraction of extra overhead on top ofpreempt_disable.
The following are related patches to inline migrate_disable():
35561bab768977c9e05f1f1a9bc00134c85f3e28 arch: Add the macro COMPILE_OFFSETS to all the asm-offsets.c 88a90315a99a9120cd471bf681515cc77cd7cdb8 rcu: Replace preempt.h with sched.h in include/linux/rcupdate.h 378b7708194fff77c9020392067329931c3fcc04 sched: Make migrate_{en,dis}able() inline