Sleepable tracepoints
Subsystems (16)
cover Sleepable tracepoints
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow specific tracer probes to take page faults. Adapt ftrace, perf, and ebpf to allow being called from sleepable context, and convert the system call enter/exit instrumentation to sleepable tracepoints.
This series only implements the tracepoint infrastructure required to allow tracers to handle page faults. Modifying each tracer to handle those page faults would be a next step after we all agree on this piece of instrumentation infrastructure.
This patchset is base on v5.9.1.
Mathieu Desnoyers (1):
tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
Michael Jeanson (5):
tracing: introduce sleepable tracepoints
tracing: ftrace: add support for sleepable tracepoints
tracing: bpf-trace: add support for sleepable tracepoints
tracing: perf: add support for sleepable tracepoints
tracing: convert sys_enter/exit to sleepable tracepoints
include/linux/tracepoint-defs.h | 11 ++++ include/linux/tracepoint.h | 104 +++++++++++++++++++++----------- include/trace/bpf_probe.h | 23 ++++++- include/trace/define_trace.h | 7 +++ include/trace/events/syscalls.h | 4 +- include/trace/perf.h | 26 ++++++-- include/trace/trace_events.h | 79 ++++++++++++++++++++++-- init/Kconfig | 1 + kernel/trace/bpf_trace.c | 5 +- kernel/trace/trace_events.c | 15 ++++- kernel/trace/trace_syscalls.c | 68 +++++++++++++-------- kernel/tracepoint.c | 104 +++++++++++++++++++++++++------- 12 files changed, 351 insertions(+), 96 deletions(-)
Signature · 1 line
2.25.1
RFC PATCH 1/6 tracing: introduce sleepable tracepoints
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a sleepable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those sleepable tracepoints. When the TRACEPOINT_MAYSLEEP flag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Sleepable probes can only be registered on sleepable tracepoints and non-sleepable probes on non-sleepable tracepoints.
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to sleepable probes unregistration and teardown.
--- include/linux/tracepoint-defs.h | 11 ++++ include/linux/tracepoint.h | 85 +++++++++++++++++++++----- include/trace/define_trace.h | 7 +++ include/trace/trace_events.h | 6 ++ init/Kconfig | 1 + kernel/tracepoint.c | 103 ++++++++++++++++++++++++++------ 6 files changed, 181 insertions(+), 32 deletions(-)
include/linux/tracepoint-defs.h +11 -0
@@ -27,12 +27,23 @@ struct tracepoint_func {int prio; }; +/** +* enum tracepoint_flags - Tracepoint flags +* @TRACEPOINT_MAYSLEEP: The tracepoint probe callback will be called with +* preemption enabled, and is allowed to take page +* faults. +*/ +enum tracepoint_flags { +TRACEPOINT_MAYSLEEP = (1 <<0 ), +}; +struct tracepoint {const char *name;/* Tracepoint name */ struct static_key key;int (*regfunc)(void );void (*unregfunc)(void );struct tracepoint_func __rcu *funcs; +unsigned int flags; };#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS
include/linux/tracepoint.h +70 -15
217 lines not on this page · raw message
include/trace/define_trace.h +7 -0
@@ -41,6 +41,12 @@ assign, print, reg, unreg) \ DEFINE_TRACE_FN(name, reg, unreg) +/* Define a trace event with the MAYSLEEP flag set */ +#undef TRACE_EVENT_FN_MAYSLEEP +#define TRACE_EVENT_FN_MAYSLEEP (name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + DEFINE_TRACE_FN_FLAGS(name, reg, unreg, TRACEPOINT_MAYSLEEP) +#undef TRACE_EVENT_FN_COND#define TRACE_EVENT_FN_COND (name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \
@@ -106,6 +112,7 @@#undef TRACE_EVENT#undef TRACE_EVENT_FN +#undef TRACE_EVENT_FN_MAYSLEEP#undef TRACE_EVENT_FN_COND#undef TRACE_EVENT_CONDITION#undef TRACE_EVENT_NOP
include/trace/trace_events.h +6 -0
@@ -138,6 +138,12 @@ TRACE_MAKE_SYSTEM_STR();TRACE_EVENT (name,PARAMS (proto),PARAMS (args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_MAYSLEEP +#define TRACE_EVENT_FN_MAYSLEEP (name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_COND#define TRACE_EVENT_FN_COND (name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \
init/Kconfig +1 -0
@@ -2018,6 +2018,7 @@ config PROFILING # config TRACEPOINTS bool + select TASKS_TRACE_RCU endmenu # General setup
kernel/tracepoint.c +86 -17
165 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [RFC PATCH 1/6] tracing: introduce sleepable tracepoints
Quoted 12 lines · Michael Jeanson
include/linux/tracepoint.h +2
-1-#define __DO_TRACE (tp, proto, args, cond, rcuidle) \ +#define __DO_TRACE (tp, proto, args, cond, rcuidle, tp_flags) \ do { \ struct tracepoint_func *it_func_ptr; \ void *it_func; \ void *__data; \ int __maybe_unused __idx = 0; \ + bool maysleep = (tp_flags) & TRACEPOINT_MAYSLEEP; \ \ if (!(cond)) \ return; \
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 182
Quoted 7 lines
include/linux/tracepoint.h +2
-2@@ -170,8 +178,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)/* srcu can't be used from NMI */ \WARN_ON_ONCE (rcuidle &&in_nmi ()); \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (maysleep) { \ +might_sleep (); \
The main purpose of the patch set is to access user memory in tracepoints, right? In such case I suggest to use stronger might_fault() here.
We used might_sleep() in sleepable bpf and it wasn't enough to catch a combination where sleepable hook was invoked while mm->mmap_lock was taken which may cause a deadlock.
Quoted 5 lines
include/linux/tracepoint.h +5
-0+rcu_read_lock_trace (); \ + }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \
Mathieu Desnoyers
reviewerRe: [RFC PATCH 1/6] tracing: introduce sleepable tracepoints
Quoted 16 lines · ----- On Oct 26, 2020, at 6:43 PM, Alexei Starovoitov…
Quoted 15 lines · Michael Jeanson
include/linux/tracepoint.h +2
-1-#define __DO_TRACE (tp, proto, args, cond, rcuidle) \ +#define __DO_TRACE (tp, proto, args, cond, rcuidle, tp_flags) \ do { \ struct tracepoint_func *it_func_ptr; \ void *it_func; \ void *__data; \ int __maybe_unused __idx = 0; \ + bool maysleep = (tp_flags) & TRACEPOINT_MAYSLEEP; \ \ if (!(cond)) \ return; \@@ -170,8 +178,13 @@ static inline struct tracepoint *tracepoint_ptr_deref (tracepoint_ptr_t *p) /* srcu can't be used from NMI */ \WARN_ON_ONCE (rcuidle &&in_nmi ()); \
Quoted 8 lines
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 182
Quoted 5 lines
include/linux/tracepoint.h +2
-2\ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (maysleep) { \ +might_sleep (); \The main purpose of the patch set is to access user memory in tracepoints, right?
Yes, exactly.
Quoted 4 lines
In such case I suggest to use stronger
might_fault()here. We usedmight_sleep()in sleepable bpf and it wasn't enough to catch a combination where sleepable hook was invoked whilemm->mmap_lockwas taken which may cause a deadlock.
Good point! We will do that for the next round.
By the way, we named this "sleepable" tracepoint (with flag TRACEPOINT_MAYSLEEP), but we are open to a better name. Would TRACEPOINT_MAYFAULT be more descriptive ? (a "faultable" tracepoint sounds weird though)
Thanks,
Mathieu
Quoted 6 lines
Quoted 6 lines
include/linux/tracepoint.h +5
-0+rcu_read_lock_trace (); \ + }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 1/6] tracing: introduce sleepable tracepoints
Quoted 35 lines · Mathieu Desnoyers
Quoted 16 lines · ----- On Oct 26, 2020, at 6:43 PM, Alexei Starovoitov…
Quoted 15 lines · Michael Jeanson
include/linux/tracepoint.h +2
-1-#define __DO_TRACE (tp, proto, args, cond, rcuidle) \ +#define __DO_TRACE (tp, proto, args, cond, rcuidle, tp_flags) \ do { \ struct tracepoint_func *it_func_ptr; \ void *it_func; \ void *__data; \ int __maybe_unused __idx = 0; \ + bool maysleep = (tp_flags) & TRACEPOINT_MAYSLEEP; \ \ if (!(cond)) \ return; \@@ -170,8 +178,13 @@ static inline struct tracepoint *tracepoint_ptr_deref (tracepoint_ptr_t *p) /* srcu can't be used from NMI */ \WARN_ON_ONCE (rcuidle &&in_nmi ()); \Quoted 8 lines
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 182
Quoted 5 lines
include/linux/tracepoint.h +2
-2\ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (maysleep) { \ +might_sleep (); \The main purpose of the patch set is to access user memory in tracepoints, right?
Yes, exactly.
Quoted 4 lines
In such case I suggest to use stronger
might_fault()here. We usedmight_sleep()in sleepable bpf and it wasn't enough to catch a combination where sleepable hook was invoked whilemm->mmap_lockwas taken which may cause a deadlock.
Quoted 5 lines
Good point! We will do that for the next round.
By the way, we named this "sleepable" tracepoint (with flag
TRACEPOINT_MAYSLEEP), but we are open to a better name. WouldTRACEPOINT_MAYFAULTbe more descriptive ? (a "faultable" tracepoint sounds weird though)
bpf kept 'sleepable' as a name. 'faultable' is too misleading.
Mathieu Desnoyers
reviewerRe: [RFC PATCH 1/6] tracing: introduce sleepable tracepoints
Quoted 39 lines · ----- On Oct 28, 2020, at 5:23 PM, Alexei Starovoitov…
Quoted 38 lines · Mathieu Desnoyers
----- On Oct 26, 2020, at 6:43 PM, Alexei Starovoitov
Quoted 16 lines · [email protected] wrote:
Quoted 15 lines · Michael Jeanson
include/linux/tracepoint.h +2
-1-#define __DO_TRACE (tp, proto, args, cond, rcuidle) \ +#define __DO_TRACE (tp, proto, args, cond, rcuidle, tp_flags) \ do { \ struct tracepoint_func *it_func_ptr; \ void *it_func; \ void *__data; \ int __maybe_unused __idx = 0; \ + bool maysleep = (tp_flags) & TRACEPOINT_MAYSLEEP; \ \ if (!(cond)) \ return; \@@ -170,8 +178,13 @@ static inline struct tracepoint *tracepoint_ptr_deref (tracepoint_ptr_t *p) /* srcu can't be used from NMI */ \WARN_ON_ONCE (rcuidle &&in_nmi ()); \Quoted 8 lines
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 182
Quoted 5 lines
include/linux/tracepoint.h +2
-2\ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (maysleep) { \ +might_sleep (); \The main purpose of the patch set is to access user memory in tracepoints, right?
Yes, exactly.
Quoted 4 lines
In such case I suggest to use stronger
might_fault()here. We usedmight_sleep()in sleepable bpf and it wasn't enough to catch a combination where sleepable hook was invoked whilemm->mmap_lockwas taken which may cause a deadlock.Good point! We will do that for the next round.
Quoted 7 lines
Quoted 5 lines
By the way, we named this "sleepable" tracepoint (with flag
TRACEPOINT_MAYSLEEP),
but we are open to a better name. WouldTRACEPOINT_MAYFAULTbe more descriptive ?
(a "faultable" tracepoint sounds weird though)bpf kept 'sleepable' as a name. 'faultable' is too misleading.
We're working on an updated patchset for those "sleepable tracepoints", and considering that those are really "tracepoints allowing page faults", I must admit that I am uncomfortable with the confusion between "sleep" and "fault" in the naming here.
I am tempted to do the following changes:
- Change name from "sleepable tracepoints" to a better suited "tracepoints allowing page faults",
- Use might_fault() rather than might_sleep() in __DO_TRACE(), effectively guaranteeing that all probes connecting to a tracepoint which allows page faults can indeed take page faults.
- Change TRACEPOINT_MAYSLEEP into TRACEPOINT_MAYFAULT.
Any objections ?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Re: [RFC PATCH 1/6] tracing: introduce sleepable tracepoints
In reply to Mathieu DesnoyersQuoted 35 lines · Mathieu Desnoyers
Quoted 16 lines · ----- On Oct 26, 2020, at 6:43 PM, Alexei Starovoitov…
Quoted 15 lines · Michael Jeanson
include/linux/tracepoint.h +2
-1-#define __DO_TRACE (tp, proto, args, cond, rcuidle) \ +#define __DO_TRACE (tp, proto, args, cond, rcuidle, tp_flags) \ do { \ struct tracepoint_func *it_func_ptr; \ void *it_func; \ void *__data; \ int __maybe_unused __idx = 0; \ + bool maysleep = (tp_flags) & TRACEPOINT_MAYSLEEP; \ \ if (!(cond)) \ return; \@@ -170,8 +178,13 @@ static inline struct tracepoint *tracepoint_ptr_deref (tracepoint_ptr_t *p) /* srcu can't be used from NMI */ \WARN_ON_ONCE (rcuidle &&in_nmi ()); \Quoted 8 lines
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 182
Quoted 5 lines
include/linux/tracepoint.h +2
-2\ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (maysleep) { \ +might_sleep (); \The main purpose of the patch set is to access user memory in tracepoints, right?
Yes, exactly.
Quoted 4 lines
In such case I suggest to use stronger
might_fault()here. We usedmight_sleep()in sleepable bpf and it wasn't enough to catch a combination where sleepable hook was invoked whilemm->mmap_lockwas taken which may cause a deadlock.
Quoted 5 lines
Good point! We will do that for the next round.
By the way, we named this "sleepable" tracepoint (with flag
TRACEPOINT_MAYSLEEP), but we are open to a better name. WouldTRACEPOINT_MAYFAULTbe more descriptive ? (a "faultable" tracepoint sounds weird though)
What about keeping it might_sleep() here and then adding might_fault() in the probe handler? Since the probe handler knows that it may cause page fault, it could itself make sure about it.
One more thought: Should we make _all_ tracepoints sleepable, and then move the preempt_disable() bit to the probe handler as needed? That could simplify the tracepoint API as well. Steven said before that whoever registers probes knows what they are doing so I am ok with that.
No strong feelings one way or the other, for either of these though.
thanks,
- Joel
Quoted 16 lines
Thanks,
Mathieu
Quoted 6 lines
Quoted 6 lines
include/linux/tracepoint.h +5
-0+rcu_read_lock_trace (); \ + }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
RFC PATCH 2/6 tracing: ftrace: add support for sleepable tracepoints
In reply to Michael JeansonIn preparation for converting system call enter/exit instrumentation into sleepable tracepoints, make sure that ftrace can handle registering to such tracepoints by explicitly disabling preemption within the ftrace tracepoint probes to respect the current expectations within ftrace ring buffer code.
This change does not yet allow ftrace to take page faults per se within its probe, but allows its existing probes to connect to sleepable tracepoints.
--- include/trace/trace_events.h | 75 +++++++++++++++++++++++++++++++++--- kernel/trace/trace_events.c | 15 +++++++- 2 files changed, 83 insertions(+), 7 deletions(-)
include/trace/trace_events.h +70 -5
@@ -80,6 +80,16 @@ TRACE_MAKE_SYSTEM_STR(); PARAMS(print)); \DEFINE_EVENT (name, name,PARAMS (proto),PARAMS (args)); +#undef TRACE_EVENT_MAYSLEEP +#define TRACE_EVENT_MAYSLEEP (name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS_MAYSLEEP(name, \ + PARAMS(proto), \ + PARAMS(args), \ + PARAMS(tstruct), \ + PARAMS(assign), \ + PARAMS(print)); \ + DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args)); +#undef __field#define __field (type, item) type item;
@@ -118,6 +128,12 @@ TRACE_MAKE_SYSTEM_STR(); \ static struct trace_event_class event_class_##name; +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (name, proto, args, \ + tstruct, assign, print) \ + DECLARE_EVENT_CLASS(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, name, proto, args) \ static struct trace_event_call __used \
@@ -141,7 +157,7 @@ TRACE_MAKE_SYSTEM_STR();#undef TRACE_EVENT_FN_MAYSLEEP#define TRACE_EVENT_FN_MAYSLEEP (name, proto, args, tstruct, \ assign, print, reg, unreg) \ - TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + TRACE_EVENT_MAYSLEEP(name, PARAMS(proto), PARAMS(args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \#undef TRACE_EVENT_FN_COND
@@ -212,6 +228,12 @@ TRACE_MAKE_SYSTEM_STR(); tstruct; \ }; +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, \ + tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, name, proto, args)
@@ -378,6 +400,12 @@ static struct trace_event_functions trace_event_type_funcs_##call = { \ .trace = trace_raw_output_##call, \ }; +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, \ + tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT_PRINT#define DEFINE_EVENT_PRINT (template, call, proto, args, print) \ static notrace enum print_line_t \
@@ -448,6 +476,12 @@ static struct trace_event_fields trace_event_fields_##call[] = { \ tstruct \ {} }; +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, \ + tstruct, func, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(func), PARAMS(print)) +#undef DEFINE_EVENT_PRINT#define DEFINE_EVENT_PRINT (template, name, proto, args, print)
@@ -524,6 +558,12 @@ static inline notrace int trace_event_get_offsets_##call( \return __data_size; \ } +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, \ + tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#include TRACE_INCLUDE (TRACE_INCLUDE_FILE )/*
@@ -673,8 +713,8 @@ static inline notrace int trace_event_get_offsets_##call( \#undef __perf_task#define __perf_task (t) (t) -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \
@@ -685,8 +725,11 @@ trace_event_raw_event_##call(void *__data, proto) \ struct trace_event_raw_##call *entry; \ int __data_size; \ \ + if ((tp_flags) & TRACEPOINT_MAYSLEEP) \ + preempt_disable_notrace(); \ + \ if (trace_trigger_soft_disabled(trace_file)) \ - return; \ + goto end; \ \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \
@@ -694,14 +737,30 @@ trace_event_raw_event_##call(void *__data, proto) \ sizeof(*entry) + __data_size); \ \if (!entry) \ -return ; \ + goto end; \ \ tstruct \ \ { assign; } \ \trace_event_buffer_commit (&fbuffer); \ +end: \ +if ((tp_flags) &TRACEPOINT_MAYSLEEP ) \ +preempt_enable_notrace (); \ } + +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), \ + PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), \ + PARAMS(print), TRACEPOINT_MAYSLEEP) +/* * The ftrace_test_probe is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the ftrace probe will
@@ -748,6 +807,12 @@ static struct trace_event_class __used __refdata event_class_##call = { \ _TRACE_PERF_INIT(call) \ }; +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, \ + tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, call, proto, args) \ \
kernel/trace/trace_events.c +13 -2
@@ -290,9 +290,15 @@ int trace_event_reg(struct trace_event_call *call,WARN_ON (!(call->flags &TRACE_EVENT_FL_TRACEPOINT ));switch (type) {case TRACE_REG_REGISTER : -return tracepoint_probe_register (call->tp, +if (call->tp->flags &TRACEPOINT_MAYSLEEP ) +return tracepoint_probe_register_maysleep (call->tp, call->class->probe, file); +else +return tracepoint_probe_register (call->tp, + call->class->probe, + file); +case TRACE_REG_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->probe,
@@ -301,7 +307,12 @@ int trace_event_reg(struct trace_event_call *call,#ifdef CONFIG_PERF_EVENTS case TRACE_REG_PERF_REGISTER : -return tracepoint_probe_register (call->tp, +if (call->tp->flags &TRACEPOINT_MAYSLEEP ) +return tracepoint_probe_register_maysleep (call->tp, + call->class->perf_probe, + call); +else +return tracepoint_probe_register (call->tp, call->class->perf_probe, call);case TRACE_REG_PERF_UNREGISTER :
Signature · 1 line
2.25.1
RFC PATCH 3/6 tracing: bpf-trace: add support for sleepable tracepoints
In reply to Michael JeansonIn preparation for converting system call enter/exit instrumentation into sleepable tracepoints, make sure that bpf can handle registering to such tracepoints by explicitly disabling preemption within the bpf tracepoint probes to respect the current expectations within bpf tracing code.
This change does not yet allow bpf to take page faults per se within its probe, but allows its existing probes to connect to sleepable tracepoints.
--- include/trace/bpf_probe.h | 23 +++++++++++++++++++++-- kernel/trace/bpf_trace.c | 5 ++++- 2 files changed, 25 insertions(+), 3 deletions(-)
include/trace/bpf_probe.h +21 -2
@@ -55,15 +55,34 @@/* tracepoints with more than 12 arguments will hit build error */ #define CAST_TO_U64 (...) CONCATENATE(__CAST, COUNT_ARGS(__VA_ARGS__))(__VA_ARGS__) -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ static notrace void \ __bpf_trace_##call(void *__data, proto) \ { \ struct bpf_prog *prog = __data; \ + \ + if ((tp_flags) & TRACEPOINT_MAYSLEEP) \ + preempt_disable_notrace(); \ + \ CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(prog, CAST_TO_U64(args)); \ + \ + if ((tp_flags) & TRACEPOINT_MAYSLEEP) \ + preempt_enable_notrace(); \ } +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, tstruct, \ + assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), \ + TRACEPOINT_MAYSLEEP) +/* * This part is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the
kernel/trace/bpf_trace.c +4 -1
@@ -1947,7 +1947,10 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *if (prog->aux->max_tp_access > btp->writable_size)return -EINVAL ; -return tracepoint_probe_register (tp, (void *)btp->bpf_func, prog); +if (tp->flags &TRACEPOINT_MAYSLEEP ) +return tracepoint_probe_register_maysleep (tp, (void *)btp->bpf_func, prog); +else +return tracepoint_probe_register (tp, (void *)btp->bpf_func, prog); }int bpf_probe_register (struct bpf_raw_event_map *btp,struct bpf_prog *prog)
Signature · 1 line
2.25.1
RFC PATCH 4/6 tracing: perf: add support for sleepable tracepoints
In reply to Michael JeansonIn preparation for converting system call enter/exit instrumentation into sleepable tracepoints, make sure that perf can handle registering to such tracepoints by explicitly disabling preemption within the perf tracepoint probes to respect the current expectations within perf ring buffer code.
This change does not yet allow perf to take page faults per se within its probe, but allows its existing probes to connect to sleepable tracepoints.
--- include/trace/perf.h | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-)
include/trace/perf.h +22 -4
@@ -27,8 +27,8 @@#undef __perf_task#define __perf_task (t) (__task = (t)) -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ static notrace void \ perf_trace_##call(void *__data, proto) \ { \
@@ -43,13 +43,16 @@ perf_trace_##call(void *__data, proto) \int __data_size; \int rctx; \ \ +if ((tp_flags) &TRACEPOINT_MAYSLEEP ) \ +preempt_disable_notrace (); \ + \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \ head = this_cpu_ptr(event_call->perf_events); \ if (!bpf_prog_array_valid(event_call) && \ __builtin_constant_p(!__task) && !__task && \ hlist_empty(head)) \ - return; \ + goto end; \ \ __entry_size = ALIGN(__data_size + sizeof(*entry) + sizeof(u32),\ sizeof(u64)); \
@@ -57,7 +60,7 @@ perf_trace_##call(void *__data, proto) \ \ entry =perf_trace_buf_alloc (__entry_size, &__regs, &rctx); \if (!entry) \ -return ; \ + goto end; \ \perf_fetch_caller_regs (__regs); \ \
@@ -68,8 +71,23 @@ perf_trace_##call(void *__data, proto) \perf_trace_run_bpf_submit (entry, __entry_size, rctx, \ event_call, __count, __regs, \ head, __task); \ +end: \ +if ((tp_flags) &TRACEPOINT_MAYSLEEP ) \ +preempt_enable_notrace (); \ } +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAYSLEEP +#define DECLARE_EVENT_CLASS_MAYSLEEP (call, proto, args, tstruct, \ + assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), \ + TRACEPOINT_MAYSLEEP) +/* * This part is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the
Signature · 1 line
2.25.1
RFC PATCH 5/6 tracing: convert sys_enter/exit to sleepable tracepoints
In reply to Michael JeansonConvert the definition of the system call enter/exit tracepoints to sleepable tracepoints now that all upstream tracers handle it.
--- include/trace/events/syscalls.h | 4 +- kernel/trace/trace_syscalls.c | 68 ++++++++++++++++++++------------- 2 files changed, 44 insertions(+), 28 deletions(-)
include/trace/events/syscalls.h +2 -2
@@ -15,7 +15,7 @@#ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS -TRACE_EVENT_FN(sys_enter, +TRACE_EVENT_FN_MAYSLEEP(sys_enter, TP_PROTO(struct pt_regs *regs, long id),
@@ -41,7 +41,7 @@ TRACE_EVENT_FN(sys_enter, TRACE_EVENT_FLAGS(sys_enter, TRACE_EVENT_FL_CAP_ANY) -TRACE_EVENT_FN(sys_exit, +TRACE_EVENT_FN_MAYSLEEP(sys_exit, TP_PROTO(struct pt_regs *regs, long ret),
kernel/trace/trace_syscalls.c +42 -26
215 lines not on this page · raw message
Signature · 1 line
2.25.1
RFC PATCH 6/6 tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
In reply to Michael JeansonFrom: Mathieu Desnoyers <[email protected]>
Considering that tracer callbacks expect RCU to be watching (for instance, perf uses rcu_read_lock), we need rcuidle tracepoints to issue rcu_irq_{enter,exit}_irqson around calls to the callbacks. So there is no point in using SRCU anymore given that rcuidle tracepoints need to ensure RCU is watching. Therefore, simply use sched-RCU like normal tracepoints for rcuidle tracepoints.
--- include/linux/tracepoint.h | 33 +++++++-------------------------- kernel/tracepoint.c | 25 +++++++++---------------- 2 files changed, 16 insertions(+), 42 deletions(-)
include/linux/tracepoint.h +7 -26
@@ -13,7 +13,6 @@*/ #include <linux/smp.h> -#include <linux/srcu.h> #include <linux/errno.h> #include <linux/types.h> #include <linux/cpumask.h>
@@ -33,8 +32,6 @@ struct trace_eval_map {#define TRACEPOINT_DEFAULT_PRIO 10 -extern struct srcu_struct tracepoint_srcu; -extern int tracepoint_probe_register (struct tracepoint *tp,void *probe,void *data); extern int
@@ -86,7 +83,6 @@ int unregister_tracepoint_module_notifier(struct notifier_block *nb)static inline void tracepoint_synchronize_unregister (void ) {synchronize_rcu_tasks_trace (); -synchronize_srcu (&tracepoint_srcu);synchronize_rcu (); }#else
@@ -175,25 +171,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (!(cond)) \return ; \ \ -/* srcu can't be used from NMI */ \ -WARN_ON_ONCE (rcuidle &&in_nmi ()); \ - \ -if (maysleep) { \ -might_sleep (); \ +might_sleep_if (maysleep); \ +if (rcuidle) \ +rcu_irq_enter_irqson (); \ +if (maysleep) \rcu_read_lock_trace (); \ - }else { \ -/* keep srcu and sched-rcu usage consistent */ \ +else \preempt_disable_notrace (); \ - } \ - \ -/* \ -* For rcuidle callers, use srcu since sched-rcu \ -* doesn't work from the idle path. \ -*/ \ -if (rcuidle) { \ - __idx =srcu_read_lock_notrace (&tracepoint_srcu);\ -rcu_irq_enter_irqson (); \ - } \ \ it_func_ptr =rcu_dereference_raw ((tp)->funcs); \ \
@@ -205,15 +189,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) }while ((++it_func_ptr)->func); \ } \ \ -if (rcuidle) { \ -rcu_irq_exit_irqson (); \ -srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ - } \ - \if (maysleep) \rcu_read_unlock_trace (); \else \preempt_enable_notrace (); \ +if (rcuidle) \ +rcu_irq_exit_irqson (); \ }while (0 ) #ifndef MODULE
kernel/tracepoint.c +9 -16
@@ -18,9 +18,6 @@extern tracepoint_ptr_t __start___tracepoints_ptrs[];extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; -DEFINE_SRCU (tracepoint_srcu); -EXPORT_SYMBOL_GPL (tracepoint_srcu); -/* Set to 1 to enable tracepoint debug output */ static const int tracepoint_debug;
@@ -65,14 +62,9 @@ static void rcu_tasks_trace_free_old_probes(struct rcu_head *head)kfree (container_of(head ,struct tp_probes ,rcu )); } -static void srcu_free_old_probes (struct rcu_head *head) -{ -call_rcu_tasks_trace (head, rcu_tasks_trace_free_old_probes); -} -static void rcu_free_old_probes (struct rcu_head *head) { -call_srcu (&tracepoint_srcu, head, srcu_free_old_probes); +call_rcu_tasks_trace (head, rcu_tasks_trace_free_old_probes); }static __init int release_early_probes (void )
@@ -90,7 +82,7 @@ static __init int release_early_probes(void)return 0 ; } -/* SRCU and Tasks Trace RCU are initialized at core_initcall */ +/* Tasks Trace RCU is initialized at core_initcall */ postcore_initcall (release_early_probes);static inline void release_probes (struct tracepoint_func *old)
@@ -100,9 +92,8 @@ static inline void release_probes(struct tracepoint_func *old) struct tp_probes, probes[0]);/* -* We can't free probes if SRCU and Tasks Trace RCU are not -* initialized yet. Postpone the freeing till after both are -* initialized. +* We can't free probes if Tasks Trace RCU is not initialized yet. +* Postpone the freeing till after Tasks Trace RCU is initialized. */ if (unlikely (!ok_to_free_tracepoints)) { tp_probes->rcu.next = early_probes;
@@ -111,9 +102,11 @@ static inline void release_probes(struct tracepoint_func *old) }/* -* Tracepoint probes are protected by sched RCU, SRCU and -* Tasks Trace RCU by chaining the callbacks we cover all three -* cases and wait for all three grace periods. +* Tracepoint probes are protected by both sched RCU and +* Tasks Trace RCU, by calling the Tasks Trace RCU callback in +* the sched RCU callback we cover both cases. So let us chain +* the Tasks Trace RCU and sched RCU callbacks to wait for both +* grace periods. */ call_rcu (&tp_probes->rcu, rcu_free_old_probes); }
Signature · 1 line
2.25.1
Re: [RFC PATCH 6/6] tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
Quoted 8 lines · Michael Jeanson
From: Mathieu Desnoyers <[email protected]>
Considering that tracer callbacks expect RCU to be watching (for instance, perf uses
rcu_read_lock), we need rcuidle tracepoints to issue rcu_irq_{enter,exit}_irqson around calls to the callbacks. So there is no point in using SRCU anymore given that rcuidle tracepoints need to ensure RCU is watching. Therefore, simply use sched-RCU like normal tracepoints for rcuidle tracepoints.
High level question:
IIRC, doing this increases overhead for general tracing that does not use perf, for 'rcuidle' tracepoints such as the preempt/irq enable/disable tracepoints. I remember adding SRCU because of this reason.
Can the 'rcuidle' information not be pushed down further, such that perf does it because it requires RCU to be watching, so that it does not effect, say, trace events?
thanks,
- Joel
Quoted 167 lines
Signed-off-by Mathieu Desnoyers <[email protected]>Cc Michael Jeanson <[email protected]>Cc Steven Rostedt (VMware) <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Alexei Starovoitov <[email protected]>Cc Yonghong Song <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Ingo Molnar <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Mark Rutland <[email protected]>Cc Alexander Shishkin <[email protected]>Cc Jiri Olsa <[email protected]>Cc Namhyung Kim <[email protected]>Cc Joel Fernandes (Google) <[email protected]>--- include/linux/tracepoint.h | 33 +++++++-------------------------- kernel/tracepoint.c | 25 +++++++++---------------- 2 files changed, 16 insertions(+), 42 deletions(-)include/linux/tracepoint.h +7
-26@@ -13,7 +13,6 @@*/ #include <linux/smp.h> -#include <linux/srcu.h> #include <linux/errno.h> #include <linux/types.h> #include <linux/cpumask.h> @@ -33,8 +32,6 @@ struct trace_eval_map {#define TRACEPOINT_DEFAULT_PRIO 10 -extern struct srcu_struct tracepoint_srcu; -extern int tracepoint_probe_register (struct tracepoint *tp,void *probe,void *data); extern int@@ -86,7 +83,6 @@ int unregister_tracepoint_module_notifier(struct notifier_block *nb)static inline void tracepoint_synchronize_unregister (void ) {synchronize_rcu_tasks_trace (); -synchronize_srcu (&tracepoint_srcu);synchronize_rcu (); }#else @@ -175,25 +171,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (!(cond)) \return ; \ \ -/* srcu can't be used from NMI */ \ -WARN_ON_ONCE (rcuidle &&in_nmi ()); \ - \ -if (maysleep) { \ -might_sleep (); \ +might_sleep_if (maysleep); \ +if (rcuidle) \ +rcu_irq_enter_irqson (); \ +if (maysleep) \rcu_read_lock_trace (); \ - }else { \ -/* keep srcu and sched-rcu usage consistent */ \ +else \preempt_disable_notrace (); \ - } \ - \ -/* \ -* For rcuidle callers, use srcu since sched-rcu \ -* doesn't work from the idle path. \ -*/ \ -if (rcuidle) { \ - __idx =srcu_read_lock_notrace (&tracepoint_srcu);\ -rcu_irq_enter_irqson (); \ - } \ \ it_func_ptr =rcu_dereference_raw ((tp)->funcs); \ \@@ -205,15 +189,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) }while ((++it_func_ptr)->func); \ } \ \ -if (rcuidle) { \ -rcu_irq_exit_irqson (); \ -srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ - } \ - \if (maysleep) \rcu_read_unlock_trace (); \else \preempt_enable_notrace (); \ +if (rcuidle) \ +rcu_irq_exit_irqson (); \ }while (0 ) #ifndef MODULEkernel/tracepoint.c +9
-16@@ -18,9 +18,6 @@extern tracepoint_ptr_t __start___tracepoints_ptrs[];extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; -DEFINE_SRCU (tracepoint_srcu); -EXPORT_SYMBOL_GPL (tracepoint_srcu); -/* Set to 1 to enable tracepoint debug output */ static const int tracepoint_debug;@@ -65,14 +62,9 @@ static void rcu_tasks_trace_free_old_probes(struct rcu_head *head)kfree (container_of(head ,struct tp_probes ,rcu )); } -static void srcu_free_old_probes (struct rcu_head *head) -{ -call_rcu_tasks_trace (head, rcu_tasks_trace_free_old_probes); -} -static void rcu_free_old_probes (struct rcu_head *head) { -call_srcu (&tracepoint_srcu, head, srcu_free_old_probes); +call_rcu_tasks_trace (head, rcu_tasks_trace_free_old_probes); }static __init int release_early_probes (void )@@ -90,7 +82,7 @@ static __init int release_early_probes(void)return 0 ; } -/* SRCU and Tasks Trace RCU are initialized at core_initcall */ +/* Tasks Trace RCU is initialized at core_initcall */ postcore_initcall (release_early_probes);static inline void release_probes (struct tracepoint_func *old)@@ -100,9 +92,8 @@ static inline void release_probes(struct tracepoint_func *old) struct tp_probes, probes[0]);/* -* We can't free probes if SRCU and Tasks Trace RCU are not -* initialized yet. Postpone the freeing till after both are -* initialized. +* We can't free probes if Tasks Trace RCU is not initialized yet. +* Postpone the freeing till after Tasks Trace RCU is initialized. */ if (unlikely (!ok_to_free_tracepoints)) { tp_probes->rcu.next = early_probes;@@ -111,9 +102,11 @@ static inline void release_probes(struct tracepoint_func *old) }/* -* Tracepoint probes are protected by sched RCU, SRCU and -* Tasks Trace RCU by chaining the callbacks we cover all three -* cases and wait for all three grace periods. +* Tracepoint probes are protected by both sched RCU and +* Tasks Trace RCU, by calling the Tasks Trace RCU callback in +* the sched RCU callback we cover both cases. So let us chain +* the Tasks Trace RCU and sched RCU callbacks to wait for both +* grace periods. */ call_rcu (&tp_probes->rcu, rcu_free_old_probes); }Signature · 1 line
2.25.1
Re: [RFC PATCH 6/6] tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
Quoted 12 lines · Joel Fernandes
Quoted 8 lines · Michael Jeanson
From: Mathieu Desnoyers <[email protected]>
Considering that tracer callbacks expect RCU to be watching (for instance, perf uses
rcu_read_lock), we need rcuidle tracepoints to issue rcu_irq_{enter,exit}_irqson around calls to the callbacks. So there is no point in using SRCU anymore given that rcuidle tracepoints need to ensure RCU is watching. Therefore, simply use sched-RCU like normal tracepoints for rcuidle tracepoints.High level question:
Quoted 7 lines
IIRC, doing this increases overhead for general tracing that does not use perf, for 'rcuidle' tracepoints such as the preempt/irq enable/disable tracepoints. I remember adding SRCU because of this reason.
Can the 'rcuidle' information not be pushed down further, such that perf does it because it requires RCU to be watching, so that it does not effect, say, trace events?
There's very few trace_.*_rcuidle() users left. We should eradicate them and remove the option. It's bugs to begin with.
Mathieu Desnoyers
reviewerRe: [RFC PATCH 6/6] tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
Quoted 17 lines · ----- On Oct 26, 2020, at 4:20 AM, Peter Zijlstra [email protected] wrote:
Quoted 16 lines · Joel Fernandes
Quoted 8 lines · Michael Jeanson
From: Mathieu Desnoyers <[email protected]>
Considering that tracer callbacks expect RCU to be watching (for instance, perf uses
rcu_read_lock), we need rcuidle tracepoints to issue rcu_irq_{enter,exit}_irqson around calls to the callbacks. So there is no point in using SRCU anymore given that rcuidle tracepoints need to ensure RCU is watching. Therefore, simply use sched-RCU like normal tracepoints for rcuidle tracepoints.High level question:
IIRC, doing this increases overhead for general tracing that does not use perf, for 'rcuidle' tracepoints such as the preempt/irq enable/disable tracepoints. I remember adding SRCU because of this reason.
Quoted 6 lines
Quoted 3 lines
Can the 'rcuidle' information not be pushed down further, such that perf does it because it requires RCU to be watching, so that it does not effect, say, trace events?
There's very few trace_.*_rcuidle() users left. We should eradicate them and remove the option. It's bugs to begin with.
I agree with Peter. Removing the trace_.*_rcuidle weirdness from the tracepoint API and fixing all callers to ensure they trace from a context where RCU is watching would simplify instrumentation of the Linux kernel, thus making it harder for subtle bugs to hide and be unearthed only when tracing is enabled. This is AFAIU the general approach Thomas Gleixner has been aiming for recently, and I think it is a good thing.
So if we consider this our target, and that the current state of things is that we need to have RCU watching around callback invocation, then removing the dependency on SRCU seems like an overall simplification which does not regress feature-wise nor speed-wise compared with what we have upstream today. The next steps would then be to audit all rcuidle tracepoints and make sure the context where they are placed has RCU watching already, so we can remove the tracepoint rcuidle API. That would effectively remove the calls to rcu_irq_{enter,exit}_irqson from the tracepoint code.
This is however beyond the scope of the proposed patch set.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Steven Rostedt
maintainerRe: [RFC PATCH 6/6] tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
Quoted 4 lines · On Mon, 26 Oct 2020 10:28:07 -0400 (EDT) Mathieu Desnoyers…
I agree with Peter. Removing the trace_.*_rcuidle weirdness from the tracepoint API and fixing all callers to ensure they trace from a context where RCU is watching would simplify instrumentation of the Linux kernel, thus making it harder for subtle bugs to hide and be unearthed only when tracing is enabled. This is
Note, the lockdep RCU checking of a tracepoint is outside of it being enabled or disable. So if a non rcuidle() tracepoint is in a location that RCU is not watching, it will complain loudly, even if you don't enable that tracepoint.
Quoted 9 lines
AFAIU the general approach Thomas Gleixner has been aiming for recently, and I think it is a good thing.
So if we consider this our target, and that the current state of things is that we need to have RCU watching around callback invocation, then removing the dependency on SRCU seems like an overall simplification which does not regress feature-wise nor speed-wise compared with what we have upstream today. The next steps would then be to audit all rcuidle tracepoints and make sure the context where they are placed has RCU watching already, so we can remove the tracepoint
Just remove the _rcuidle() from them, and lockdep will complain if they are being called without RCU watching.
-- Steve
Quoted 9 lines
rcuidle API. That would effectively remove the calls to rcu_irq_{enter,exit}_irqson from the tracepoint code.
This is however beyond the scope of the proposed patch set.
Thanks,
Mathieu
Mathieu Desnoyers
reviewerRe: [RFC PATCH 6/6] tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
Quoted 24 lines · ----- On Oct 26, 2020, at 4:44 PM, rostedt [email protected] wrote:
Quoted 5 lines · On Mon, 26 Oct 2020 10:28:07 -0400 (EDT) Mathieu Desnoyers…
I agree with Peter. Removing the trace_.*_rcuidle weirdness from the tracepoint API and fixing all callers to ensure they trace from a context where RCU is watching would simplify instrumentation of the Linux kernel, thus making it harder
for subtle bugs to hide and be unearthed only when tracing is enabled. This isNote, the lockdep RCU checking of a tracepoint is outside of it being enabled or disable. So if a non
rcuidle()tracepoint is in a location that RCU is not watching, it will complain loudly, even if you don't enable that tracepoint.Quoted 9 lines
AFAIU the general approach Thomas Gleixner has been aiming for recently, and I think it is a good thing.
So if we consider this our target, and that the current state of things is that we need to have RCU watching around callback invocation, then removing the dependency on SRCU seems like an overall simplification which does not regress feature-wise nor speed-wise compared with what we have upstream today. The next steps would then be to audit all rcuidle tracepoints and make sure the context where they are placed has RCU watching already, so we can remove the tracepoint
Quoted 2 lines
Just remove the
_rcuidle()from them, and lockdep will complain if they are being called without RCU watching.
That's the easy part. The patch removing rcuidle is attached to this email, and here are the splats I get just when booting the machine (see below). The part which takes more time is fixing those splats and figuring out how to restructure the code. For instance, what should we do about the rcuidle tracepoint within printk() ?
Also, how do we test rcuidle tracepoints triggered only when printk is called from printk warnings ? We'd also need to test on arm32 boards, because some arm architecture code uses rcuidle tracepoints as well.
As this is beyond the scope of this patch set, I can either drop this patch entirely (it's not required for sleepable tracepoints), or keep it as an intermediate cleanup step. Removing rcuidle tracepoints entirely is beyond the effort Michael and I can undertake now.
=============================
WARNING: suspicious RCU usage
5.9.1+ #72 Not tainted
-----------------------------
=============================./include/trace/events/preemptirq.h:42 suspicious rcu_dereference_check() usage!
WARNING: suspicious RCU usage
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
5.9.1+ #72 Not tainted
RCU used illegally from extended quiescent state!
no locks held by swapper/0/0.
-----------------------------./include/trace/events/preemptirq.h:38 suspicious rcu_dereference_check() usage!
stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.1+ #72
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: RCU used illegally from extended quiescent state! no locks held by swapper/1/0. dump_stack+0x8d/0xbb stack backtrace: ? default_idle+0xa/0x20 trace_hardirqs_on+0xed/0xf0 default_idle+0xa/0x20 default_idle_call+0x4f/0x1e0 do_idle+0x1ef/0x2c0 cpu_startup_entry+0x19/0x20 start_kernel+0x578/0x59d secondary_startup_64+0xa4/0xb0 CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.9.1+ #72
=============================
WARNING: suspicious RCU usage
5.9.1+ #72 Not tainted
-----------------------------./include/trace/events/lock.h:37 suspicious rcu_dereference_check() usage!
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
RCU used illegally from extended quiescent state!
no locks held by swapper/0/0.
stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.1+ #72 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x8d/0xbb lock_acquire+0x346/0x3b0 ? __lock_acquire+0x2e7/0x1da0 _raw_spin_lock+0x2c/0x40 ? vprintk_emit+0x9f/0x410 vprintk_emit+0x9f/0x410 printk+0x52/0x6e lockdep_rcu_suspicious+0x1b/0xe0 ? default_idle+0xa/0x20 trace_hardirqs_on+0xed/0xf0 default_idle+0xa/0x20 default_idle_call+0x4f/0x1e0 do_idle+0x1ef/0x2c0 cpu_startup_entry+0x19/0x20 start_kernel+0x578/0x59d secondary_startup_64+0xa4/0xb0
=============================
WARNING: suspicious RCU usage
5.9.1+ #72 Not tainted
-----------------------------./include/trace/events/lock.h:63 suspicious rcu_dereference_check() usage!
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
RCU used illegally from extended quiescent state!
1 lock held by swapper/0/0:
#0: ffffffff97a80158 (logbuf_lock){-...}-{2:2}, at: vprintk_emit+0x9f/0x410
stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.1+ #72 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x8d/0xbb lock_release+0x25a/0x280 _raw_spin_unlock+0x17/0x30 vprintk_emit+0xdf/0x410 printk+0x52/0x6e lockdep_rcu_suspicious+0x1b/0xe0 ? default_idle+0xa/0x20 trace_hardirqs_on+0xed/0xf0 default_idle+0xa/0x20 default_idle_call+0x4f/0x1e0 do_idle+0x1ef/0x2c0 cpu_startup_entry+0x19/0x20 start_kernel+0x578/0x59d secondary_startup_64+0xa4/0xb0
=============================
WARNING: suspicious RCU usage
5.9.1+ #72 Not tainted
-----------------------------./include/trace/events/printk.h:33 suspicious rcu_dereference_check() usage!
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
RCU used illegally from extended quiescent state!
2 locks held by swapper/0/0:
#0: ffffffff97a801a0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x126/0x410 #1: ffffffff97a7fec0 (console_owner){....}-{0:0}, at: console_unlock+0x190/0x5d0
stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.1+ #72 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x8d/0xbb console_unlock+0x5ad/0x5d0 vprintk_emit+0x133/0x410 printk+0x52/0x6e lockdep_rcu_suspicious+0x1b/0xe0 ? default_idle+0xa/0x20 trace_hardirqs_on+0xed/0xf0 default_idle+0xa/0x20 default_idle_call+0x4f/0x1e0 do_idle+0x1ef/0x2c0 cpu_startup_entry+0x19/0x20 start_kernel+0x578/0x59d secondary_startup_64+0xa4/0xb0
=============================
WARNING: suspicious RCU usage
5.9.1+ #72 Not tainted
-----------------------------./include/linux/rcupdate.h:636 rcu_read_lock() used illegally while idle!
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
RCU used illegally from extended quiescent state!
4 locks held by swapper/0/0:
#0: ffffffff97a801a0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x126/0x410 #1: ffffffff97a7fec0 (console_owner){....}-{0:0}, at: console_unlock+0x190/0x5d0 #2: ffffffff97b7d598 (printing_lock){....}-{2:2}, at: vt_console_print+0x7d/0x3e0 #3: ffffffff97a82d80 (rcu_read_lock){....}-{1:2}, at: __atomic_notifier_call_chain+0x5/0x110
stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.1+ #72 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x8d/0xbb __atomic_notifier_call_chain+0xd7/0x110 vt_console_print+0x19e/0x3e0 console_unlock+0x417/0x5d0 vprintk_emit+0x133/0x410 printk+0x52/0x6e lockdep_rcu_suspicious+0x1b/0xe0 ? default_idle+0xa/0x20 trace_hardirqs_on+0xed/0xf0 default_idle+0xa/0x20 default_idle_call+0x4f/0x1e0 do_idle+0x1ef/0x2c0 cpu_startup_entry+0x19/0x20 start_kernel+0x578/0x59d secondary_startup_64+0xa4/0xb0
=============================
WARNING: suspicious RCU usage
5.9.1+ #72 Not tainted
-----------------------------./include/linux/rcupdate.h:685 rcu_read_unlock() used illegally while idle!
other info that might help us debug this:
rcu_scheduler_active = 1, debug_locks = 1
RCU used illegally from extended quiescent state!
4 locks held by swapper/0/0:
#0: ffffffff97a801a0 (console_lock){+.+.}-{0:0}, at: vprintk_emit+0x126/0x410 #1: ffffffff97a7fec0 (console_owner){....}-{0:0}, at: console_unlock+0x190/0x5d0 #2: ffffffff97b7d598 (printing_lock){....}-{2:2}, at: vt_console_print+0x7d/0x3e0 #3: ffffffff97a82d80 (rcu_read_lock){....}-{1:2}, at: __atomic_notifier_call_chain+0x5/0x110
stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.9.1+ #72 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x8d/0xbb __atomic_notifier_call_chain+0x101/0x110 vt_console_print+0x19e/0x3e0 console_unlock+0x417/0x5d0 vprintk_emit+0x133/0x410 printk+0x52/0x6e lockdep_rcu_suspicious+0x1b/0xe0 ? default_idle+0xa/0x20 trace_hardirqs_on+0xed/0xf0 default_idle+0xa/0x20 default_idle_call+0x4f/0x1e0 do_idle+0x1ef/0x2c0 cpu_startup_entry+0x19/0x20 start_kernel+0x578/0x59d secondary_startup_64+0xa4/0xb0 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x8d/0xbb ? rcu_idle_exit+0x32/0x40 trace_hardirqs_off+0xe3/0xf0 rcu_idle_exit+0x32/0x40 default_idle_call+0x54/0x1e0 do_idle+0x1ef/0x2c0 ? lockdep_hardirqs_on_prepare+0xec/0x180 cpu_startup_entry+0x19/0x20 start_secondary+0x11c/0x160 secondary_startup_64+0xa4/0xb0
Thanks,
Mathieu
Quoted 13 lines
-- Steve
Quoted 9 lines
rcuidle API. That would effectively remove the calls to rcu_irq_{enter,exit}_irqson
from the tracepoint code.This is however beyond the scope of the proposed patch set.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
From 7a16029eb56921a42e4c16323e0dee128d724c1e Mon Sep 17 00:00:00 2001
From: Mathieu Desnoyers <[email protected]>
Date: Tue, 27 Oct 2020 09:28:46 -0400
Subject: [WIP PATCH 1/1] wip: remove rcuidle tracepoint API
--- arch/arm/kernel/smp.c | 6 +++--- arch/arm/mach-omap2/powerdomain.c | 10 +++++----- arch/arm64/kernel/smp.c | 4 ++-- drivers/base/power/runtime.c | 24 ++++++++++++------------ drivers/clk/clk.c | 8 ++++---- drivers/soc/qcom/rpmh-rsc.c | 2 +- include/linux/tracepoint.h | 26 ++------------------------ include/trace/events/preemptirq.h | 8 -------- kernel/printk/printk.c | 2 +- kernel/trace/trace_preemptirq.c | 18 ++++++------------ scripts/tags.sh | 2 -- 11 files changed, 36 insertions(+), 74 deletions(-)
arch/arm/kernel/smp.c +3 -3
@@ -521,7 +521,7 @@ static const char *ipi_types[NR_IPI] __tracepoint_string = {static void smp_cross_call (const struct cpumask *target,unsigned int ipinr) { -trace_ipi_raise_rcuidle (target, ipi_types[ipinr]); +trace_ipi_raise (target, ipi_types[ipinr]);__smp_cross_call (target, ipinr); }
@@ -633,7 +633,7 @@ void handle_IPI(int ipinr, struct pt_regs *regs)struct pt_regs *old_regs =set_irq_regs (regs);if ((unsigned )ipinr <NR_IPI ) { -trace_ipi_entry_rcuidle (ipi_types[ipinr]); +trace_ipi_entry (ipi_types[ipinr]);__inc_irq_stat (cpu, ipi_irqs[ipinr]); }
@@ -694,7 +694,7 @@ void handle_IPI(int ipinr, struct pt_regs *regs) }if ((unsigned )ipinr <NR_IPI ) -trace_ipi_exit_rcuidle (ipi_types[ipinr]); +trace_ipi_exit (ipi_types[ipinr]);set_irq_regs (old_regs); }
arch/arm/mach-omap2/powerdomain.c +5 -5
@@ -187,9 +187,9 @@ static int _pwrdm_state_switch(struct powerdomain *pwrdm, int flag) trace_state = (PWRDM_TRACE_STATES_FLAG | ((next &OMAP_POWERSTATE_MASK ) <<8 ) | ((prev &OMAP_POWERSTATE_MASK ) <<0 )); -trace_power_domain_target_rcuidle (pwrdm->name, - trace_state, -raw_smp_processor_id ()); +trace_power_domain_target (pwrdm->name, + trace_state, +raw_smp_processor_id ()); }break ;default :
@@ -541,8 +541,8 @@ int pwrdm_set_next_pwrst(struct powerdomain *pwrdm, u8 pwrst)if (arch_pwrdm && arch_pwrdm->pwrdm_set_next_pwrst) {/* Trace the pwrdm desired target state */ -trace_power_domain_target_rcuidle (pwrdm->name, pwrst, -raw_smp_processor_id ()); +trace_power_domain_target (pwrdm->name, pwrst, +raw_smp_processor_id ());/* Program the pwrdm desired target state */ ret = arch_pwrdm->pwrdm_set_next_pwrst (pwrdm, pwrst); }
arch/arm64/kernel/smp.c +2 -2
@@ -896,7 +896,7 @@ void handle_IPI(int ipinr, struct pt_regs *regs)struct pt_regs *old_regs =set_irq_regs (regs);if ((unsigned )ipinr <NR_IPI ) { -trace_ipi_entry_rcuidle (ipi_types[ipinr]); +trace_ipi_entry (ipi_types[ipinr]);__inc_irq_stat (cpu, ipi_irqs[ipinr]); }
@@ -956,7 +956,7 @@ void handle_IPI(int ipinr, struct pt_regs *regs) }if ((unsigned )ipinr <NR_IPI ) -trace_ipi_exit_rcuidle (ipi_types[ipinr]); +trace_ipi_exit (ipi_types[ipinr]);set_irq_regs (old_regs); }
drivers/base/power/runtime.c +12 -12
@@ -401,7 +401,7 @@ static int rpm_idle(struct device *dev, int rpmflags)int (*callback)(struct device *);int retval; -trace_rpm_idle_rcuidle (dev, rpmflags); +trace_rpm_idle (dev, rpmflags); retval =rpm_check_suspend_allowed (dev);if (retval <0 ) ;/* Conditions are wrong. */
@@ -437,7 +437,7 @@ static int rpm_idle(struct device *dev, int rpmflags) dev->power.request_pending = true;queue_work (pm_wq, &dev->power.work); } -trace_rpm_return_int_rcuidle (dev, _THIS_IP_,0 ); +trace_rpm_return_int (dev, _THIS_IP_,0 );return 0 ; }
@@ -452,7 +452,7 @@ static int rpm_idle(struct device *dev, int rpmflags)wake_up_all (&dev->power.wait_queue); out: -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval ? retval :rpm_suspend (dev, rpmflags |RPM_AUTO ); }
@@ -519,7 +519,7 @@ static int rpm_suspend(struct device *dev, int rpmflags)struct device *parent =NULL ;int retval; -trace_rpm_suspend_rcuidle (dev, rpmflags); +trace_rpm_suspend (dev, rpmflags); repeat: retval =rpm_check_suspend_allowed (dev);
@@ -657,7 +657,7 @@ static int rpm_suspend(struct device *dev, int rpmflags) } out: -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval;
@@ -709,7 +709,7 @@ static int rpm_resume(struct device *dev, int rpmflags)struct device *parent =NULL ;int retval =0 ; -trace_rpm_resume_rcuidle (dev, rpmflags); +trace_rpm_resume (dev, rpmflags); repeat:if (dev->power.runtime_error)
@@ -872,7 +872,7 @@ static int rpm_resume(struct device *dev, int rpmflags)spin_lock_irq (&dev->power.lock); } -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval; }
@@ -1005,7 +1005,7 @@ int __pm_runtime_idle(struct device *dev, int rpmflags)if (rpmflags &RPM_GET_PUT ) {if (!atomic_dec_and_test (&dev->power.usage_count)) { -trace_rpm_usage_rcuidle (dev, rpmflags); +trace_rpm_usage (dev, rpmflags);return 0 ; } }
@@ -1039,7 +1039,7 @@ int __pm_runtime_suspend(struct device *dev, int rpmflags)if (rpmflags &RPM_GET_PUT ) {if (!atomic_dec_and_test (&dev->power.usage_count)) { -trace_rpm_usage_rcuidle (dev, rpmflags); +trace_rpm_usage (dev, rpmflags);return 0 ; } }
@@ -1122,7 +1122,7 @@ int pm_runtime_get_if_active(struct device *dev, bool ign_usage_count) }else { retval =atomic_inc_not_zero (&dev->power.usage_count); } -trace_rpm_usage_rcuidle (dev,0 ); +trace_rpm_usage (dev,0 );spin_unlock_irqrestore (&dev->power.lock, flags);return retval;
@@ -1458,7 +1458,7 @@ void pm_runtime_allow(struct device *dev)if (atomic_dec_and_test (&dev->power.usage_count))rpm_idle (dev,RPM_AUTO |RPM_ASYNC );else -trace_rpm_usage_rcuidle (dev,RPM_AUTO |RPM_ASYNC ); +trace_rpm_usage (dev,RPM_AUTO |RPM_ASYNC ); out:spin_unlock_irq (&dev->power.lock);
@@ -1527,7 +1527,7 @@ static void update_autosuspend(struct device *dev, int old_delay, int old_use)atomic_inc (&dev->power.usage_count);rpm_resume (dev,0 ); }else { -trace_rpm_usage_rcuidle (dev,0 ); +trace_rpm_usage (dev,0 ); } }
drivers/clk/clk.c +4 -4
@@ -959,12 +959,12 @@ static void clk_core_disable(struct clk_core *core)if (--core->enable_count >0 )return ; -trace_clk_disable_rcuidle (core); +trace_clk_disable (core);if (core->ops->disable) core->ops->disable (core->hw); -trace_clk_disable_complete_rcuidle (core); +trace_clk_disable_complete (core);clk_core_disable (core->parent); }
@@ -1018,12 +1018,12 @@ static int clk_core_enable(struct clk_core *core)if (ret)return ret; -trace_clk_enable_rcuidle (core); +trace_clk_enable (core);if (core->ops->enable) ret = core->ops->enable (core->hw); -trace_clk_enable_complete_rcuidle (core); +trace_clk_enable_complete (core);if (ret) {clk_core_disable (core->parent);
drivers/soc/qcom/rpmh-rsc.c +1 -1
@@ -495,7 +495,7 @@ static void __tcs_buffer_write(struct rsc_drv *drv, int tcs_id, int cmd_id,write_tcs_cmd (drv,RSC_DRV_CMD_MSGID , tcs_id, j, msgid);write_tcs_cmd (drv,RSC_DRV_CMD_ADDR , tcs_id, j, cmd->addr);write_tcs_cmd (drv,RSC_DRV_CMD_DATA , tcs_id, j, cmd->data); -trace_rpmh_send_msg_rcuidle (drv, tcs_id, j, msgid, cmd); +trace_rpmh_send_msg (drv, tcs_id, j, msgid, cmd); }write_tcs_reg (drv,RSC_DRV_CMD_WAIT_FOR_CMPL , tcs_id, cmd_complete);
include/linux/tracepoint.h +2 -24
@@ -160,7 +160,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* has a "void" prototype, then it is invalid to declare a function * as "(void *, void)". */ -#define __DO_TRACE (tp, proto, args, cond, rcuidle, tp_flags) \ +#define __DO_TRACE (tp, proto, args, cond, tp_flags) \ do { \ struct tracepoint_func *it_func_ptr; \ void *it_func; \
@@ -172,8 +172,6 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)return ; \ \might_sleep_if (maysleep); \ -if (rcuidle) \ -rcu_irq_enter_irqson (); \if (maysleep) \rcu_read_lock_trace (); \else \
@@ -193,24 +191,8 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)rcu_read_unlock_trace (); \ else \preempt_enable_notrace (); \ -if (rcuidle) \ -rcu_irq_exit_irqson (); \ }while (0 ) -#ifndef MODULE -#define __DECLARE_TRACE_RCU (name, proto, args, cond, data_proto, data_args, tp_flags) \ - static inline void trace_##name##_rcuidle(proto) \ - { \ - if (static_key_false(&__tracepoint_##name.key)) \ - __DO_TRACE(&__tracepoint_##name, \ - TP_PROTO(data_proto), \ - TP_ARGS(data_args), \ - TP_CONDITION(cond), 1, tp_flags); \ - } -#else -#define __DECLARE_TRACE_RCU (name, proto, args, cond, data_proto, data_args, tp_flags) -#endif -/* * Make sure the alignment of the structure in the __tracepoints section will * not add unwanted padding between the beginning of the section and the
@@ -231,15 +213,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) __DO_TRACE(&__tracepoint_##name, \ TP_PROTO(data_proto), \ TP_ARGS(data_args), \ - TP_CONDITION(cond), 0, tp_flags); \ + TP_CONDITION(cond), tp_flags); \ if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ rcu_read_lock_sched_notrace(); \ rcu_dereference_sched(__tracepoint_##name.funcs);\ rcu_read_unlock_sched_notrace(); \ } \ } \ - __DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args), \ - PARAMS(cond), PARAMS(data_proto), PARAMS(data_args), tp_flags) \ static inline int \ register_trace_##name(void (*probe)(data_proto), void *data) \ { \
@@ -310,8 +290,6 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)#define __DECLARE_TRACE (name, proto, args, cond, data_proto, data_args, tp_flags) \ static inline void trace_##name(proto) \ { } \ - static inline void trace_##name##_rcuidle(proto) \ - { } \ static inline int \ register_trace_##name(void (*probe)(data_proto), \ void *data) \
include/trace/events/preemptirq.h +0 -8
@@ -43,8 +43,6 @@ DEFINE_EVENT(preemptirq_template, irq_enable,#else #define trace_irq_enable (...)#define trace_irq_disable (...) -#define trace_irq_enable_rcuidle (...) -#define trace_irq_disable_rcuidle (...)#endif #ifdef CONFIG_TRACE_PREEMPT_TOGGLE
@@ -58,8 +56,6 @@ DEFINE_EVENT(preemptirq_template, preempt_enable,#else #define trace_preempt_enable (...)#define trace_preempt_disable (...) -#define trace_preempt_enable_rcuidle (...) -#define trace_preempt_disable_rcuidle (...)#endif #endif /* _TRACE_PREEMPTIRQ_H */
@@ -69,10 +65,6 @@ DEFINE_EVENT(preemptirq_template, preempt_enable,#else /* !CONFIG_PREEMPTIRQ_TRACEPOINTS */ #define trace_irq_enable (...)#define trace_irq_disable (...) -#define trace_irq_enable_rcuidle (...) -#define trace_irq_disable_rcuidle (...)#define trace_preempt_enable (...)#define trace_preempt_disable (...) -#define trace_preempt_enable_rcuidle (...) -#define trace_preempt_disable_rcuidle (...)#endif
kernel/printk/printk.c +1 -1
@@ -1806,7 +1806,7 @@ static void call_console_drivers(const char *ext_text, size_t ext_len, {struct console *con; -trace_console_rcuidle (text, len); +trace_console (text, len);for_each_console (con) {if (exclusive_console && con != exclusive_console)
kernel/trace/trace_preemptirq.c +6 -12
@@ -40,8 +40,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_on_prepare);void trace_hardirqs_on (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace_irq_enable (CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -78,8 +77,7 @@ void trace_hardirqs_off(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace_irq_disable (CALLER_ADDR0 ,CALLER_ADDR1 ); } }EXPORT_SYMBOL (trace_hardirqs_off);
@@ -88,8 +86,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_off); __visible void trace_hardirqs_on_caller(unsigned long caller_addr) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 , caller_addr); +trace_irq_enable (CALLER_ADDR0 , caller_addr);tracer_hardirqs_on (CALLER_ADDR0 , caller_addr);this_cpu_write (tracing_irq_cpu,0 ); }
@@ -107,8 +104,7 @@ __visible void trace_hardirqs_off_caller(unsigned long caller_addr)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 , caller_addr); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 , caller_addr); +trace_irq_disable (CALLER_ADDR0 , caller_addr); } }EXPORT_SYMBOL (trace_hardirqs_off_caller);
@@ -119,15 +115,13 @@ NOKPROBE_SYMBOL(trace_hardirqs_off_caller);void trace_preempt_on (unsigned long a0,unsigned long a1) { -if (!in_nmi ()) -trace_preempt_enable_rcuidle (a0, a1); +trace_preempt_enable (a0, a1);tracer_preempt_on (a0, a1); }void trace_preempt_off (unsigned long a0,unsigned long a1) { -if (!in_nmi ()) -trace_preempt_disable_rcuidle (a0, a1); +trace_preempt_disable (a0, a1);tracer_preempt_off (a0, a1); }#endif
scripts/tags.sh +0 -2
@@ -147,9 +147,7 @@ regex_c=( '/^BPF_CALL_[0-9](\([[:alnum:]_]*\).*/\1/' '/^COMPAT_SYSCALL_DEFINE[0-9](\([[:alnum:]_]*\).*/compat_sys_\1/' '/^TRACE_EVENT(\([[:alnum:]_]*\).*/trace_\1/' - '/^TRACE_EVENT(\([[:alnum:]_]*\).*/trace_\1_rcuidle/' '/^DEFINE_EVENT([^,)]*, *\([[:alnum:]_]*\).*/trace_\1/' - '/^DEFINE_EVENT([^,)]*, *\([[:alnum:]_]*\).*/trace_\1_rcuidle/' '/^DEFINE_INSN_CACHE_OPS(\([[:alnum:]_]*\).*/get_\1_slot/' '/^DEFINE_INSN_CACHE_OPS(\([[:alnum:]_]*\).*/free_\1_slot/' '/^PAGEFLAG(\([[:alnum:]_]*\).*/Page\1/'
Signature · 1 line
2.17.1
Re: [RFC PATCH 6/6] tracing: use sched-RCU instead of SRCU for rcuidle tracepoints
In reply to Mathieu DesnoyersQuoted 42 lines · Mathieu Desnoyers
Quoted 17 lines · ----- On Oct 26, 2020, at 4:20 AM, Peter Zijlstra [email protected] wrote:
Quoted 16 lines · Joel Fernandes
Quoted 8 lines · Michael Jeanson
From: Mathieu Desnoyers <[email protected]>
Considering that tracer callbacks expect RCU to be watching (for instance, perf uses
rcu_read_lock), we need rcuidle tracepoints to issue rcu_irq_{enter,exit}_irqson around calls to the callbacks. So there is no point in using SRCU anymore given that rcuidle tracepoints need to ensure RCU is watching. Therefore, simply use sched-RCU like normal tracepoints for rcuidle tracepoints.High level question:
IIRC, doing this increases overhead for general tracing that does not use perf, for 'rcuidle' tracepoints such as the preempt/irq enable/disable tracepoints. I remember adding SRCU because of this reason.
Quoted 6 lines
Quoted 3 lines
Can the 'rcuidle' information not be pushed down further, such that perf does it because it requires RCU to be watching, so that it does not effect, say, trace events?
There's very few trace_.*_rcuidle() users left. We should eradicate them and remove the option. It's bugs to begin with.
I agree with Peter. Removing the trace_.*_rcuidle weirdness from the tracepoint API and fixing all callers to ensure they trace from a context where RCU is watching would simplify instrumentation of the Linux kernel, thus making it harder for subtle bugs to hide and be unearthed only when tracing is enabled. This is AFAIU the general approach Thomas Gleixner has been aiming for recently, and I think it is a good thing.
So if we consider this our target, and that the current state of things is that we need to have RCU watching around callback invocation, then removing the dependency on SRCU seems like an overall simplification which does not regress feature-wise nor speed-wise compared with what we have upstream today. The next steps would then be to audit all rcuidle tracepoints and make sure the context where they are placed has RCU watching already, so we can remove the tracepoint rcuidle API. That would effectively remove the calls to rcu_irq_{enter,exit}_irqson from the tracepoint code.
Quoted 1 line
This is however beyond the scope of the proposed patch set.
You are right, it doesn't regress speedwise - I got confused since the code was modified to call rcu_enter_irqson() even for the rcuidle case (which I had avoided when I added SRCU). So in current code, SRCU is kind of pointless. I think keep the patch in the series.
thanks,
- Joel
Quoted 10 lines · Michael Jeanson
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow specific tracer probes to take page faults. Adapt ftrace, perf, and ebpf to allow being called from sleepable context, and convert the system call enter/exit instrumentation to sleepable tracepoints.
Will this not be a problem for analyse of the trace? It get two relevant times, one it when it is called and one when it returns.
It makes things harder to correlate in what order things happen.
And handling of tracing of contexts that already are not preamptable?
Eg the same tracepoint are used in different places and contexts.
Quoted 45 lines
This series only implements the tracepoint infrastructure required to allow tracers to handle page faults. Modifying each tracer to handle those page faults would be a next step after we all agree on this piece of instrumentation infrastructure.
This patchset is base on v5.9.1.
Cc Steven Rostedt (VMware) <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Alexei Starovoitov <[email protected]>Cc Yonghong Song <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Ingo Molnar <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Mark Rutland <[email protected]>Cc Alexander Shishkin <[email protected]>Cc Jiri Olsa <[email protected]>Cc Namhyung Kim <[email protected]>Cc Joel Fernandes (Google) <[email protected]>Mathieu Desnoyers (1):
tracing: use sched-RCU instead of SRCU for rcuidle tracepointsMichael Jeanson (5):
tracing: introduce sleepable tracepoints
tracing: ftrace: add support for sleepable tracepoints
tracing: bpf-trace: add support for sleepable tracepoints
tracing: perf: add support for sleepable tracepoints
tracing: convert sys_enter/exit to sleepable tracepointsinclude/linux/tracepoint-defs.h | 11 ++++ include/linux/tracepoint.h | 104 +++++++++++++++++++++----------- include/trace/bpf_probe.h | 23 ++++++- include/trace/define_trace.h | 7 +++ include/trace/events/syscalls.h | 4 +- include/trace/perf.h | 26 ++++++-- include/trace/trace_events.h | 79 ++++++++++++++++++++++-- init/Kconfig | 1 + kernel/trace/bpf_trace.c | 5 +- kernel/trace/trace_events.c | 15 ++++- kernel/trace/trace_syscalls.c | 68 +++++++++++++-------- kernel/tracepoint.c | 104 +++++++++++++++++++++++++------- 12 files changed, 351 insertions(+), 96 deletions(-)
Mathieu Desnoyers
reviewerQuoted 7 lines · ----- On Oct 26, 2020, at 8:05 AM, peter enderborg [email protected]…
Quoted 6 lines · Michael Jeanson
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Quoted 7 lines
Quoted 4 lines
Extend the tracepoint and trace event APIs to allow specific tracer probes to take page faults. Adapt ftrace, perf, and ebpf to allow being called from sleepable context, and convert the system call enter/exit instrumentation to sleepable tracepoints.
Will this not be a problem for analyse of the trace? It get two relevant times, one it when it is called and one when it returns.
It will depend on what the tracer chooses to do. If we call the side-effect of what is being traced a "transaction" (e.g. actually opening a file descriptor and adding it to a process'file descriptor table as the result of an open(2) system call), we have to consider that already today the timestamp which we get is either slightly before or after the actual side-effect of the transaction in the kernel. That is true even without being preemptable.
Sometimes it's not relevant to have a tracepoint before and after the transaction, e.g. when all we care about is to know that the transaction has successfully happened or not.
In the case of system calls, we have sys_enter and sys_exit to mark the beginning and end of the "transaction". Whatever side-effects are done by the system call happens in between.
I think the question here is whether it is relevant to know whether page faults triggered by accessing system call input parameters need to happen after we trace a "system call entry" event. If the tracers care, then it would be up to them to first trace that "system call entry" event, and have a separate event for the argument payload. But there are other ways to identify whether page faults happen within the system call or from user-space, for instance by using the instruction pointer associated with the page fault. So when observing page faults happening before sys enter, but associated with a kernel instruction pointer, a trace analysis tool could theoretically figure out who is to blame for that page fault, *if* it cares.
Quoted 2 lines
It makes things harder to correlate in what order things happen.
The alternative is to have partial payloads like LTTng does today for system call arguments. If reading a string from userspace (e.g. open(2) file name) requires to take a page fault, LTTng truncates the string. This is pretty bad for automated analysis as well.
Quoted 2 lines
And handling of tracing of contexts that already are not preamptable?
The sleepable tracepoints are only meant to be used in contexts which can sleep. For tracepoints placed in non-preemptible contexts, those should never take a page fault to begin with.
Quoted 2 lines
Eg the same tracepoint are used in different places and contexts.
As far as considering that a given tracepoint "name" could be registered to by both a sleepable and non-sleepable tracer probes, I would like to see an actual use-case for this. I don't have any.
I can envision that some tracer code will want to be allowed to work in both sleepable and non-sleepable context, e.g. take page faults in sleepable context (and called from a sleepable tracepoint), but have a truncation behavior when called from non-sleepable context. This can actually be done by looking at the new "TRACEPOINT_MAYSLEEP" tp flag. Passing that tp_flags to code shared between sleepable and non-sleepable probes would allow the callee to know whether it can take a page fault or not.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com