Faultable Tracepoints
Subsystems (4)
Mathieu Desnoyers
reviewerv4 cover Faultable Tracepoints
Wire up the system call tracepoints with Tasks Trace RCU to allow the ftrace, perf, and eBPF tracers to handle page faults.
This series does the initial wire-up allowing tracers to handle page faults, but leaves out the actual handling of said page faults as future work.
I have tested this against a feature branch of lttng-modules which implements handling of page faults for the filename argument of the openat(2) system call.
This series is based on kernel v6.6.2.
Thanks,
Mathieu
Mathieu Desnoyers (5):
tracing: Introduce faultable tracepoints (v4) tracing/ftrace: Add support for faultable tracepoints tracing/bpf-trace: add support for faultable tracepoints tracing/perf: add support for faultable tracepoints
tracing: convert sys_enter/exit to faultable tracepoints
include/linux/tracepoint-defs.h | 14 +++++ include/linux/tracepoint.h | 88 ++++++++++++++++++++++--------- include/trace/bpf_probe.h | 21 ++++++-- include/trace/define_trace.h | 7 +++ include/trace/events/syscalls.h | 4 +- include/trace/perf.h | 27 ++++++++-- include/trace/trace_events.h | 73 ++++++++++++++++++++++++-- init/Kconfig | 1 + kernel/trace/bpf_trace.c | 10 +++- kernel/trace/trace_events.c | 26 +++++++--- kernel/trace/trace_fprobe.c | 5 +- kernel/trace/trace_syscalls.c | 92 +++++++++++++++++++++++---------- kernel/tracepoint.c | 58 +++++++++++---------- 13 files changed, 325 insertions(+), 101 deletions(-)
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerPATCH v4 1/5 tracing: Introduce faultable tracepoints
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the TRACEPOINT_MAY_FAULT flag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
---
Changes since v1:
- Cleanup __DO_TRACE() implementation.
- Rename "sleepable tracepoints" to "faultable tracepoints", MAYSLEEP to MAYFAULT, and use might_fault() rather than might_sleep(), to properly convey that the tracepoints are meant to be able to take a page fault, which requires to be able to sleep *and* to hold the mmap_sem.
Changes since v2:
- Rename MAYFAULT to MAY_FAULT.
- Rebased on 6.5.5.
- Introduce MAY_EXIST tracepoint flag.
Changes since v3:
- Rebased on 6.6.2.
--- include/linux/tracepoint-defs.h | 14 ++++++ include/linux/tracepoint.h | 88 +++++++++++++++++++++++---------- include/trace/define_trace.h | 7 +++ include/trace/trace_events.h | 6 +++ init/Kconfig | 1 + kernel/trace/bpf_trace.c | 5 +- kernel/trace/trace_fprobe.c | 5 +- kernel/tracepoint.c | 58 ++++++++++++---------- 8 files changed, 129 insertions(+), 55 deletions(-)
include/linux/tracepoint-defs.h +14 -0
@@ -29,6 +29,19 @@ struct tracepoint_func {int prio; }; +/** +* enum tracepoint_flags - Tracepoint flags +* @TRACEPOINT_MAY_EXIST: Don't return an error if the tracepoint does not +* exist upon registration. +* @TRACEPOINT_MAY_FAULT: The tracepoint probe callback will be called with +* preemption enabled, and is allowed to take page +* faults. +*/ +enum tracepoint_flags { +TRACEPOINT_MAY_EXIST = (1 <<0 ), +TRACEPOINT_MAY_FAULT = (1 <<1 ), +}; +struct tracepoint {const char *name;/* Tracepoint name */ struct static_key key;
@@ -39,6 +52,7 @@ struct tracepoint {int (*regfunc)(void );void (*unregfunc)(void );struct tracepoint_func __rcu *funcs; +unsigned int flags; };#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS
include/linux/tracepoint.h +63 -25
214 lines not on this page · raw message
include/trace/define_trace.h +7 -0
@@ -41,6 +41,12 @@ assign, print, reg, unreg) \ DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args)) +#undef TRACE_EVENT_FN_MAY_FAULT +#define TRACE_EVENT_FN_MAY_FAULT (name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + DEFINE_TRACE_FN_FLAGS(name, reg, unreg, PARAMS(proto), \ + PARAMS(args), TRACEPOINT_MAY_FAULT) +#undef TRACE_EVENT_FN_COND#define TRACE_EVENT_FN_COND (name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \
@@ -106,6 +112,7 @@#undef TRACE_EVENT#undef TRACE_EVENT_FN +#undef TRACE_EVENT_FN_MAY_FAULT#undef TRACE_EVENT_FN_COND#undef TRACE_EVENT_CONDITION#undef TRACE_EVENT_NOP
include/trace/trace_events.h +6 -0
@@ -77,6 +77,12 @@TRACE_EVENT (name,PARAMS (proto),PARAMS (args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_MAY_FAULT +#define TRACE_EVENT_FN_MAY_FAULT (name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_COND#define TRACE_EVENT_FN_COND (name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \
init/Kconfig +1 -0
@@ -1919,6 +1919,7 @@ config BINDGEN_VERSION_TEXT # config TRACEPOINTS bool + select TASKS_TRACE_RCU source "kernel/Kconfig.kexec"
kernel/trace/bpf_trace.c +3 -2
@@ -2368,8 +2368,9 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *if (prog->aux->max_tp_access > btp->writable_size)return -EINVAL ; -return tracepoint_probe_register_may_exist (tp, (void *)btp->bpf_func, - prog); +return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, + prog,TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_EXIST ); }int bpf_probe_register (struct bpf_raw_event_map *btp,struct bpf_prog *prog)
kernel/trace/trace_fprobe.c +3 -2
@@ -687,8 +687,9 @@ static int __register_trace_fprobe(struct trace_fprobe *tf)* At first, put __probestub_##TP function on the tracepoint * and put a fprobe on the stub function. */ - ret =tracepoint_probe_register_prio_may_exist (tpoint, - tpoint->probestub,NULL ,0 ); + ret =tracepoint_probe_register_prio_flags (tpoint, + tpoint->probestub,NULL ,0 , +TRACEPOINT_MAY_EXIST );if (ret <0 )return ret;return register_fprobe_ips (&tf->fp, &ip,1 );
kernel/tracepoint.c +32 -26
127 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 16 lines · Mathieu Desnoyers
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the
TRACEPOINT_MAY_FAULTflag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.
Quoted 3 lines
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
What is trace-trace rcu and why is it needed here? What's wrong with SRCU ?
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Mathieu Desnoyers
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the
TRACEPOINT_MAY_FAULTflag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.
Quoted 6 lines
Quoted 3 lines
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
What is trace-trace rcu and why is it needed here? What's wrong with SRCU ?
Tasks Trace RCU avoids SRCU's full barriers and the array accesses in the read-side primitives. This can be important when tracing low-overhead components of fast paths.
Thanx, Paul
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 22 lines · Paul E. McKenney
Quoted 21 lines · Peter Zijlstra
Quoted 19 lines · Mathieu Desnoyers
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the
TRACEPOINT_MAY_FAULTflag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
Quoted 6 lines
Quoted 2 lines
What is trace-trace rcu and why is it needed here? What's wrong with SRCU ?
Tasks Trace RCU avoids SRCU's full barriers and the array accesses in the read-side primitives. This can be important when tracing low-overhead components of fast paths.
So why wasn't SRCU improved? That is, the above doesn't much explain.
What is the trade-off made to justify adding yet another RCU flavour?
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Paul E. McKenney
Quoted 23 lines · Peter Zijlstra
Quoted 16 lines · Mathieu Desnoyers
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the
TRACEPOINT_MAY_FAULTflag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.Quoted 3 lines
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
What is trace-trace rcu and why is it needed here? What's wrong with SRCU ?
Quoted 7 lines
Quoted 3 lines
Tasks Trace RCU avoids SRCU's full barriers and the array accesses in the read-side primitives. This can be important when tracing low-overhead components of fast paths.
So why wasn't SRCU improved? That is, the above doesn't much explain.
What is the trade-off made to justify adding yet another RCU flavour?
We didn't think you would be all that happy about having each and every context switch iterating through many tens or even hundreds of srcu_struct structures. For that matter, we didn't think that anyone else would be all that happy either. Us included.
Thanx, Paul
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 31 lines · Paul E. McKenney
Quoted 30 lines · Peter Zijlstra
Quoted 28 lines · Paul E. McKenney
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Mathieu Desnoyers
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the
TRACEPOINT_MAY_FAULTflag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.Quoted 6 lines
Quoted 3 lines
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
What is trace-trace rcu and why is it needed here? What's wrong with SRCU ?
Tasks Trace RCU avoids SRCU's full barriers and the array accesses in the read-side primitives. This can be important when tracing low-overhead components of fast paths.
Quoted 8 lines
Quoted 3 lines
So why wasn't SRCU improved? That is, the above doesn't much explain.
What is the trade-off made to justify adding yet another RCU flavour?
We didn't think you would be all that happy about having each and every context switch iterating through many tens or even hundreds of
srcu_structstructures. For that matter, we didn't think that anyone else would be all that happy either. Us included.
So again, what is task-trace RCU ? How does it differ from say preemptible rcu, which AFAICT could be used here too, no?
Mathieu Desnoyers
reviewerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Paul E. McKenney
Quoted 33 lines · Peter Zijlstra
Quoted 22 lines · Paul E. McKenney
Quoted 21 lines · Peter Zijlstra
Quoted 19 lines · Mathieu Desnoyers
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the
TRACEPOINT_MAY_FAULTflag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
Quoted 6 lines
Quoted 2 lines
What is trace-trace rcu and why is it needed here? What's wrong with SRCU ?
Tasks Trace RCU avoids SRCU's full barriers and the array accesses in the read-side primitives. This can be important when tracing low-overhead components of fast paths.
So why wasn't SRCU improved? That is, the above doesn't much explain.
What is the trade-off made to justify adding yet another RCU flavour?
Quoted 7 lines
Quoted 4 lines
We didn't think you would be all that happy about having each and every context switch iterating through many tens or even hundreds of
srcu_structstructures. For that matter, we didn't think that anyone else would be all that happy either. Us included.So again, what is task-trace RCU ? How does it differ from say preemptible rcu, which AFAICT could be used here too, no?
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slow synchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.
Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
- AFAIU the mmap_sem used within the page fault handler does not have priority inheritance.
Please let me know if I'm missing something.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 11 lines · Mathieu Desnoyers
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slowsynchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
Your counter points are confused, we simply don't build preemptible RCU unless PREEMPT=y, but that could surely be fixed and exposed as a separate flavour.
Quoted 1 line
- AFAIU the
mmap_semused within the page fault handler does not have priority inheritance.
What's that got to do with anything?
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
Mathieu Desnoyers
reviewerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 17 lines · Peter Zijlstra
Quoted 11 lines · Mathieu Desnoyers
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slowsynchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
Your counter points are confused, we simply don't build preemptible RCU unless PREEMPT=y, but that could surely be fixed and exposed as a separate flavour.
Quoted 6 lines
Quoted 1 line
- AFAIU the
mmap_semused within the page fault handler does not have priority inheritance.What's that got to do with anything?
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
In addition to taking the mmap_sem, the page fault handler need to block until its requested pages are faulted in, which may depend on disk I/O. Is it acceptable to wait for I/O while holding preemptible RCU read-side?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 20 lines · Mathieu Desnoyers
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Mathieu Desnoyers
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slowsynchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
Your counter points are confused, we simply don't build preemptible RCU unless PREEMPT=y, but that could surely be fixed and exposed as a separate flavour.
Quoted 1 line
- AFAIU the
mmap_semused within the page fault handler does not have priority inheritance.
Quoted 8 lines
Quoted 4 lines
What's that got to do with anything?
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
In addition to taking the
mmap_sem, the page fault handler need to block until its requested pages are faulted in, which may depend on disk I/O. Is it acceptable to wait for I/O while holding preemptible RCU read-side?
I don't know, preemptible rcu already needs to track task state anyway, it needs to ensure all tasks have passed through a safe spot etc.. vs regular RCU which only needs to ensure all CPUs have passed through start.
Why is this such a hard question?
Mathieu Desnoyers
reviewerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 30 lines · Peter Zijlstra
Quoted 20 lines · Mathieu Desnoyers
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Mathieu Desnoyers
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slowsynchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
Your counter points are confused, we simply don't build preemptible RCU unless PREEMPT=y, but that could surely be fixed and exposed as a separate flavour.
Quoted 1 line
- AFAIU the
mmap_semused within the page fault handler does not have priority inheritance.Quoted 8 lines
Quoted 4 lines
What's that got to do with anything?
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
In addition to taking the
mmap_sem, the page fault handler need to block until its requested pages are faulted in, which may depend on disk I/O. Is it acceptable to wait for I/O while holding preemptible RCU read-side?
Quoted 5 lines
I don't know, preemptible rcu already needs to track task state anyway, it needs to ensure all tasks have passed through a safe spot etc.. vs regular RCU which only needs to ensure all CPUs have passed through start.
Why is this such a hard question?
Personally what I am looking for is a clear documentation of preemptible rcu with respect to whether it is possible to block on I/O (take a page fault, call schedule() explicitly) from within a preemptible rcu critical section. I guess this is a hard question because there is no clear statement to that effect in the kernel documentation.
If it is allowed (which I doubt), then I wonder about the effect of those long readers on grace period delays. Things like expedited grace periods may suffer.
Based on Documentation/RCU/rcu.rst:
Preemptible variants of RCU (CONFIG_PREEMPT_RCU) get the same effect, but require that the readers manipulate CPU-local counters. These counters allow limited types of blocking within RCU read-side critical sections. SRCU also uses CPU-local counters, and permits general blocking within RCU read-side critical sections. These variants of RCU detect grace periods by sampling these counters.
Then we just have to find a definition of "limited types of blocking" vs "general blocking".
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 16 lines · Mathieu Desnoyers
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Mathieu Desnoyers
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Mathieu Desnoyers
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slowsynchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Your counter points are confused, we simply don't build preemptible RCU unless PREEMPT=y, but that could surely be fixed and exposed as a separate flavour.
Quoted 1 line
- AFAIU the
mmap_semused within the page fault handler does not have priority inheritance.What's that got to do with anything?
Preemptible RCU allows blocking/preemption only in those cases where priority inheritance applies. As Mathieu says below, this prevents indefinite postponement of a global grace period. Such postponement is especially problematic in kernels built with PREEMPT_RCU=y. For but one example, consider a situation where someone maps a file served by NFS. We can debate the wisdom of creating such a map, but having the kernel OOM would be a completely unacceptable "error message".
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
Task Trace RCU allows general blocking, which it tolerates by stringent restrictions on what exactly it is used for (tracing in cases where the memory to be included in the tracing might page fault). Preemptible RCU does not permit general blocking.
Tasks Trace RCU is a very specialized tool for a very specific use case.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
In addition to taking the
mmap_sem, the page fault handler need to block until its requested pages are faulted in, which may depend on disk I/O. Is it acceptable to wait for I/O while holding preemptible RCU read-side?I don't know, preemptible rcu already needs to track task state anyway, it needs to ensure all tasks have passed through a safe spot etc.. vs regular RCU which only needs to ensure all CPUs have passed through start.
Why is this such a hard question?
It is not a hard question. Nor is the answer, which is that preemptible RCU is not a good fit for this task for all the reasons that Mathieu has laid out.
Quoted 20 lines
Personally what I am looking for is a clear documentation of preemptible rcu with respect to whether it is possible to block on I/O (take a page fault, call
schedule()explicitly) from within a preemptible rcu critical section. I guess this is a hard question because there is no clear statement to that effect in the kernel documentation.If it is allowed (which I doubt), then I wonder about the effect of those long readers on grace period delays. Things like expedited grace periods may suffer.
Based on
Documentation/RCU/rcu.rst:Preemptible variants of RCU (
CONFIG_PREEMPT_RCU) get the same effect, but require that the readers manipulate CPU-local counters. These counters allow limited types of blocking within RCU read-side critical sections. SRCU also uses CPU-local counters, and permits general blocking within RCU read-side critical sections. These variants of RCU detect grace periods by sampling these counters.
Quoted 2 lines
Then we just have to find a definition of "limited types of blocking" vs "general blocking".
The key point is that you are not allowed to place any source code in a preemptible RCU reader that would not be legal in a non-preemptible RCU reader. The rationale again is that the cases in which preemptible RCU readers call schedule are cases to which priority boosting applies.
It is quite possible that the documentation needs upgrading. ;-)
Thanx, Paul
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
In reply to Peter ZijlstraQuoted 1 line · Peter Zijlstra
Why is this such a hard question?
Anyway, recapping from IRC:
preemptible, SRCU:
counter-array based, GP advances by increasing array index and waiting for previous index to drop to 0.
notably, a GP can pass while a task is preempted but not within a critical section.
SRCU has smp_mb() in the critical sections to improve GP.
tasks:
waits for every task to pass schedule()
ensures that any pieces of text rendered unreachable before, is actually unused after.
tasks-rude:
like tasks, but different? build to handle tracing while rcu-idle, even though that was already deemed bad?
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?
Can someone complete, please?
Mathieu Desnoyers
reviewerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
Why is this such a hard question?
Anyway, recapping from IRC:
Quoted 8 lines
preemptible, SRCU:
counter-array based, GP advances by increasing array index and waiting for previous index to drop to 0.notably, a GP can pass while a task is preempted but not within a critical section.
SRCU has
smp_mb()in the critical sections to improve GP.
Also:
preemptible only allows blocking when priority inheritance is guarantees, which excludes doing I/O, and thus page faults. Otherwise a long I/O could cause the system to OOM.
SRCU allows all kind of blocking, as long as the entire SRCU domain does not mind waiting for a while before readers complete.
Quoted 7 lines
tasks:
waits for every task to passschedule()ensures that any pieces of text rendered unreachable before, is actually unused after.
Quoted 7 lines
tasks-rude:
like tasks, but different? build to handle tracing while rcu-idle, even though that was already deemed bad?tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?
tasks-trace-rcu is meant to allow tasks to block/take a page fault within the read-side. It is specialized for tracing and has a single domain. It does not need the smp_mb on the read-side, which makes it lower-overhead than SRCU.
Thanks,
Mathieu
Quoted 6 lines
Can someone complete, please?
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Steven Rostedt
maintainerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 8 lines · On Tue, 21 Nov 2023 11:00:13 -0500 Mathieu Desnoyers…
Quoted 3 lines
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?tasks-trace-rcu is meant to allow tasks to block/take a page fault within the read-side. It is specialized for tracing and has a single domain. It does not need the
smp_mbon the read-side, which makes it lower-overhead than SRCU.
IOW, task-trace-rcu allows the call to schedule in its critical section, whereas task-rcu does not?
-- Steve
Mathieu Desnoyers
reviewerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Tue, 21 Nov 2023 11:00:13 -0500 Mathieu Desnoyers…
Quoted 3 lines
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?
Quoted 7 lines
Quoted 4 lines
tasks-trace-rcu is meant to allow tasks to block/take a page fault within the read-side. It is specialized for tracing and has a single domain. It does not need the
smp_mbon the read-side, which makes it lower-overhead than SRCU.IOW, task-trace-rcu allows the call to schedule in its critical section, whereas task-rcu does not?
Correct.
And unlike preemptible rcu, tasks-trace-rcu allows calls to schedule which do not provide priority inheritance guarantees (such as I/O triggered by page faults).
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 13 lines · Mathieu Desnoyers
Quoted 12 lines · Steven Rostedt
Quoted 8 lines · On Tue, 21 Nov 2023 11:00:13 -0500 Mathieu Desnoyers…
Quoted 3 lines
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?tasks-trace-rcu is meant to allow tasks to block/take a page fault within the read-side. It is specialized for tracing and has a single domain. It does not need the
smp_mbon the read-side, which makes it lower-overhead than SRCU.
Quoted 8 lines
Quoted 2 lines
IOW, task-trace-rcu allows the call to schedule in its critical section, whereas task-rcu does not?
Correct.
And unlike preemptible rcu, tasks-trace-rcu allows calls to schedule which do not provide priority inheritance guarantees (such as I/O triggered by page faults).
But please keep it to things allowed in vanilla RCU read-side critical sections and I/O triggered by page faults. If you need something else, that is a discussion between all the current users of RCU Tasks Trace.
Thanx, Paul
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
In reply to Mathieu DesnoyersQuoted 16 lines · Mathieu Desnoyers
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
Why is this such a hard question?
Anyway, recapping from IRC:
Quoted 8 lines
preemptible, SRCU:
counter-array based, GP advances by increasing array index and waiting for previous index to drop to 0.notably, a GP can pass while a task is preempted but not within a critical section.
SRCU has
smp_mb()in the critical sections to improve GP.
Quoted 8 lines
Also:
preemptible only allows blocking when priority inheritance is guarantees, which excludes doing I/O, and thus page faults. Otherwise a long I/O could cause the system to OOM.
SRCU allows all kind of blocking, as long as the entire SRCU domain does not mind waiting for a while before readers complete.
Well, no. Fundamentally both SRCU and preemptible (and many other flavours) are just a counter-array. The non-blocking for preempt comes from the fact that it is the main global rcu instance and allowing all that would make GPs too rare and cause you memory trouble.
But that's not because of how it's implemented, but because of it being the main global instance.
Quoted 10 lines
Quoted 10 lines
tasks:
waits for every task to passschedule()ensures that any pieces of text rendered unreachable before, is actually unused after.
tasks-rude:
like tasks, but different? build to handle tracing while rcu-idle, even though that was already deemed bad?
Quoted 8 lines
Quoted 3 lines
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?tasks-trace-rcu is meant to allow tasks to block/take a page fault within the read-side. It is specialized for tracing and has a single domain. It does not need the
smp_mbon the read-side, which makes it lower-overhead than SRCU.
That's what it's meant for, not what it is.
Turns out that tasks-tracing is a per-task counter based thing, and as such does not require all tasks to pass through schedule() and does not imply the tasks flavour (nor the tasks-rude) despite the similarity in naming.
But now I am again left wondering what the fundamental difference is between a per-task counter and a per-cpu counter.
At the end of the day, you still have to wait for the thing to hit 0.
So I'm once again confused, ...
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 60 lines · Peter Zijlstra
Quoted 16 lines · Mathieu Desnoyers
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
Why is this such a hard question?
Anyway, recapping from IRC:
Quoted 8 lines
preemptible, SRCU:
counter-array based, GP advances by increasing array index and waiting for previous index to drop to 0.notably, a GP can pass while a task is preempted but not within a critical section.
SRCU has
smp_mb()in the critical sections to improve GP.Quoted 8 lines
Also:
preemptible only allows blocking when priority inheritance is guarantees, which excludes doing I/O, and thus page faults. Otherwise a long I/O could cause the system to OOM.
SRCU allows all kind of blocking, as long as the entire SRCU domain does not mind waiting for a while before readers complete.
Well, no. Fundamentally both SRCU and preemptible (and many other flavours) are just a counter-array. The non-blocking for preempt comes from the fact that it is the main global rcu instance and allowing all that would make GPs too rare and cause you memory trouble.
But that's not because of how it's implemented, but because of it being the main global instance.
Quoted 10 lines
Quoted 10 lines
tasks:
waits for every task to passschedule()ensures that any pieces of text rendered unreachable before, is actually unused after.
tasks-rude:
like tasks, but different? build to handle tracing while rcu-idle, even though that was already deemed bad?Quoted 8 lines
Quoted 3 lines
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?tasks-trace-rcu is meant to allow tasks to block/take a page fault within the read-side. It is specialized for tracing and has a single domain. It does not need the
smp_mbon the read-side, which makes it lower-overhead than SRCU.That's what it's meant for, not what it is.
Turns out that tasks-tracing is a per-task counter based thing, and as such does not require all tasks to pass through
schedule()and does not imply the tasks flavour (nor the tasks-rude) despite the similarity in naming.
Quoted 6 lines
But now I am again left wondering what the fundamental difference is between a per-task counter and a per-cpu counter.
At the end of the day, you still have to wait for the thing to hit 0.
So I'm once again confused, ...
Updating myself.. so task-tracing-rcu is in fact *very* similar to regular preemptible-rcu but is slightly different mostly because it is *not* the main global instance.
Both are a single per-task counter (and not the per-cpu summing that I remember from many many *many* years ago; OLS'07), mostly because this helps identify which task is to blame when things go sideways.
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
In reply to Peter ZijlstraQuoted 3 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra
Why is this such a hard question?
The place to look is here:
https://docs.kernel.org/RCU/Design/Requirements/Requirements.html
Or, if you prefer, Documentation/RCU/Design/Requirements/Requirements.rst.
Quoted 10 lines
Anyway, recapping from IRC:
preemptible, SRCU:
counter-array based, GP advances by increasing array index and waiting for previous index to drop to 0.notably, a GP can pass while a task is preempted but not within a critical section.
SRCU has
smp_mb()in the critical sections to improve GP.
https://docs.kernel.org/RCU/Design/Requirements/Requirements.html#sleepable-rcu
Allows general blocking in SRCU readers, which it tolerates by giving each user its own SRCU via DEFINE_SRCU(), DEFINE_STATIC_SRCU() or a srcu_struct structure. Users blocking too much in SRCU read-side critical sections hurt only themselves. Yes, heavy-weight readers.
Quoted 5 lines
tasks:
waits for every task to passschedule()ensures that any pieces of text rendered unreachable before, is actually unused after.
But does not wait for tasks where RCU is not watching, including the idle loop.
Quoted 3 lines
tasks-rude:
like tasks, but different? build to handle tracing while rcu-idle, even though that was already deemed bad?
This waits for the tasks that RCU Tasks cannot wait for. If noinstr is fully fixed, RCU Tasks Rude can go away.
Quoted 3 lines
tasks-tracing-rcu:
extention of tasks to have critical-sections ? Should this simply be tasks?
Tasks Trace RCU is its own thing. It uses rcu_read_lock_trace() and rcu_read_unlock_trace() to mark its readers. It can detect quiescent states even when the task in question does not call schedule(). Unlike Tasks RCU, Tasks Trace RCU does not scan the full task list. (It used to, but that caused latency blows on non-realtime workloads.) Tasks Trace RCU allows preemption and blocking for page faults in its readers. Also blocking on non-raw spinlocks in PREEMPT_RT, but I am not sure that anyone cares. If you want to block on anything else, you need to talk to the current Tasks Trace RCU users.
Thanx, Paul
Quoted 4 lines
Can someone complete, please?
Steven Rostedt
maintainerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
In reply to Peter ZijlstraQuoted 3 lines · On Tue, 21 Nov 2023 15:36:47 +0100 Peter Zijlstra <[email protected]> wrote:
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
Is this similar to synchronize_rcu_tasks()? As I understand that one (grace period continues until all tasks have voluntarily scheduled or gone into user space). But I'm a bit confused by synchronize_rcu_tasks_trace()?
Note, that for syncronize_rcu_tasks() the critical sections must not call schedule (although it is OK to be preempted).
-- Steve
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Tue, 21 Nov 2023 15:36:47 +0100 Peter Zijlstra <[email protected]> wrote:
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
Quoted 6 lines
Is this similar to
synchronize_rcu_tasks()? As I understand that one (grace period continues until all tasks have voluntarily scheduled or gone into user space). But I'm a bit confused bysynchronize_rcu_tasks_trace()?Note, that for
syncronize_rcu_tasks()the critical sections must not call schedule (although it is OK to be preempted).
The synchronize_rcu_tasks() and synchronize_rcu_tasks_trace() functions are quite different, as noted elsewhere in this thread.
Thanx, Paul
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 7 lines · Mathieu Desnoyers
Task trace RCU fits a niche that has the following set of requirements/tradeoffs:
- Allow page faults within RCU read-side (like SRCU),
- Has a low-overhead read lock-unlock (without the memory barrier overhead of SRCU),
- The tradeoff: Has a rather slowsynchronize_rcu(), but tracers should not care about that. Hence, this is not meant to be a generic replacement for SRCU.
Quoted 8 lines
Quoted 4 lines
Based on my reading of https://lwn.net/Articles/253651/ , preemptible RCU is not a good fit for the following reasons:
- It disallows blocking within a RCU read-side on non-CONFIG_PREEMPT kernels,
Your counter points are confused, we simply don't build preemptible RCU unless PREEMPT=y, but that could surely be fixed and exposed as a separate flavour.
It certainly used to be available as a separate flavor, but only in CONFIG_PREEMPT=y kernels. In CONFIG_PREEMPT=n kernels, the API mapped to the non-preemptible flavor, as in synchronize_sched() and friends. And we need tracing in the full set of kernels.
Quoted 6 lines
Quoted 1 line
- AFAIU the
mmap_semused within the page fault handler does not have priority inheritance.What's that got to do with anything?
Still utterly confused about what task-tracing rcu is and how it is different from preemptible rcu.
Tasks Trace RCU allows general blocking in its readers, not just the subject-to-priority-boosting blocking permitted within preemptible RCU readers. Restrictions on the use of Tasks Trace RCU are in place to allow getting away with this general blocking. Even systems generously endowed with memory are not going to do well when the RCU grace period is blocked on I/O, especially if that I/O is across a network to a slow file server.
Which means a separate RCU instance is needed. Which is Tasks Trace RCU.
Thanx, Paul
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 8 lines · Paul E. McKenney
Tasks Trace RCU allows general blocking in its readers, not just the subject-to-priority-boosting blocking permitted within preemptible RCU readers. Restrictions on the use of Tasks Trace RCU are in place to allow getting away with this general blocking. Even systems generously endowed with memory are not going to do well when the RCU grace period is blocked on I/O, especially if that I/O is across a network to a slow file server.
Which means a separate RCU instance is needed. Which is Tasks Trace RCU.
Separate instance not a problem, nor really the question.
What is the basic mechanism of task-tracing? Is it really the existing tasks-rcu extended with read-side critical sections and call_rcu ?
If so, then why not have it be tasks-rcu?
Or is it a variant of the preemptible/SRCU class of RCUs that are counter-array based? I suspect not.
So once again, what exactly is tasks-tracing ?
Re: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 16 lines · Peter Zijlstra
Quoted 8 lines · Paul E. McKenney
Tasks Trace RCU allows general blocking in its readers, not just the subject-to-priority-boosting blocking permitted within preemptible RCU readers. Restrictions on the use of Tasks Trace RCU are in place to allow getting away with this general blocking. Even systems generously endowed with memory are not going to do well when the RCU grace period is blocked on I/O, especially if that I/O is across a network to a slow file server.
Which means a separate RCU instance is needed. Which is Tasks Trace RCU.
Separate instance not a problem, nor really the question.
What is the basic mechanism of task-tracing? Is it really the existing tasks-rcu extended with read-side critical sections and
call_rcu?
Quoted 6 lines
If so, then why not have it be tasks-rcu?
Or is it a variant of the preemptible/SRCU class of RCUs that are counter-array based? I suspect not.
So once again, what exactly is tasks-tracing ?
I think we covered this elsewhere in the thread, but if there are lingering questions, you know where to find us. ;-)
Thanx, Paul
Steven Rostedt
maintainerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
In reply to Mathieu DesnoyersQuoted 25 lines · On Mon, 20 Nov 2023 15:54:14 -0500 Mathieu Desnoyers…
include/linux/tracepoint-defs.h +13
-0@@ -29,6 +29,19 @@ struct tracepoint_func {int prio; }; +/** +* enum tracepoint_flags - Tracepoint flags +* @TRACEPOINT_MAY_EXIST: Don't return an error if the tracepoint does not +* exist upon registration. +* @TRACEPOINT_MAY_FAULT: The tracepoint probe callback will be called with +* preemption enabled, and is allowed to take page +* faults. +*/ +enum tracepoint_flags { +TRACEPOINT_MAY_EXIST = (1 <<0 ), +TRACEPOINT_MAY_FAULT = (1 <<1 ), +}; +struct tracepoint {const char *name;/* Tracepoint name */ struct static_key key;
include/linux/tracepoint-defs.h struct tracepoint { · line 55
Quoted 4 lines
include/linux/tracepoint-defs.h +1
-0@@ -39,6 +52,7 @@ struct tracepoint {int (*regfunc)(void );void (*unregfunc)(void );struct tracepoint_func __rcu *funcs; +unsigned int flags;
Since faultable and non-faultable events are mutually exclusive, why not just allocated them separately? Then you could have the __DO_TRACE() macro get passed in whether the event can be faulted or not, by the created trace.
Quoted 36 lines
include/linux/tracepoint-defs.h +0
-0};#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS include/linux/tracepoint.h +3
-9@@ -18,6 +18,7 @@#include <linux/types.h> #include <linux/cpumask.h> #include <linux/rcupdate.h> +#include <linux/rcupdate_trace.h> #include <linux/tracepoint-defs.h> #include <linux/static_call.h> @@ -41,17 +42,10 @@ extern int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data, int prio);extern int -tracepoint_probe_register_prio_may_exist (struct tracepoint *tp,void *probe,void *data, -int prio); +tracepoint_probe_register_prio_flags (struct tracepoint *tp,void *probe,void *data, +int prio,unsigned int flags);extern int tracepoint_probe_unregister (struct tracepoint *tp,void *probe,void *data); -static inline int -tracepoint_probe_register_may_exist (struct tracepoint *tp,void *probe, -void *data) -{ -return tracepoint_probe_register_prio_may_exist (tp, probe, data, -TRACEPOINT_DEFAULT_PRIO ); -}extern void for_each_kernel_tracepoint (void (*fct)(struct tracepoint *tp,void *priv),void *priv);
include/linux/tracepoint.h int unregister_tracepoint_module_notifier(struct notifier_block *nb) · line 87
Quoted 4 lines
include/linux/tracepoint.h +1
-0@@ -90,6 +84,7 @@ int unregister_tracepoint_module_notifier(struct notifier_block *nb)#ifdef CONFIG_TRACEPOINTS static inline void tracepoint_synchronize_unregister (void ) { +synchronize_rcu_tasks_trace ();
As Peter mentioned, why not use the srcu below?
Quoted 17 lines
include/linux/tracepoint.h +2
-1synchronize_srcu (&tracepoint_srcu);synchronize_rcu (); }@@ -192,9 +187,10 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* it_func[0] is never NULL because there is at least one element in the array * when the array itself is non NULL. */ -#define __DO_TRACE (name, args, cond, rcuidle) \ +#define __DO_TRACE (name, args, cond, rcuidle, tp_flags) \ do { \ int __maybe_unused __idx = 0; \ + bool mayfault = (tp_flags) & TRACEPOINT_MAY_FAULT; \ \ if (!(cond)) \ return; \@@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 206
Quoted 10 lines
include/linux/tracepoint.h +6
-2return ; \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (mayfault) { \ +rcu_read_lock_trace (); \ + }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \
Change the above comment and have:
if (!mayfault) preempt_disable_notrace();
And we can have:
if (rcuidle || mayfault) {
__idx = srcu_read_lock_notrace(&tracepoint_srcu);
if (!mayfault)
ct_irq_enter_irqson();
}Quoted 32 lines
include/linux/tracepoint.h +7
-4\/* \ * For rcuidle callers, use srcu since sched-rcu \ @@ -221,20 +221,23 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ } \ \ -preempt_enable_notrace (); \ +if (mayfault) \ +rcu_read_unlock_trace (); \ +else \ +preempt_enable_notrace (); \ } while (0)#ifndef MODULE -#define __DECLARE_TRACE_RCU (name, proto, args, cond) \ +#define __DECLARE_TRACE_RCU (name, proto, args, cond, tp_flags) \ static inline void trace_##name##_rcuidle(proto) \ { \ if (static_key_false(&__tracepoint_##name.key)) \ __DO_TRACE(name, \ TP_ARGS(args), \ - TP_CONDITION(cond), 1); \ + TP_CONDITION(cond), 1, tp_flags); \ }#else -#define __DECLARE_TRACE_RCU (name, proto, args, cond) +#define __DECLARE_TRACE_RCU (name, proto, args, cond, tp_flags)#endif /*
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 254
Quoted 5 lines
include/linux/tracepoint.h +1
-1@@ -248,7 +251,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* site if it is not watching, as it will need to be active when the * tracepoint is enabled. */ -#define __DECLARE_TRACE (name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE (name, proto, args, cond, data_proto, tp_flags) \
Instead of adding "tp_flags" just pass the "mayfault" boolean in.
Quoted 36 lines
include/linux/tracepoint.h +11
-2extern int __traceiter_##name(data_proto); \ DECLARE_STATIC_CALL(tp_func_##name, __traceiter_##name); \ extern struct tracepoint __tracepoint_##name; \@@ -257,13 +260,15 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) if (static_key_false(&__tracepoint_##name.key)) \ __DO_TRACE(name, \ TP_ARGS(args), \ - TP_CONDITION(cond), 0); \ + TP_CONDITION(cond), 0, tp_flags); \ if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ WARN_ON_ONCE(!rcu_is_watching()); \ } \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + might_fault(); \ } \ __DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args), \ - PARAMS(cond)) \ + PARAMS(cond), tp_flags) \ static inline int \ register_trace_##name(void (*probe)(data_proto), void *data) \ { \@@ -278,6 +283,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) (void *)probe, data, prio); \ } \ static inline int \ + register_trace_prio_flags_##name(void (*probe)(data_proto), void *data, \ + int prio, unsigned int flags) \ + { \ + return tracepoint_probe_register_prio_flags(&__tracepoint_##name, \ + (void *)probe, data, prio, flags); \ + } \ + static inline int \ unregister_trace_##name(void (*probe)(data_proto), void *data) \ { \ return tracepoint_probe_unregister(&__tracepoint_##name,\
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 313
Quoted 5 lines
include/linux/tracepoint.h +1
-1@@ -298,7 +310,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* structures, so we create an array of pointers that will be used for iteration * on the tracepoints. */ -#define DEFINE_TRACE_FN (_name, _reg, _unreg, proto, args) \ +#define DEFINE_TRACE_FN_FLAGS (_name, _reg, _unreg, proto, args, tp_flags) \
Instead of passing in flags, I'm thinking that the faultable tracepoints need to go into its own section, and possibly have a register_trace_mayfault_##event() to make it highly distinguishable from events that don't expect to fault.
Since everything is made by macros, it's not hard to keep all the above code, and wrap it in other macros so that the faultable and non-faultable tracepoints share most of the code.
But as tracepoints live in __section("__tracepoints"), I'm thinking we may want __section("__tracepoints_mayfault") to keep them separate.
Thoughts?
-- Steve
Quoted 59 lines
include/linux/tracepoint.h +20
-3static const char __tpstrtab_##_name[] \ __section("__tracepoints_strings") = #_name; \ extern struct static_call_key STATIC_CALL_KEY(tp_func_##_name); \@@ -314,7 +326,9 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) .probestub = &__probestub_##_name, \ .regfunc = _reg, \ .unregfunc = _unreg, \ - .funcs = NULL }; \ + .funcs = NULL, \ + .flags = (tp_flags), \ + }; \__TRACEPOINT_ENTRY (_name); \ int __traceiter_##_name(void *__data, proto) \ { \@@ -337,8 +351,11 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) } \ DEFINE_STATIC_CALL(tp_func_##_name, __traceiter_##_name); +#define DEFINE_TRACE_FN (_name, _reg, _unreg, proto, args) \ + DEFINE_TRACE_FN_FLAGS(_name, _reg, _unreg, PARAMS(proto), PARAMS(args), 0) +#define DEFINE_TRACE (name, proto, args) \ - DEFINE_TRACE_FN(name, NULL, NULL, PARAMS(proto), PARAMS(args)); + DEFINE_TRACE_FN(name, NULL, NULL, PARAMS(proto), PARAMS(args))#define EXPORT_TRACEPOINT_SYMBOL_GPL (name) \ EXPORT_SYMBOL_GPL(__tracepoint_##name); \@@ -351,7 +368,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)#else /* !TRACEPOINTS_ENABLED */ -#define __DECLARE_TRACE (name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE (name, proto, args, cond, data_proto, tp_flags) \ static inline void trace_##name(proto) \ { } \ static inline void trace_##name##_rcuidle(proto) \@@ -363,6 +380,18 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)return -ENOSYS ; \ } \ static inline int \ + register_trace_prio_##name(void (*probe)(data_proto), \ + void *data, int prio) \ + { \ + return -ENOSYS; \ + } \ + static inline int \ + register_trace_prio_flags_##name(void (*probe)(data_proto), \ + void *data, int prio, unsigned int flags) \ + { \ + return -ENOSYS; \ + } \ + static inline int \ unregister_trace_##name(void (*probe)(data_proto), \ void *data) \ { \@@ -377,6 +406,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)return false; \ }
Mathieu Desnoyers
reviewerRe: [PATCH v4 1/5] tracing: Introduce faultable tracepoints
Quoted 28 lines · Steven Rostedt
Quoted 25 lines · On Mon, 20 Nov 2023 15:54:14 -0500 Mathieu Desnoyers…
include/linux/tracepoint-defs.h +13
-0@@ -29,6 +29,19 @@ struct tracepoint_func {int prio; }; +/** +* enum tracepoint_flags - Tracepoint flags +* @TRACEPOINT_MAY_EXIST: Don't return an error if the tracepoint does not +* exist upon registration. +* @TRACEPOINT_MAY_FAULT: The tracepoint probe callback will be called with +* preemption enabled, and is allowed to take page +* faults. +*/ +enum tracepoint_flags { +TRACEPOINT_MAY_EXIST = (1 <<0 ), +TRACEPOINT_MAY_FAULT = (1 <<1 ), +}; +struct tracepoint {const char *name;/* Tracepoint name */ struct static_key key;@@ -39,6 +52,7 @@ struct tracepoint {
Quoted 8 lines
include/linux/tracepoint-defs.h struct tracepoint { · line 55
Quoted 4 lines
include/linux/tracepoint-defs.h +1
-0int (*regfunc)(void );void (*unregfunc)(void );struct tracepoint_func __rcu *funcs; +unsigned int flags;Since faultable and non-faultable events are mutually exclusive, why not just allocated them separately? Then you could have the
__DO_TRACE()macro get passed in whether the event can be faulted or not, by the created trace.
Hi Steven,
Sorry for the delayed reply. We're now resuming work on this series.
We already have may_exit and want to introduce may_fault. I want to avoid:
- combinatory explosion of the number of tracepoint API functions,
- allocating tracepoints into different sections based on their characteristics, which will make it unclear how additional axes will later fit into the scheme.
- passing a set of booleans to functions as an API, which I find more error prone than explicit flags. I prefer:
func(..., TRACEPOINT_MAY_FAULT | TRACEPOINT_MAY_EXIST);
over:
func(..., true, true);
So technically we could split faultable and non-faultable tracepoints into different sections, but how would it be an improvement over the proposed approach ? Note that the registration function checks that the faultable flag of the probe matches the faultable flag of the tracepoint, which prevents mixups already.
Quoted 36 lines
Quoted 36 lines
include/linux/tracepoint-defs.h +0
-0};#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS include/linux/tracepoint.h +3
-9@@ -18,6 +18,7 @@#include <linux/types.h> #include <linux/cpumask.h> #include <linux/rcupdate.h> +#include <linux/rcupdate_trace.h> #include <linux/tracepoint-defs.h> #include <linux/static_call.h> @@ -41,17 +42,10 @@ extern int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data, int prio);extern int -tracepoint_probe_register_prio_may_exist (struct tracepoint *tp,void *probe,void *data, -int prio); +tracepoint_probe_register_prio_flags (struct tracepoint *tp,void *probe,void *data, +int prio,unsigned int flags);extern int tracepoint_probe_unregister (struct tracepoint *tp,void *probe,void *data); -static inline int -tracepoint_probe_register_may_exist (struct tracepoint *tp,void *probe, -void *data) -{ -return tracepoint_probe_register_prio_may_exist (tp, probe, data, -TRACEPOINT_DEFAULT_PRIO ); -} extern void for_each_kernel_tracepoint(void (*fct)(struct tracepoint *tp, void *priv),
Quoted 8 lines
include/linux/tracepoint.h int unregister_tracepoint_module_notifier(struct notifier_block *nb) · line 87
Quoted 6 lines
include/linux/tracepoint.h +1
-0void *priv);@@ -90,6 +84,7 @@ int unregister_tracepoint_module_notifier(struct notifier_block *nb)#ifdef CONFIG_TRACEPOINTS static inline void tracepoint_synchronize_unregister (void ) { +synchronize_rcu_tasks_trace ();As Peter mentioned, why not use the srcu below?
This was discussed thoroughly in a separate thread. See
https://lore.kernel.org/lkml/e3721b80-4dfb-4914-acfb-b315b8cc45b8@paulmck-laptop/
Quoted 34 lines
Quoted 18 lines
include/linux/tracepoint.h +2
-1synchronize_srcu (&tracepoint_srcu);synchronize_rcu (); }@@ -192,9 +187,10 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* it_func[0] is never NULL because there is at least one element in the array * when the array itself is non NULL. */ -#define __DO_TRACE (name, args, cond, rcuidle) \ +#define __DO_TRACE (name, args, cond, rcuidle, tp_flags) \ do { \ int __maybe_unused __idx = 0; \ + bool mayfault = (tp_flags) & TRACEPOINT_MAY_FAULT; \ \ if (!(cond)) \ return; \@@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 206
Quoted 10 lines
include/linux/tracepoint.h +6
-2return ; \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (mayfault) { \ +rcu_read_lock_trace (); \ + }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \Change the above comment and have:
if (!mayfault) preempt_disable_notrace();
Quoted 7 lines
And we can have:
if (rcuidle || mayfault) { __idx = srcu_read_lock_notrace(&tracepoint_srcu); if (!mayfault) ct_irq_enter_irqson(); }
Not needed if we keep rcu_read_lock_trace() which exists for this purpose.
Quoted 31 lines
Quoted 31 lines
include/linux/tracepoint.h +7
-4\/* \ * For rcuidle callers, use srcu since sched-rcu \ @@ -221,20 +221,23 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ } \ \ -preempt_enable_notrace (); \ +if (mayfault) \ +rcu_read_unlock_trace (); \ +else \ +preempt_enable_notrace (); \ } while (0)#ifndef MODULE -#define __DECLARE_TRACE_RCU (name, proto, args, cond) \ +#define __DECLARE_TRACE_RCU (name, proto, args, cond, tp_flags) \ static inline void trace_##name##_rcuidle(proto) \ { \ if (static_key_false(&__tracepoint_##name.key)) \ __DO_TRACE(name, \ TP_ARGS(args), \ - TP_CONDITION(cond), 1); \ + TP_CONDITION(cond), 1, tp_flags); \ }#else -#define __DECLARE_TRACE_RCU (name, proto, args, cond) +#define __DECLARE_TRACE_RCU (name, proto, args, cond, tp_flags)#endif
Quoted 9 lines
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 254
Quoted 7 lines
include/linux/tracepoint.h +1
-1/* @@ -248,7 +251,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* site if it is not watching, as it will need to be active when the * tracepoint is enabled. */ -#define __DECLARE_TRACE (name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE (name, proto, args, cond, data_proto, tp_flags) \Instead of adding "
tp_flags" just pass the "mayfault" boolean in.
As explained above, I want to avoid combinatory explosion of the number of API functions with ..._may_exist_may_fault_may_xxx_may_yyy(). I also want to avoid the pattern where we need as many booleans, e.g.:
...(..., bool may_exist, bool may_fault, bool may_xxx, bool may_yyy)
which then looks like a maze of (... true, false, true, false) in the caller macros. This is error prone and tricky to review.
The solution I propose to this problem is introducing the tp_flags. In the case of __DECLARE_TRACE, the tp_flags are only used for their TRACEPOINT_MAY_FAULT bit, but it keeps things consistent everywhere:
at tracepoint declaration, registration and use.
Note that ((tp_flags) & TRACEPOINT_MAY_FAULT) evaluates to a constant, so there is no performance overhead involved.
I would favor keeping the tp_flags to keep everything consistent.
Quoted 39 lines
Quoted 39 lines
include/linux/tracepoint.h +11
-2extern int __traceiter_##name(data_proto); \ DECLARE_STATIC_CALL(tp_func_##name, __traceiter_##name); \ extern struct tracepoint __tracepoint_##name; \@@ -257,13 +260,15 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) if (static_key_false(&__tracepoint_##name.key)) \ __DO_TRACE(name, \ TP_ARGS(args), \ - TP_CONDITION(cond), 0); \ + TP_CONDITION(cond), 0, tp_flags); \ if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ WARN_ON_ONCE(!rcu_is_watching()); \ } \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + might_fault(); \ } \ __DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args), \ - PARAMS(cond)) \ + PARAMS(cond), tp_flags) \ static inline int \ register_trace_##name(void (*probe)(data_proto), void *data) \ { \@@ -278,6 +283,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) (void *)probe, data, prio); \ } \ static inline int \ + register_trace_prio_flags_##name(void (*probe)(data_proto), void *data, \ + int prio, unsigned int flags) \ + { \ + return tracepoint_probe_register_prio_flags(&__tracepoint_##name, \ + (void *)probe, data, prio, flags); \ + } \ + static inline int \ unregister_trace_##name(void (*probe)(data_proto), void *data) \ { \ return tracepoint_probe_unregister(&__tracepoint_##name,\@@ -298,7 +310,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)* structures, so we create an array of pointers that will be used for iteration * on the tracepoints.
Quoted 8 lines
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 313
Quoted 3 lines
include/linux/tracepoint.h +1
-1*/ -#define DEFINE_TRACE_FN (_name, _reg, _unreg, proto, args) \ +#define DEFINE_TRACE_FN_FLAGS (_name, _reg, _unreg, proto, args, tp_flags) \Instead of passing in flags, I'm thinking that the faultable tracepoints need to go into its own section, and possibly have a register_trace_mayfault_##event() to make it highly distinguishable from events that don't expect to fault.
Registering a probe over a tracepoint with may_fault bit mismatch is already rejected.
I'm concerned about multiplying the number of API functions. It may look fine just now to add "just one more" axis and have ...may_fault_may_exist_prio(), but I already find it has reached its limits. This is why I favor the flags.
As for placing the faultable tracepoints into their own section, what is the benefit in doing that ?
Quoted 7 lines
Since everything is made by macros, it's not hard to keep all the above code, and wrap it in other macros so that the faultable and non-faultable tracepoints share most of the code.
But as tracepoints live in
__section("__tracepoints"), I'm thinking we may want__section("__tracepoints_mayfault")to keep them separate.
We could do that, but I'm not sure what we'd gain, and it would certainly make things awkward when other mutually exclusive "may_..." axes need to be added in the future.
Thanks,
Mathieu
Quoted 65 lines
Thoughts?
-- Steve
Quoted 59 lines
include/linux/tracepoint.h +20
-3static const char __tpstrtab_##_name[] \ __section("__tracepoints_strings") = #_name; \ extern struct static_call_key STATIC_CALL_KEY(tp_func_##_name); \@@ -314,7 +326,9 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) .probestub = &__probestub_##_name, \ .regfunc = _reg, \ .unregfunc = _unreg, \ - .funcs = NULL }; \ + .funcs = NULL, \ + .flags = (tp_flags), \ + }; \__TRACEPOINT_ENTRY (_name); \ int __traceiter_##_name(void *__data, proto) \ { \@@ -337,8 +351,11 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) } \ DEFINE_STATIC_CALL(tp_func_##_name, __traceiter_##_name); +#define DEFINE_TRACE_FN (_name, _reg, _unreg, proto, args) \ + DEFINE_TRACE_FN_FLAGS(_name, _reg, _unreg, PARAMS(proto), PARAMS(args), 0) +#define DEFINE_TRACE (name, proto, args) \ - DEFINE_TRACE_FN(name, NULL, NULL, PARAMS(proto), PARAMS(args)); + DEFINE_TRACE_FN(name, NULL, NULL, PARAMS(proto), PARAMS(args))#define EXPORT_TRACEPOINT_SYMBOL_GPL (name) \ EXPORT_SYMBOL_GPL(__tracepoint_##name); \@@ -351,7 +368,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)#else /* !TRACEPOINTS_ENABLED */ -#define __DECLARE_TRACE (name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE (name, proto, args, cond, data_proto, tp_flags) \ static inline void trace_##name(proto) \ { } \ static inline void trace_##name##_rcuidle(proto) \@@ -363,6 +380,18 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)return -ENOSYS ; \ } \ static inline int \ + register_trace_prio_##name(void (*probe)(data_proto), \ + void *data, int prio) \ + { \ + return -ENOSYS; \ + } \ + static inline int \ + register_trace_prio_flags_##name(void (*probe)(data_proto), \ + void *data, int prio, unsigned int flags) \ + { \ + return -ENOSYS; \ + } \ + static inline int \ unregister_trace_##name(void (*probe)(data_proto), \ void *data) \ { \@@ -377,6 +406,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)return false; \ }
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Mathieu Desnoyers
reviewerPATCH v4 2/5 tracing/ftrace: Add support for faultable tracepoints
In reply to Mathieu DesnoyersIn preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that ftrace can handle registering to such tracepoints by explicitly disabling preemption within the ftrace tracepoint probes to respect the current expectations within ftrace ring buffer code.
This change does not yet allow ftrace to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints.
--- include/trace/trace_events.h | 69 +++++++++++++++++++++++++++++++++--- kernel/trace/trace_events.c | 26 ++++++++++---- 2 files changed, 84 insertions(+), 11 deletions(-)
include/trace/trace_events.h +64 -5
@@ -45,6 +45,16 @@ PARAMS(print)); \DEFINE_EVENT (name, name,PARAMS (proto),PARAMS (args)); +#undef TRACE_EVENT_MAY_FAULT +#define TRACE_EVENT_MAY_FAULT (name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS_MAY_FAULT(name, \ + PARAMS(proto), \ + PARAMS(args), \ + PARAMS(tstruct), \ + PARAMS(assign), \ + PARAMS(print)); \ + DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args)); +#include "stages/stage1_struct_define.h" #undef DECLARE_EVENT_CLASS
@@ -57,6 +67,11 @@ \ static struct trace_event_class event_class_##name; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, name, proto, args) \ static struct trace_event_call __used \
@@ -80,7 +95,7 @@#undef TRACE_EVENT_FN_MAY_FAULT#define TRACE_EVENT_FN_MAY_FAULT (name, proto, args, tstruct, \ assign, print, reg, unreg) \ - TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + TRACE_EVENT_MAY_FAULT(name, PARAMS(proto), PARAMS(args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \#undef TRACE_EVENT_FN_COND
@@ -123,6 +138,11 @@ tstruct; \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, name, proto, args)
@@ -214,6 +234,11 @@ static struct trace_event_functions trace_event_type_funcs_##call = { \ .trace = trace_raw_output_##call, \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT_PRINT#define DEFINE_EVENT_PRINT (template, call, proto, args, print) \ static notrace enum print_line_t \
@@ -250,6 +275,11 @@ static struct trace_event_fields trace_event_fields_##call[] = { \ tstruct \ {} }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT_PRINT#define DEFINE_EVENT_PRINT (template, name, proto, args, print)
@@ -271,6 +301,11 @@ static inline notrace int trace_event_get_offsets_##call( \return __data_size; \ } +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#include TRACE_INCLUDE (TRACE_INCLUDE_FILE )/*
@@ -380,8 +415,8 @@ static inline notrace int trace_event_get_offsets_##call( \#include "stages/stage6_event_callback.h" -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \
@@ -392,8 +427,13 @@ trace_event_raw_event_##call(void *__data, proto) \ struct trace_event_raw_##call *entry; \ int __data_size; \ \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ if (trace_trigger_soft_disabled(trace_file)) \ - return; \ + goto end; \ \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \
@@ -401,14 +441,28 @@ trace_event_raw_event_##call(void *__data, proto) \ sizeof(*entry) + __data_size); \ \if (!entry) \ -return ; \ + goto end; \ \ tstruct \ \ { assign; } \ \trace_event_buffer_commit (&fbuffer); \ +end: \ +if ((tp_flags) &TRACEPOINT_MAY_FAULT ) \ +preempt_enable_notrace (); \ } + +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), TRACEPOINT_MAY_FAULT) +/* * The ftrace_test_probe is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the ftrace probe will
@@ -440,6 +494,11 @@ static struct trace_event_class __used __refdata event_class_##call = { \ _TRACE_PERF_INIT(call) \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, call, proto, args) \ \
kernel/trace/trace_events.c +20 -6
@@ -532,9 +532,16 @@ int trace_event_reg(struct trace_event_call *call,WARN_ON (!(call->flags &TRACE_EVENT_FL_TRACEPOINT ));switch (type) {case TRACE_REG_REGISTER : -return tracepoint_probe_register (call->tp, - call->class->probe, - file); +if (call->tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (call->tp, + call->class->probe, + file, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register (call->tp, + call->class->probe, + file);case TRACE_REG_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->probe,
@@ -543,9 +550,16 @@ int trace_event_reg(struct trace_event_call *call,#ifdef CONFIG_PERF_EVENTS case TRACE_REG_PERF_REGISTER : -return tracepoint_probe_register (call->tp, - call->class->perf_probe, - call); +if (call->tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (call->tp, + call->class->perf_probe, + call, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register (call->tp, + call->class->perf_probe, + call);case TRACE_REG_PERF_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->perf_probe,
Signature · 1 line
2.25.1
Re: [PATCH v4 2/5] tracing/ftrace: Add support for faultable tracepoints
Quoted 77 lines · Mathieu Desnoyers
(quoted diff) +12
-4@@ -380,8 +415,8 @@ static inline notrace int trace_event_get_offsets_##call( \ #include "stages/stage6_event_callback.h" -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS(call, proto, args, tstruct, assign, print, tp_flags) \ \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \@@ -392,8 +427,13 @@ trace_event_raw_event_##call(void *__data, proto) \ struct trace_event_raw_##call *entry; \ int __data_size; \ \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ if (trace_trigger_soft_disabled(trace_file)) \ - return; \ + goto end; \ \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \@@ -401,14 +441,28 @@ trace_event_raw_event_##call(void *__data, proto) \ sizeof(*entry) + __data_size); \ \ if (!entry) \ - return; \ + goto end; \ \ tstruct \ \ { assign; } \ \ trace_event_buffer_commit(&fbuffer); \ +end: \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + preempt_enable_notrace(); \ }kernel/trace/trace_events.c +13
-6@@ -532,9 +532,16 @@ int trace_event_reg(struct trace_event_call *call,WARN_ON (!(call->flags &TRACE_EVENT_FL_TRACEPOINT ));switch (type) {case TRACE_REG_REGISTER : -return tracepoint_probe_register (call->tp, - call->class->probe, - file); +if (call->tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (call->tp, + call->class->probe, + file, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register (call->tp, + call->class->probe, + file);case TRACE_REG_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->probe,@@ -543,9 +550,16 @@ int trace_event_reg(struct trace_event_call *call,#ifdef CONFIG_PERF_EVENTS case TRACE_REG_PERF_REGISTER : -return tracepoint_probe_register (call->tp, - call->class->perf_probe, - call); +if (call->tp->flags &TRACEPOINT_MAY_FAULT ) + return tracepoint_probe_register_prio_flags(call->tp, + call->class->perf_probe,
kernel/trace/trace_events.c int trace_event_reg(struct trace_event_call *call, · line 565
Quoted 10 lines
kernel/trace/trace_events.c +7
-0+ call, +TRACEPOINT_DEFAULT_PRIO , + TRACEPOINT_MAY_FAULT); + else + return tracepoint_probe_register(call->tp, + call->class->perf_probe, + call);case TRACE_REG_PERF_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->perf_probe,
I think something like the below (which is on top of the cleanup patch in tip/locking/core) might just do...
---
include/linux/cleanup.h +20 -1
@@ -151,7 +151,9 @@ static inline class_##_name##_t class_##_name##ext##_constructor(_init_args) \#define DEFINE_GUARD (_name, _type, _lock, _unlock) \ DEFINE_CLASS(_name, _type, if (_T) { _unlock; }, ({ _lock; _T; }), _type _T); \ static inline void * class_##_name##_lock_ptr(class_##_name##_t *_T) \ - { return *_T; } + { return *_T; } \ + static inline class_##_name##_t class_##_name##_null(void) \ + { return NULL; }#define DEFINE_GUARD_COND (_name, _ext, _condlock) \ EXTEND_CLASS(_name, _ext, \
@@ -175,6 +177,17 @@ static inline class_##_name##_t class_##_name##ext##_constructor(_init_args) \if (!__guard_ptr (_name)(&scope)) _fail; \ else + +#define __guard_if (_cond, _name, var) \ + class_##_name##_t var __cleanup(class_##_name##_destructor) = \ + class_##_name##_null(); \ + if (_cond) \ + var = class_##_name##_constructor + +#define guard_if (_cond, _name) \ + __guard_if(_cond, _name, __UNIQUE_ID(guard)) + +/* * Additional helper macros for generating lock guards with types, either for * locks that don't have a native type (eg. RCU, preempt) or those that need a
@@ -209,6 +222,12 @@ static inline void class_##_name##_destructor(class_##_name##_t *_T) \ static inline void *class_##_name##_lock_ptr(class_##_name##_t *_T) \ { \return _T->lock; \ +} \ + \ +static inline class_##_name##_t class_##_name##_null(void) \ +{ \ + class_##_name##_t _t = { }; \ +return _t; \ }
include/trace/trace_events.h +3 -9
@@ -427,13 +427,10 @@ trace_event_raw_event_##call(void *__data, proto) \ struct trace_event_raw_##call *entry; \ int __data_size; \ \ - if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ - might_fault(); \ - preempt_disable_notrace(); \ - } \ + guard_if((tp_flags) & TRACEPOINT_MAY_FAULT, preempt_notrace)(); \ \ if (trace_trigger_soft_disabled(trace_file)) \ - goto end; \ + return; \ \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \
@@ -441,16 +438,13 @@ trace_event_raw_event_##call(void *__data, proto) \ sizeof(*entry) + __data_size); \ \if (!entry) \ - goto end; \ +return ; \ \ tstruct \ \ { assign; } \ \trace_event_buffer_commit (&fbuffer); \ -end: \ -if ((tp_flags) &TRACEPOINT_MAY_FAULT ) \ -preempt_enable_notrace (); \ }#undef DECLARE_EVENT_CLASS
kernel/trace/trace_events.c +10 -20
@@ -532,16 +532,11 @@ int trace_event_reg(struct trace_event_call *call,WARN_ON (!(call->flags &TRACE_EVENT_FL_TRACEPOINT ));switch (type) {case TRACE_REG_REGISTER : -if (call->tp->flags &TRACEPOINT_MAY_FAULT ) -return tracepoint_probe_register_prio_flags (call->tp, - call->class->probe, - file, -TRACEPOINT_DEFAULT_PRIO , -TRACEPOINT_MAY_FAULT ); -else -return tracepoint_probe_register (call->tp, - call->class->probe, - file); +return tracepoint_probe_register_prio_flags (call->tp, + call->class->probe, + file, +TRACEPOINT_DEFAULT_PRIO , + call->tp->flags &TRACEPOINT_MAY_FAULT );case TRACE_REG_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->probe,
@@ -550,16 +545,11 @@ int trace_event_reg(struct trace_event_call *call,#ifdef CONFIG_PERF_EVENTS case TRACE_REG_PERF_REGISTER : -if (call->tp->flags &TRACEPOINT_MAY_FAULT ) -return tracepoint_probe_register_prio_flags (call->tp, - call->class->perf_probe, - call, -TRACEPOINT_DEFAULT_PRIO , -TRACEPOINT_MAY_FAULT ); -else -return tracepoint_probe_register (call->tp, - call->class->perf_probe, - call); +return tracepoint_probe_register_prio_flags (call->tp, + call->class->perf_probe, + call, +TRACEPOINT_DEFAULT_PRIO , + call->tp->flags &TRACEPOINT_MAY_FAULT );case TRACE_REG_PERF_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->perf_probe,
Mathieu Desnoyers
reviewerRe: [PATCH v4 2/5] tracing/ftrace: Add support for faultable tracepoints
Quoted 3 lines · Peter Zijlstra
Quoted 2 lines · Mathieu Desnoyers
(quoted diff) +0
-0@@ -380,8 +415,8 @@ static inline notrace int trace_event_get_offsets_##call( \
[...]
Quoted 3 lines
I think something like the below (which is on top of the cleanup patch in tip/locking/core) might just do...
[ guard stuff ]
Yes, we will apply those suggestions, thanks!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Mathieu Desnoyers
reviewerPATCH v4 3/5 tracing/bpf-trace: add support for faultable tracepoints
In reply to Mathieu DesnoyersIn preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that bpf can handle registering to such tracepoints by explicitly disabling preemption within the bpf tracepoint probes to respect the current expectations within bpf tracing code.
This change does not yet allow bpf to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints.
--- include/trace/bpf_probe.h | 21 +++++++++++++++++---- kernel/trace/bpf_trace.c | 11 ++++++++--- 2 files changed, 25 insertions(+), 7 deletions(-)
include/trace/bpf_probe.h +17 -4
@@ -42,17 +42,30 @@/* tracepoints with more than 12 arguments will hit build error */ #define CAST_TO_U64 (...) CONCATENATE(__CAST, COUNT_ARGS(__VA_ARGS__))(__VA_ARGS__) -#define __BPF_DECLARE_TRACE (call, proto, args) \ +#define __BPF_DECLARE_TRACE (call, proto, args, tp_flags) \ static notrace void \ __bpf_trace_##call(void *__data, proto) \ { \ struct bpf_prog *prog = __data; \ + \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(prog, CAST_TO_U64(args)); \ + \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + preempt_enable_notrace(); \ }#undef DECLARE_EVENT_CLASS#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), TRACEPOINT_MAY_FAULT)/* * This part is compiled out, it is only here as a build time check
@@ -106,13 +119,13 @@ static inline void bpf_test_buffer_##call(void) \#undef DECLARE_TRACE#define DECLARE_TRACE (call, proto, args) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) \ __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), 0)#undef DECLARE_TRACE_WRITABLE#define DECLARE_TRACE_WRITABLE (call, proto, args, size) \ __CHECK_WRITABLE_BUF_SIZE(call, PARAMS(proto), PARAMS(args), size) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) \ __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), size)#include TRACE_INCLUDE (TRACE_INCLUDE_FILE )
kernel/trace/bpf_trace.c +8 -3
@@ -2368,9 +2368,14 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *if (prog->aux->max_tp_access > btp->writable_size)return -EINVAL ; -return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, - prog,TRACEPOINT_DEFAULT_PRIO , -TRACEPOINT_MAY_EXIST ); +if (tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, + prog,TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_EXIST |TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, + prog,TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_EXIST ); }int bpf_probe_register (struct bpf_raw_event_map *btp,struct bpf_prog *prog)
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerPATCH v4 4/5 tracing/perf: add support for faultable tracepoints
In reply to Mathieu DesnoyersIn preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that perf can handle registering to such tracepoints by explicitly disabling preemption within the perf tracepoint probes to respect the current expectations within perf ring buffer code.
This change does not yet allow perf to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints.
--- include/trace/perf.h | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-)
include/trace/perf.h +23 -4
@@ -12,8 +12,8 @@#undef __perf_task#define __perf_task (t) (__task = (t)) -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ static notrace void \ perf_trace_##call(void *__data, proto) \ { \
@@ -28,13 +28,18 @@ perf_trace_##call(void *__data, proto) \int __data_size; \int rctx; \ \ +if ((tp_flags) &TRACEPOINT_MAY_FAULT ) { \ +might_fault (); \ +preempt_disable_notrace (); \ + } \ + \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \ head = this_cpu_ptr(event_call->perf_events); \ if (!bpf_prog_array_valid(event_call) && \ __builtin_constant_p(!__task) && !__task && \ hlist_empty(head)) \ - return; \ + goto end; \ \ __entry_size = ALIGN(__data_size + sizeof(*entry) + sizeof(u32),\ sizeof(u64)); \
@@ -42,7 +47,7 @@ perf_trace_##call(void *__data, proto) \ \ entry =perf_trace_buf_alloc (__entry_size, &__regs, &rctx); \if (!entry) \ -return ; \ + goto end; \ \perf_fetch_caller_regs (__regs); \ \
@@ -53,8 +58,22 @@ perf_trace_##call(void *__data, proto) \perf_trace_run_bpf_submit (entry, __entry_size, rctx, \ event_call, __count, __regs, \ head, __task); \ +end: \ +if ((tp_flags) &TRACEPOINT_MAY_FAULT ) \ +preempt_enable_notrace (); \ } +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), \ + TRACEPOINT_MAY_FAULT) +/* * This part is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerPATCH v4 5/5 tracing: convert sys_enter/exit to faultable tracepoints
In reply to Mathieu DesnoyersConvert the definition of the system call enter/exit tracepoints to faultable tracepoints now that all upstream tracers handle it.
This allows tracers to fault-in userspace system call arguments such as path strings within their probe callbacks.
--- include/trace/events/syscalls.h | 4 +- kernel/trace/trace_syscalls.c | 92 +++++++++++++++++++++++---------- 2 files changed, 68 insertions(+), 28 deletions(-)
include/trace/events/syscalls.h +2 -2
@@ -15,7 +15,7 @@#ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS -TRACE_EVENT_FN(sys_enter, +TRACE_EVENT_FN_MAY_FAULT(sys_enter, TP_PROTO(struct pt_regs *regs, long id),
@@ -41,7 +41,7 @@ TRACE_EVENT_FN(sys_enter, TRACE_EVENT_FLAGS(sys_enter, TRACE_EVENT_FL_CAP_ANY) -TRACE_EVENT_FN(sys_exit, +TRACE_EVENT_FN_MAY_FAULT(sys_exit, TP_PROTO(struct pt_regs *regs, long ret),
kernel/trace/trace_syscalls.c +66 -26
235 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 5/5] tracing: convert sys_enter/exit to faultable tracepoints
kernel/trace/trace_syscalls.c static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id) · line 306
Quoted 13 lines · Mathieu Desnoyers
kernel/trace/trace_syscalls.c +5
-0@@ -299,27 +299,33 @@ static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id)int syscall_nr;int size; +/* +* Probe called with preemption enabled (may_fault), but ring buffer and +* per-cpu data require preemption to be disabled. +*/ +preempt_disable_notrace ();
guard(preempt_notrace)();
and ditch all the goto crap.
Quoted 40 lines
kernel/trace/trace_syscalls.c +8
-5+ syscall_nr =trace_get_syscall_nr (current, regs);if (syscall_nr <0 || syscall_nr >= NR_syscalls) -return ; + goto end;/* Here we're inside tp handler's rcu_read_lock_sched (__DO_TRACE) */ trace_file =rcu_dereference_sched (tr->enter_syscall_files[syscall_nr]);if (!trace_file) -return ; + goto end;if (trace_trigger_soft_disabled (trace_file)) -return ; + goto end; sys_data =syscall_nr_to_meta (syscall_nr);if (!sys_data) -return ; + goto end; size =sizeof (*entry) +sizeof (unsigned long ) * sys_data->nb_args; entry =trace_event_buffer_reserve (&fbuffer, trace_file, size);if (!entry) -return ; + goto end; entry =ring_buffer_event_data (fbuffer.event); entry->nr = syscall_nr;@@ -327,6 +333,8 @@ static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id)memcpy (entry->args, args,sizeof (unsigned long ) * sys_data->nb_args);trace_event_buffer_commit (&fbuffer); +end: +preempt_enable_notrace (); }static void ftrace_syscall_exit (void *data,struct pt_regs *regs,long ret)
kernel/trace/trace_syscalls.c static void ftrace_syscall_exit(void *data, struct pt_regs *regs, long ret) · line 353
Quoted 8 lines
kernel/trace/trace_syscalls.c +5
-0@@ -338,31 +346,39 @@ static void ftrace_syscall_exit(void *data, struct pt_regs *regs, long ret)struct trace_event_buffer fbuffer;int syscall_nr; +/* +* Probe called with preemption enabled (may_fault), but ring buffer and +* per-cpu data require preemption to be disabled. +*/ +preempt_disable_notrace ();
Idem.
Quoted 48 lines
kernel/trace/trace_syscalls.c +11
-6+ syscall_nr =trace_get_syscall_nr (current, regs);if (syscall_nr <0 || syscall_nr >= NR_syscalls) -return ; + goto end;/* Here we're inside tp handler's rcu_read_lock_sched (__DO_TRACE()) */ trace_file =rcu_dereference_sched (tr->exit_syscall_files[syscall_nr]);if (!trace_file) -return ; + goto end;if (trace_trigger_soft_disabled (trace_file)) -return ; + goto end; sys_data =syscall_nr_to_meta (syscall_nr);if (!sys_data) -return ; + goto end; entry =trace_event_buffer_reserve (&fbuffer, trace_file,sizeof (*entry));if (!entry) -return ; + goto end; entry =ring_buffer_event_data (fbuffer.event); entry->nr = syscall_nr; entry->ret =syscall_get_return_value (current, regs);trace_event_buffer_commit (&fbuffer); +end: +preempt_enable_notrace (); } static int reg_event_syscall_enter(struct trace_event_file *file,@@ -377,7 +393,9 @@ static int reg_event_syscall_enter(struct trace_event_file *file,return -ENOSYS ;mutex_lock (&syscall_trace_lock);if (!tr->sys_refcount_enter) - ret =register_trace_sys_enter (ftrace_syscall_enter, tr); + ret =register_trace_prio_flags_sys_enter (ftrace_syscall_enter, tr, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT );if (!ret) {rcu_assign_pointer (tr->enter_syscall_files[num], file); tr->sys_refcount_enter++;
kernel/trace/trace_syscalls.c static int reg_event_syscall_exit(struct trace_event_file *file, · line 441
Quoted 10 lines
kernel/trace/trace_syscalls.c +3
-1@@ -415,7 +433,9 @@ static int reg_event_syscall_exit(struct trace_event_file *file,return -ENOSYS ;mutex_lock (&syscall_trace_lock);if (!tr->sys_refcount_exit) - ret =register_trace_sys_exit (ftrace_syscall_exit, tr); + ret =register_trace_prio_flags_sys_exit (ftrace_syscall_exit, tr, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT );if (!ret) {rcu_assign_pointer (tr->exit_syscall_files[num], file); tr->sys_refcount_exit++;
/me hands you a bucket of {}, free of charge.
kernel/trace/trace_syscalls.c static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · line 609
Quoted 9 lines
kernel/trace/trace_syscalls.c +5
-0@@ -582,20 +602,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id)int rctx;int size; +/* +* Probe called with preemption enabled (may_fault), but ring buffer and +* per-cpu data require preemption to be disabled. +*/ +preempt_disable_notrace ();
Again, guard(preempt_notrace)();
Quoted 48 lines
kernel/trace/trace_syscalls.c +9
-6+ syscall_nr =trace_get_syscall_nr (current, regs);if (syscall_nr <0 || syscall_nr >= NR_syscalls) -return ; + goto end;if (!test_bit (syscall_nr, enabled_perf_enter_syscalls)) -return ; + goto end; sys_data =syscall_nr_to_meta (syscall_nr);if (!sys_data) -return ; + goto end; head =this_cpu_ptr (sys_data->enter_event->perf_events); valid_prog_array =bpf_prog_array_valid (sys_data->enter_event);if (!valid_prog_array &&hlist_empty (head)) -return ; + goto end;/* get the size after alignment with the u32 buffer size field */ size =sizeof (unsigned long ) * sys_data->nb_args +sizeof (*rec);@@ -604,7 +630,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) rec =perf_trace_buf_alloc (size,NULL , &rctx);if (!rec) -return ; + goto end; rec->nr = syscall_nr;syscall_get_arguments (current, regs, args);@@ -614,12 +640,14 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) !perf_call_bpf_enter(sys_data->enter_event, regs, sys_data, rec)) || hlist_empty(head)) {perf_swevent_put_recursion_context (rctx); -return ; + goto end; }perf_trace_buf_submit (rec, size, rctx, sys_data->enter_event->event.type,1 , regs, head,NULL ); +end: +preempt_enable_notrace (); }static int perf_sysenter_enable (struct trace_event_call *call)
kernel/trace/trace_syscalls.c static int perf_sysenter_enable(struct trace_event_call *call) · line 664
Quoted 7 lines
kernel/trace/trace_syscalls.c +3
-1@@ -631,7 +659,9 @@ static int perf_sysenter_enable(struct trace_event_call *call)mutex_lock (&syscall_trace_lock);if (!sys_perf_refcount_enter) - ret =register_trace_sys_enter (perf_syscall_enter,NULL ); + ret =register_trace_prio_flags_sys_enter (perf_syscall_enter,NULL , +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT );
More {}
kernel/trace/trace_syscalls.c static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) · line 719
Quoted 12 lines
kernel/trace/trace_syscalls.c +5
-0if (ret) {pr_info ("event trace: Could not activate syscall entry trace point" ); }else {@@ -682,20 +712,26 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret)int rctx;int size; +/* +* Probe called with preemption enabled (may_fault), but ring buffer and +* per-cpu data require preemption to be disabled. +*/ +preempt_disable_notrace ();
Guess?
Quoted 47 lines
kernel/trace/trace_syscalls.c +9
-6+ syscall_nr =trace_get_syscall_nr (current, regs);if (syscall_nr <0 || syscall_nr >= NR_syscalls) -return ; + goto end;if (!test_bit (syscall_nr, enabled_perf_exit_syscalls)) -return ; + goto end; sys_data =syscall_nr_to_meta (syscall_nr);if (!sys_data) -return ; + goto end; head =this_cpu_ptr (sys_data->exit_event->perf_events); valid_prog_array =bpf_prog_array_valid (sys_data->exit_event);if (!valid_prog_array &&hlist_empty (head)) -return ; + goto end;/* We can probably do that at build time */ size =ALIGN (sizeof (*rec) +sizeof (u32),sizeof (u64));@@ -703,7 +739,7 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) rec =perf_trace_buf_alloc (size,NULL , &rctx);if (!rec) -return ; + goto end; rec->nr = syscall_nr; rec->ret =syscall_get_return_value (current, regs);@@ -712,11 +748,13 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) !perf_call_bpf_exit(sys_data->exit_event, regs, rec)) || hlist_empty(head)) {perf_swevent_put_recursion_context (rctx); -return ; + goto end; }perf_trace_buf_submit (rec, size, rctx, sys_data->exit_event->event.type,1 , regs, head,NULL ); +end: +preempt_enable_notrace (); }static int perf_sysexit_enable (struct trace_event_call *call)
kernel/trace/trace_syscalls.c static int perf_sysexit_enable(struct trace_event_call *call) · line 771
Quoted 7 lines
kernel/trace/trace_syscalls.c +3
-1@@ -728,7 +766,9 @@ static int perf_sysexit_enable(struct trace_event_call *call)mutex_lock (&syscall_trace_lock);if (!sys_perf_refcount_exit) - ret =register_trace_sys_exit (perf_syscall_exit,NULL ); + ret =register_trace_prio_flags_sys_exit (perf_syscall_exit,NULL , +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT );
And yet more {}
Quoted 6 lines
kernel/trace/trace_syscalls.c +0
-0if (ret) {pr_info ("event trace: Could not activate syscall exit trace point" ); }else {Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerRe: [PATCH v4 5/5] tracing: convert sys_enter/exit to faultable tracepoints
Quoted 18 lines · Peter Zijlstra
kernel/trace/trace_syscalls.c static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id) · line 306
Quoted 13 lines · Mathieu Desnoyers
kernel/trace/trace_syscalls.c +5
-0@@ -299,27 +299,33 @@ static void ftrace_syscall_enter(void *data, struct pt_regs *regs, long id)int syscall_nr;int size; +/* +* Probe called with preemption enabled (may_fault), but ring buffer and +* per-cpu data require preemption to be disabled. +*/ +preempt_disable_notrace ();guard(preempt_notrace)();
Quoted 1 line
and ditch all the goto crap.
[ more guard stuff ]
Sure, will do, thanks!
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com