Faultable Tracepoints
Subsystems (6)
Mathieu Desnoyers
reviewerv3 cover Faultable Tracepoints
Wire up the system call tracepoints with Tasks Trace RCU to allow the ftrace, perf, and eBPF tracers to handle page faults.
This series does the initial wire-up allowing tracers to handle page faults, but leaves out the actual handling of said page faults as future work.
This series is based on kernel v6.5.5.
Feedback is welcome!
Thanks,
Mathieu
Mathieu Desnoyers (5):
tracing: Introduce faultable tracepoints (v3) tracing/ftrace: Add support for faultable tracepoints tracing/bpf-trace: add support for faultable tracepoints tracing/perf: add support for faultable tracepoints
tracing: convert sys_enter/exit to faultable tracepoints
include/linux/tracepoint-defs.h | 14 +++++ include/linux/tracepoint.h | 88 ++++++++++++++++++++++--------- include/trace/bpf_probe.h | 21 ++++++-- include/trace/define_trace.h | 7 +++ include/trace/events/syscalls.h | 4 +- include/trace/perf.h | 27 ++++++++-- include/trace/trace_events.h | 73 ++++++++++++++++++++++++-- init/Kconfig | 1 + kernel/trace/bpf_trace.c | 10 +++- kernel/trace/trace_events.c | 26 +++++++--- kernel/trace/trace_fprobe.c | 5 +- kernel/trace/trace_syscalls.c | 92 +++++++++++++++++++++++---------- kernel/tracepoint.c | 58 +++++++++++---------- 13 files changed, 325 insertions(+), 101 deletions(-)
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerRFC PATCH v3 1/5 tracing: Introduce faultable tracepoints (v3)
When invoked from system call enter/exit instrumentation, accessing user-space data is a common use-case for tracers. However, tracepoints currently disable preemption around iteration on the registered tracepoint probes and invocation of the probe callbacks, which prevents tracers from handling page faults.
Extend the tracepoint and trace event APIs to allow defining a faultable tracepoint which invokes its callback with preemption enabled.
Also extend the tracepoint API to allow tracers to request specific probes to be connected to those faultable tracepoints. When the TRACEPOINT_MAY_FAULT flag is provided on registration, the probe callback will be called with preemption enabled, and is allowed to take page faults. Faultable probes can only be registered on faultable tracepoints and non-faultable probes on non-faultable tracepoints.
The tasks trace rcu mechanism is used to synchronize read-side marshalling of the registered probes with respect to faultable probes unregistration and teardown.
---
Changes since v1:
- Cleanup __DO_TRACE() implementation.
- Rename "sleepable tracepoints" to "faultable tracepoints", MAYSLEEP to MAYFAULT, and use might_fault() rather than might_sleep(), to properly convey that the tracepoints are meant to be able to take a page fault, which requires to be able to sleep *and* to hold the mmap_sem.
Changes since v2:
- Rename MAYFAULT to MAY_FAULT.
- Rebased on 6.5.5.
- Introduce MAY_EXIST tracepoint flag.
--- include/linux/tracepoint-defs.h | 14 ++++++ include/linux/tracepoint.h | 88 +++++++++++++++++++++++---------- include/trace/define_trace.h | 7 +++ include/trace/trace_events.h | 6 +++ init/Kconfig | 1 + kernel/trace/bpf_trace.c | 5 +- kernel/trace/trace_fprobe.c | 5 +- kernel/tracepoint.c | 58 ++++++++++++---------- 8 files changed, 129 insertions(+), 55 deletions(-)
include/linux/tracepoint-defs.h +14 -0
@@ -29,6 +29,19 @@ struct tracepoint_func {int prio; }; +/** +* enum tracepoint_flags - Tracepoint flags +* @TRACEPOINT_MAY_EXIST: Don't return an error if the tracepoint does not +* exist upon registration. +* @TRACEPOINT_MAY_FAULT: The tracepoint probe callback will be called with +* preemption enabled, and is allowed to take page +* faults. +*/ +enum tracepoint_flags { +TRACEPOINT_MAY_EXIST = (1 <<0 ), +TRACEPOINT_MAY_FAULT = (1 <<1 ), +}; +struct tracepoint {const char *name;/* Tracepoint name */ struct static_key key;
@@ -39,6 +52,7 @@ struct tracepoint {int (*regfunc)(void );void (*unregfunc)(void );struct tracepoint_func __rcu *funcs; +unsigned int flags; };#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS
include/linux/tracepoint.h +63 -25
214 lines not on this page · raw message
include/trace/define_trace.h +7 -0
@@ -41,6 +41,12 @@ assign, print, reg, unreg) \ DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args)) +#undef TRACE_EVENT_FN_MAY_FAULT +#define TRACE_EVENT_FN_MAY_FAULT (name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + DEFINE_TRACE_FN_FLAGS(name, reg, unreg, PARAMS(proto), \ + PARAMS(args), TRACEPOINT_MAY_FAULT) +#undef TRACE_EVENT_FN_COND#define TRACE_EVENT_FN_COND (name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \
@@ -106,6 +112,7 @@#undef TRACE_EVENT#undef TRACE_EVENT_FN +#undef TRACE_EVENT_FN_MAY_FAULT#undef TRACE_EVENT_FN_COND#undef TRACE_EVENT_CONDITION#undef TRACE_EVENT_NOP
include/trace/trace_events.h +6 -0
@@ -77,6 +77,12 @@TRACE_EVENT (name,PARAMS (proto),PARAMS (args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_MAY_FAULT +#define TRACE_EVENT_FN_MAY_FAULT (name, proto, args, tstruct, \ + assign, print, reg, unreg) \ + TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \ +#undef TRACE_EVENT_FN_COND#define TRACE_EVENT_FN_COND (name, proto, args, cond, tstruct, \ assign, print, reg, unreg) \
init/Kconfig +1 -0
@@ -1927,6 +1927,7 @@ config BINDGEN_VERSION_TEXT # config TRACEPOINTS bool + select TASKS_TRACE_RCU endmenu # General setup
kernel/trace/bpf_trace.c +3 -2
@@ -2327,8 +2327,9 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *if (prog->aux->max_tp_access > btp->writable_size)return -EINVAL ; -return tracepoint_probe_register_may_exist (tp, (void *)btp->bpf_func, - prog); +return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, + prog,TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_EXIST ); }int bpf_probe_register (struct bpf_raw_event_map *btp,struct bpf_prog *prog)
kernel/trace/trace_fprobe.c +3 -2
@@ -687,8 +687,9 @@ static int __register_trace_fprobe(struct trace_fprobe *tf)* At first, put __probestub_##TP function on the tracepoint * and put a fprobe on the stub function. */ - ret =tracepoint_probe_register_prio_may_exist (tpoint, - tpoint->probestub,NULL ,0 ); + ret =tracepoint_probe_register_prio_flags (tpoint, + tpoint->probestub,NULL ,0 , +TRACEPOINT_MAY_EXIST );if (ret <0 )return ret;return register_fprobe_ips (&tf->fp, &ip,1 );
kernel/tracepoint.c +32 -26
127 lines not on this page · raw message
Signature · 1 line
2.25.1
Steven Rostedt
maintainerRe: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 202
Quoted 8 lines · On Mon, 2 Oct 2023 16:25:27 -0400 Mathieu Desnoyers…
include/linux/tracepoint.h +2
-2@@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \return ; \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (mayfault) { \ +rcu_read_lock_trace (); \
I thought rcu_trace was for the case that a task can not voluntarily call schedule. If this tracepoint tries to read user space memory that isn't paged in, and faults, can't the faulting logic call schedule and break this requirement?
-- Steve
Quoted 7 lines
include/linux/tracepoint.h +4
-0+ }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \ \/* \ * For rcuidle callers, use srcu since sched-rcu \
Paul E. McKenney
maintainerRe: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 16 lines · Steven Rostedt
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 202
Quoted 8 lines · On Mon, 2 Oct 2023 16:25:27 -0400 Mathieu Desnoyers…
include/linux/tracepoint.h +2
-2@@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \return ; \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (mayfault) { \ +rcu_read_lock_trace (); \I thought
rcu_tracewas for the case that a task can not voluntarily call schedule. If this tracepoint tries to read user space memory that isn't paged in, and faults, can't the faulting logic call schedule and break this requirement?
Well, additional new uses of rcu_read_lock_trace() do bear close scrutiny, but RCU Tasks Trace readers are permitted to block for page faults. The BPF folks already use it for this purpose, so this should be OK. (If for some unknown-to-me reason it isn't, I am sure that Alexei, who is on CC, will not suffer in silence.)
One way of thinking of RCU Tasks Trace is as a form of SRCU with lightweight readers. Except that, unlike SRCU, there is only one global RCU Tasks Trace. This means that all RCU Tasks Trace users need to keep each other informed, because one users' unruly readers will affect all RCU Tasks Trace users.
But given that the BPF folks already have page faults in RCU Tasks Trace readers, this one should be OK.
Thanx, Paul
Quoted 10 lines
-- Steve
Quoted 7 lines
include/linux/tracepoint.h +4
-0+ }else { \ +/* keep srcu and sched-rcu usage consistent */ \ +preempt_disable_notrace (); \ + } \ \/* \ * For rcuidle callers, use srcu since sched-rcu \
Steven Rostedt
maintainerRe: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 24 lines · On Mon, 2 Oct 2023 17:14:39 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 16 lines · Steven Rostedt
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 202
Quoted 8 lines · On Mon, 2 Oct 2023 16:25:27 -0400 Mathieu Desnoyers…
include/linux/tracepoint.h +2
-2@@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \return ; \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (mayfault) { \ + rcu_read_lock_trace(); \I thought
rcu_tracewas for the case that a task can not voluntarily call schedule. If this tracepoint tries to read user space memory that isn't paged in, and faults, can't the faulting logic call schedule and break this requirement?Well, additional new uses of
rcu_read_lock_trace()do bear close scrutiny, but RCU Tasks Trace readers are permitted to block for page faults. The BPF folks already use it for this purpose, so this should be OK. (If for some unknown-to-me reason it isn't, I am sure that Alexei, who is on CC, will not suffer in silence.)
Quoted 8 lines
One way of thinking of RCU Tasks Trace is as a form of SRCU with lightweight readers. Except that, unlike SRCU, there is only one global RCU Tasks Trace. This means that all RCU Tasks Trace users need to keep each other informed, because one users' unruly readers will affect all RCU Tasks Trace users.
But given that the BPF folks already have page faults in RCU Tasks Trace readers, this one should be OK.
Then I think we should update the documentation.
From: Documentation/RCU/checklist.rst:
If the updater uses call_rcu_tasks() or synchronize_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from blocking. If the updater uses call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then the corresponding readers must use rcu_read_lock_trace() and rcu_read_unlock_trace(). If an updater uses call_rcu_tasks_rude() or synchronize_rcu_tasks_rude(), then the corresponding readers must use anything that disables preemption, for example, preempt_disable() and preempt_enable().
Because it's all one paragraph it's a bit confusing to know what uses what. Perhaps it should be broken up a bit more?
If the updater uses call_rcu_tasks() or synchronize_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from blocking.
If the updater uses call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then the corresponding readers must use rcu_read_lock_trace() and rcu_read_unlock_trace().
If an updater uses call_rcu_tasks_rude() or synchronize_rcu_tasks_rude(), then the corresponding readers must use anything that disables preemption, for example, preempt_disable() and preempt_enable().
That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that rcu_read_lock_trace() required tasks to not block.
-- Steve
Paul E. McKenney
maintainerRe: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 61 lines · Steven Rostedt
Quoted 24 lines · On Mon, 2 Oct 2023 17:14:39 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 16 lines · Steven Rostedt
include/linux/tracepoint.h static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p) · line 202
Quoted 8 lines · On Mon, 2 Oct 2023 16:25:27 -0400 Mathieu Desnoyers…
include/linux/tracepoint.h +2
-2@@ -202,8 +198,12 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \return ; \ \ -/* keep srcu and sched-rcu usage consistent */ \ -preempt_disable_notrace (); \ +if (mayfault) { \ + rcu_read_lock_trace(); \I thought
rcu_tracewas for the case that a task can not voluntarily call schedule. If this tracepoint tries to read user space memory that isn't paged in, and faults, can't the faulting logic call schedule and break this requirement?Well, additional new uses of
rcu_read_lock_trace()do bear close scrutiny, but RCU Tasks Trace readers are permitted to block for page faults. The BPF folks already use it for this purpose, so this should be OK. (If for some unknown-to-me reason it isn't, I am sure that Alexei, who is on CC, will not suffer in silence.)Quoted 8 lines
One way of thinking of RCU Tasks Trace is as a form of SRCU with lightweight readers. Except that, unlike SRCU, there is only one global RCU Tasks Trace. This means that all RCU Tasks Trace users need to keep each other informed, because one users' unruly readers will affect all RCU Tasks Trace users.
But given that the BPF folks already have page faults in RCU Tasks Trace readers, this one should be OK.
Then I think we should update the documentation.
From:
Documentation/RCU/checklist.rst:If the updater uses
call_rcu_tasks()orsynchronize_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from blocking. If the updater usescall_rcu_tasks_trace()orsynchronize_rcu_tasks_trace(), then the corresponding readers must usercu_read_lock_trace()andrcu_read_unlock_trace(). If an updater usescall_rcu_tasks_rude()orsynchronize_rcu_tasks_rude(), then the corresponding readers must use anything that disables preemption, for example,preempt_disable()andpreempt_enable().Because it's all one paragraph it's a bit confusing to know what uses what. Perhaps it should be broken up a bit more?
If the updater uses
call_rcu_tasks()orsynchronize_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from blocking.If the updater uses
call_rcu_tasks_trace()orsynchronize_rcu_tasks_trace(), then the corresponding readers must usercu_read_lock_trace()andrcu_read_unlock_trace().
Quoted 7 lines
If an updater uses
call_rcu_tasks_rude()orsynchronize_rcu_tasks_rude(), then the corresponding readers must use anything that disables preemption, for example,preempt_disable()andpreempt_enable().That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that
rcu_read_lock_trace()required tasks to not block.
That would work for me. Would you like to send a patch, or would you rather we made the adjustments?
Thanx, Paul
Steven Rostedt
maintainerRe: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 6 lines · On Tue, 3 Oct 2023 06:44:50 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 3 lines
That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that
rcu_read_lock_trace()required tasks to not block.That would work for me. Would you like to send a patch, or would you rather we made the adjustments?
Which ever.
-- Steve
Paul E. McKenney
maintainerRe: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 11 lines · Steven Rostedt
Quoted 6 lines · On Tue, 3 Oct 2023 06:44:50 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 3 lines
That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that
rcu_read_lock_trace()required tasks to not block.That would work for me. Would you like to send a patch, or would you rather we made the adjustments?
Which ever.
OK, how about like this?
Thanx, Paul
------------------------------------------------------------------------
commit 973eb79ec46c16f13bb5b47ad14d44a1f1c79dc9
Author: Paul E. McKenney <[email protected]> Date: Tue Oct 3 10:30:01 2023 -0700
doc: Clarify RCU Tasks reader/updater checklist
Currently, the reader/updater compatibility rules for the three RCU Tasks flavors are squished together in a single paragraph, which can result in confusion. This commit therefore splits them out into a list, clearly showing the distinction between these flavors.
Documentation/RCU/checklist.rst +16 -9
@@ -241,15 +241,22 @@ over a rather long period of time, but improvements are always welcome! srcu_struct. The rules for the expedited RCU grace-period-wait primitives are the same as for their non-expedited counterparts. - If the updater uses call_rcu_tasks() or synchronize_rcu_tasks(), - then the readers must refrain from executing voluntary - context switches, that is, from blocking. If the updater uses - call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then - the corresponding readers must use rcu_read_lock_trace() and - rcu_read_unlock_trace(). If an updater uses call_rcu_tasks_rude() - or synchronize_rcu_tasks_rude(), then the corresponding readers - must use anything that disables preemption, for example, - preempt_disable() and preempt_enable(). + Similarly, it is necssary to correctly use the RCU Tasks flavors: + + a. If the updater uses synchronize_rcu_tasks() or + call_rcu_tasks(), then the readers must refrain from + executing voluntary context switches, that is, from + blocking. + + b. If the updater uses call_rcu_tasks_trace() + or synchronize_rcu_tasks_trace(), then the + corresponding readers must use rcu_read_lock_trace() + and rcu_read_unlock_trace(). + + c. If an updater uses call_rcu_tasks_rude() or + synchronize_rcu_tasks_rude(), then the corresponding + readers must use anything that disables preemption, + for example, preempt_disable() and preempt_enable(). Mixing things up will result in confusion and broken kernels, and has even resulted in an exploitable security issue. Therefore,
Re: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 23 lines · Paul E. McKenney
Quoted 11 lines · Steven Rostedt
Quoted 6 lines · On Tue, 3 Oct 2023 06:44:50 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 3 lines
That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that
rcu_read_lock_trace()required tasks to not block.That would work for me. Would you like to send a patch, or would you rather we made the adjustments?
Which ever.
OK, how about like this?
Thanx, Paul
------------------------------------------------------------------------
commit
973eb79ec46c16f13bb5b47ad14d44a1f1c79dc9Author: Paul E. McKenney <[email protected]> Date: Tue Oct 3 10:30:01 2023 -0700
Quoted 7 lines
doc: Clarify RCU Tasks reader/updater checklist
Currently, the reader/updater compatibility rules for the three RCU Tasks flavors are squished together in a single paragraph, which can result in confusion. This commit therefore splits them out into a list, clearly showing the distinction between these flavors.
Thanks!
Mathieu
Quoted 39 lines
Reported-by Steven Rostedt <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>Documentation/RCU/checklist.rst +16
-9@@ -241,15 +241,22 @@ over a rather long period of time, but improvements are always welcome! srcu_struct. The rules for the expedited RCU grace-period-wait primitives are the same as for their non-expedited counterparts. - If the updater uses call_rcu_tasks() or synchronize_rcu_tasks(), - then the readers must refrain from executing voluntary - context switches, that is, from blocking. If the updater uses - call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then - the corresponding readers must use rcu_read_lock_trace() and - rcu_read_unlock_trace(). If an updater uses call_rcu_tasks_rude() - or synchronize_rcu_tasks_rude(), then the corresponding readers - must use anything that disables preemption, for example, - preempt_disable() and preempt_enable(). + Similarly, it is necssary to correctly use the RCU Tasks flavors: + + a. If the updater uses synchronize_rcu_tasks() or + call_rcu_tasks(), then the readers must refrain from + executing voluntary context switches, that is, from + blocking. + + b. If the updater uses call_rcu_tasks_trace() + or synchronize_rcu_tasks_trace(), then the + corresponding readers must use rcu_read_lock_trace() + and rcu_read_unlock_trace(). + + c. If an updater uses call_rcu_tasks_rude() or + synchronize_rcu_tasks_rude(), then the corresponding + readers must use anything that disables preemption, + for example, preempt_disable() and preempt_enable(). Mixing things up will result in confusion and broken kernels, and has even resulted in an exploitable security issue. Therefore,
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
In reply to Paul E. McKenneyQuoted 8 lines · On Tue, 3 Oct 2023 10:33:33 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Tue, 3 Oct 2023 06:44:50 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 3 lines
That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that
rcu_read_lock_trace()required tasks to not block.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
That would work for me. Would you like to send a patch, or would you rather we made the adjustments?
Which ever.
OK, how about like this?
-- Steve
Quoted 55 lines
Thanx, Paul
------------------------------------------------------------------------
commit
973eb79ec46c16f13bb5b47ad14d44a1f1c79dc9Author: Paul E. McKenney <[email protected]> Date: Tue Oct 3 10:30:01 2023 -0700doc: Clarify RCU Tasks reader/updater checklist
Currently, the reader/updater compatibility rules for the three RCU Tasks flavors are squished together in a single paragraph, which can result in confusion. This commit therefore splits them out into a list, clearly showing the distinction between these flavors.
Reported-by Steven Rostedt <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>Documentation/RCU/checklist.rst +16
-9@@ -241,15 +241,22 @@ over a rather long period of time, but improvements are always welcome! srcu_struct. The rules for the expedited RCU grace-period-wait primitives are the same as for their non-expedited counterparts. - If the updater uses call_rcu_tasks() or synchronize_rcu_tasks(), - then the readers must refrain from executing voluntary - context switches, that is, from blocking. If the updater uses - call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then - the corresponding readers must use rcu_read_lock_trace() and - rcu_read_unlock_trace(). If an updater uses call_rcu_tasks_rude() - or synchronize_rcu_tasks_rude(), then the corresponding readers - must use anything that disables preemption, for example, - preempt_disable() and preempt_enable(). + Similarly, it is necssary to correctly use the RCU Tasks flavors: + + a. If the updater uses synchronize_rcu_tasks() or + call_rcu_tasks(), then the readers must refrain from + executing voluntary context switches, that is, from + blocking. + + b. If the updater uses call_rcu_tasks_trace() + or synchronize_rcu_tasks_trace(), then the + corresponding readers must use rcu_read_lock_trace() + and rcu_read_unlock_trace(). + + c. If an updater uses call_rcu_tasks_rude() or + synchronize_rcu_tasks_rude(), then the corresponding + readers must use anything that disables preemption, + for example, preempt_disable() and preempt_enable(). Mixing things up will result in confusion and broken kernels, and has even resulted in an exploitable security issue. Therefore,
Re: [RFC PATCH v3 1/5] tracing: Introduce faultable tracepoints (v3)
Quoted 14 lines · Steven Rostedt
Quoted 11 lines · On Tue, 3 Oct 2023 10:33:33 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Tue, 3 Oct 2023 06:44:50 -0700 "Paul E. McKenney" <[email protected]> wrote:
Quoted 3 lines
That way it is clear what uses what, as I read the original paragraph a couple of times and could have sworn that
rcu_read_lock_trace()required tasks to not block.That would work for me. Would you like to send a patch, or would you rather we made the adjustments?
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Which ever.
OK, how about like this?
Reviewed-by Steven Rostedt (Google) <[email protected]>
Thank you both! Updated as shown below.
Thanx, Paul
------------------------------------------------------------------------
commit 4d2115e8919760c690e30f48cae2f017c1581546
Author: Paul E. McKenney <[email protected]> Date: Tue Oct 3 10:30:01 2023 -0700
doc: Clarify RCU Tasks reader/updater checklist
Currently, the reader/updater compatibility rules for the three RCU Tasks flavors are squished together in a single paragraph, which can result in confusion. This commit therefore splits them out into a list, clearly showing the distinction between these flavors.
Documentation/RCU/checklist.rst +16 -9
@@ -241,15 +241,22 @@ over a rather long period of time, but improvements are always welcome! srcu_struct. The rules for the expedited RCU grace-period-wait primitives are the same as for their non-expedited counterparts. - If the updater uses call_rcu_tasks() or synchronize_rcu_tasks(), - then the readers must refrain from executing voluntary - context switches, that is, from blocking. If the updater uses - call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then - the corresponding readers must use rcu_read_lock_trace() and - rcu_read_unlock_trace(). If an updater uses call_rcu_tasks_rude() - or synchronize_rcu_tasks_rude(), then the corresponding readers - must use anything that disables preemption, for example, - preempt_disable() and preempt_enable(). + Similarly, it is necssary to correctly use the RCU Tasks flavors: + + a. If the updater uses synchronize_rcu_tasks() or + call_rcu_tasks(), then the readers must refrain from + executing voluntary context switches, that is, from + blocking. + + b. If the updater uses call_rcu_tasks_trace() + or synchronize_rcu_tasks_trace(), then the + corresponding readers must use rcu_read_lock_trace() + and rcu_read_unlock_trace(). + + c. If an updater uses call_rcu_tasks_rude() or + synchronize_rcu_tasks_rude(), then the corresponding + readers must use anything that disables preemption, + for example, preempt_disable() and preempt_enable(). Mixing things up will result in confusion and broken kernels, and has even resulted in an exploitable security issue. Therefore,
Mathieu Desnoyers
reviewerRFC PATCH v3 2/5 tracing/ftrace: Add support for faultable tracepoints
In reply to Mathieu DesnoyersIn preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that ftrace can handle registering to such tracepoints by explicitly disabling preemption within the ftrace tracepoint probes to respect the current expectations within ftrace ring buffer code.
This change does not yet allow ftrace to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints.
--- include/trace/trace_events.h | 69 +++++++++++++++++++++++++++++++++--- kernel/trace/trace_events.c | 26 ++++++++++---- 2 files changed, 84 insertions(+), 11 deletions(-)
include/trace/trace_events.h +64 -5
@@ -45,6 +45,16 @@ PARAMS(print)); \DEFINE_EVENT (name, name,PARAMS (proto),PARAMS (args)); +#undef TRACE_EVENT_MAY_FAULT +#define TRACE_EVENT_MAY_FAULT (name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS_MAY_FAULT(name, \ + PARAMS(proto), \ + PARAMS(args), \ + PARAMS(tstruct), \ + PARAMS(assign), \ + PARAMS(print)); \ + DEFINE_EVENT(name, name, PARAMS(proto), PARAMS(args)); +#include "stages/stage1_struct_define.h" #undef DECLARE_EVENT_CLASS
@@ -57,6 +67,11 @@ \ static struct trace_event_class event_class_##name; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (name, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(name, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, name, proto, args) \ static struct trace_event_call __used \
@@ -80,7 +95,7 @@#undef TRACE_EVENT_FN_MAY_FAULT#define TRACE_EVENT_FN_MAY_FAULT (name, proto, args, tstruct, \ assign, print, reg, unreg) \ - TRACE_EVENT(name, PARAMS(proto), PARAMS(args), \ + TRACE_EVENT_MAY_FAULT(name, PARAMS(proto), PARAMS(args), \ PARAMS(tstruct), PARAMS(assign), PARAMS(print)) \#undef TRACE_EVENT_FN_COND
@@ -123,6 +138,11 @@ tstruct; \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, name, proto, args)
@@ -214,6 +234,11 @@ static struct trace_event_functions trace_event_type_funcs_##call = { \ .trace = trace_raw_output_##call, \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT_PRINT#define DEFINE_EVENT_PRINT (template, call, proto, args, print) \ static notrace enum print_line_t \
@@ -250,6 +275,11 @@ static struct trace_event_fields trace_event_fields_##call[] = { \ tstruct \ {} }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT_PRINT#define DEFINE_EVENT_PRINT (template, name, proto, args, print)
@@ -271,6 +301,11 @@ static inline notrace int trace_event_get_offsets_##call( \return __data_size; \ } +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#include TRACE_INCLUDE (TRACE_INCLUDE_FILE )/*
@@ -380,8 +415,8 @@ static inline notrace int trace_event_get_offsets_##call( \#include "stages/stage6_event_callback.h" -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \
@@ -392,8 +427,13 @@ trace_event_raw_event_##call(void *__data, proto) \ struct trace_event_raw_##call *entry; \ int __data_size; \ \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ if (trace_trigger_soft_disabled(trace_file)) \ - return; \ + goto end; \ \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \
@@ -401,14 +441,28 @@ trace_event_raw_event_##call(void *__data, proto) \ sizeof(*entry) + __data_size); \ \if (!entry) \ -return ; \ + goto end; \ \ tstruct \ \ { assign; } \ \trace_event_buffer_commit (&fbuffer); \ +end: \ +if ((tp_flags) &TRACEPOINT_MAY_FAULT ) \ +preempt_enable_notrace (); \ } + +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), TRACEPOINT_MAY_FAULT) +/* * The ftrace_test_probe is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the ftrace probe will
@@ -440,6 +494,11 @@ static struct trace_event_class __used __refdata event_class_##call = { \ _TRACE_PERF_INIT(call) \ }; +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print)) +#undef DEFINE_EVENT#define DEFINE_EVENT (template, call, proto, args) \ \
kernel/trace/trace_events.c +20 -6
@@ -532,9 +532,16 @@ int trace_event_reg(struct trace_event_call *call,WARN_ON (!(call->flags &TRACE_EVENT_FL_TRACEPOINT ));switch (type) {case TRACE_REG_REGISTER : -return tracepoint_probe_register (call->tp, - call->class->probe, - file); +if (call->tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (call->tp, + call->class->probe, + file, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register (call->tp, + call->class->probe, + file);case TRACE_REG_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->probe,
@@ -543,9 +550,16 @@ int trace_event_reg(struct trace_event_call *call,#ifdef CONFIG_PERF_EVENTS case TRACE_REG_PERF_REGISTER : -return tracepoint_probe_register (call->tp, - call->class->perf_probe, - call); +if (call->tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (call->tp, + call->class->perf_probe, + call, +TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register (call->tp, + call->class->perf_probe, + call);case TRACE_REG_PERF_UNREGISTER : tracepoint_probe_unregister(call->tp, call->class->perf_probe,
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerRFC PATCH v3 3/5 tracing/bpf-trace: add support for faultable tracepoints
In reply to Mathieu DesnoyersIn preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that bpf can handle registering to such tracepoints by explicitly disabling preemption within the bpf tracepoint probes to respect the current expectations within bpf tracing code.
This change does not yet allow bpf to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints.
--- include/trace/bpf_probe.h | 21 +++++++++++++++++---- kernel/trace/bpf_trace.c | 11 ++++++++--- 2 files changed, 25 insertions(+), 7 deletions(-)
include/trace/bpf_probe.h +17 -4
@@ -40,17 +40,30 @@/* tracepoints with more than 12 arguments will hit build error */ #define CAST_TO_U64 (...) CONCATENATE(__CAST, COUNT_ARGS(__VA_ARGS__))(__VA_ARGS__) -#define __BPF_DECLARE_TRACE (call, proto, args) \ +#define __BPF_DECLARE_TRACE (call, proto, args, tp_flags) \ static notrace void \ __bpf_trace_##call(void *__data, proto) \ { \ struct bpf_prog *prog = __data; \ + \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) { \ + might_fault(); \ + preempt_disable_notrace(); \ + } \ + \ CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(prog, CAST_TO_U64(args)); \ + \ + if ((tp_flags) & TRACEPOINT_MAY_FAULT) \ + preempt_enable_notrace(); \ }#undef DECLARE_EVENT_CLASS#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), TRACEPOINT_MAY_FAULT)/* * This part is compiled out, it is only here as a build time check
@@ -104,13 +117,13 @@ static inline void bpf_test_buffer_##call(void) \#undef DECLARE_TRACE#define DECLARE_TRACE (call, proto, args) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) \ __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), 0)#undef DECLARE_TRACE_WRITABLE#define DECLARE_TRACE_WRITABLE (call, proto, args, size) \ __CHECK_WRITABLE_BUF_SIZE(call, PARAMS(proto), PARAMS(args), size) \ - __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args)) \ + __BPF_DECLARE_TRACE(call, PARAMS(proto), PARAMS(args), 0) \ __DEFINE_EVENT(call, call, PARAMS(proto), PARAMS(args), size)#include TRACE_INCLUDE (TRACE_INCLUDE_FILE )
kernel/trace/bpf_trace.c +8 -3
@@ -2327,9 +2327,14 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *if (prog->aux->max_tp_access > btp->writable_size)return -EINVAL ; -return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, - prog,TRACEPOINT_DEFAULT_PRIO , -TRACEPOINT_MAY_EXIST ); +if (tp->flags &TRACEPOINT_MAY_FAULT ) +return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, + prog,TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_EXIST |TRACEPOINT_MAY_FAULT ); +else +return tracepoint_probe_register_prio_flags (tp, (void *)btp->bpf_func, + prog,TRACEPOINT_DEFAULT_PRIO , +TRACEPOINT_MAY_EXIST ); }int bpf_probe_register (struct bpf_raw_event_map *btp,struct bpf_prog *prog)
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerRFC PATCH v3 4/5 tracing/perf: add support for faultable tracepoints
In reply to Mathieu DesnoyersIn preparation for converting system call enter/exit instrumentation into faultable tracepoints, make sure that perf can handle registering to such tracepoints by explicitly disabling preemption within the perf tracepoint probes to respect the current expectations within perf ring buffer code.
This change does not yet allow perf to take page faults per se within its probe, but allows its existing probes to connect to faultable tracepoints.
--- include/trace/perf.h | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-)
include/trace/perf.h +23 -4
@@ -12,8 +12,8 @@#undef __perf_task#define __perf_task (t) (__task = (t)) -#undef DECLARE_EVENT_CLASS -#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ +#undef _DECLARE_EVENT_CLASS +#define _DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print, tp_flags) \ static notrace void \ perf_trace_##call(void *__data, proto) \ { \
@@ -28,13 +28,18 @@ perf_trace_##call(void *__data, proto) \int __data_size; \int rctx; \ \ +if ((tp_flags) &TRACEPOINT_MAY_FAULT ) { \ +might_fault (); \ +preempt_disable_notrace (); \ + } \ + \ __data_size = trace_event_get_offsets_##call(&__data_offsets, args); \ \ head = this_cpu_ptr(event_call->perf_events); \ if (!bpf_prog_array_valid(event_call) && \ __builtin_constant_p(!__task) && !__task && \ hlist_empty(head)) \ - return; \ + goto end; \ \ __entry_size = ALIGN(__data_size + sizeof(*entry) + sizeof(u32),\ sizeof(u64)); \
@@ -42,7 +47,7 @@ perf_trace_##call(void *__data, proto) \ \ entry =perf_trace_buf_alloc (__entry_size, &__regs, &rctx); \if (!entry) \ -return ; \ + goto end; \ \perf_fetch_caller_regs (__regs); \ \
@@ -53,8 +58,22 @@ perf_trace_##call(void *__data, proto) \perf_trace_run_bpf_submit (entry, __entry_size, rctx, \ event_call, __count, __regs, \ head, __task); \ +end: \ +if ((tp_flags) &TRACEPOINT_MAY_FAULT ) \ +preempt_enable_notrace (); \ } +#undef DECLARE_EVENT_CLASS +#define DECLARE_EVENT_CLASS (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), 0) + +#undef DECLARE_EVENT_CLASS_MAY_FAULT +#define DECLARE_EVENT_CLASS_MAY_FAULT (call, proto, args, tstruct, assign, print) \ + _DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args), \ + PARAMS(tstruct), PARAMS(assign), PARAMS(print), \ + TRACEPOINT_MAY_FAULT) +/* * This part is compiled out, it is only here as a build time check * to make sure that if the tracepoint handling changes, the
Signature · 1 line
2.25.1
Mathieu Desnoyers
reviewerRFC PATCH v3 5/5 tracing: convert sys_enter/exit to faultable tracepoints
In reply to Mathieu DesnoyersConvert the definition of the system call enter/exit tracepoints to faultable tracepoints now that all upstream tracers handle it.
This allows tracers to fault-in userspace system call arguments such as path strings within their probe callbacks.
--- include/trace/events/syscalls.h | 4 +- kernel/trace/trace_syscalls.c | 92 +++++++++++++++++++++++---------- 2 files changed, 68 insertions(+), 28 deletions(-)
include/trace/events/syscalls.h +2 -2
@@ -15,7 +15,7 @@#ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS -TRACE_EVENT_FN(sys_enter, +TRACE_EVENT_FN_MAY_FAULT(sys_enter, TP_PROTO(struct pt_regs *regs, long id),
@@ -41,7 +41,7 @@ TRACE_EVENT_FN(sys_enter, TRACE_EVENT_FLAGS(sys_enter, TRACE_EVENT_FL_CAP_ANY) -TRACE_EVENT_FN(sys_exit, +TRACE_EVENT_FN_MAY_FAULT(sys_exit, TP_PROTO(struct pt_regs *regs, long ret),
kernel/trace/trace_syscalls.c +66 -26
235 lines not on this page · raw message
Signature · 1 line
2.25.1