ftrace: make extra rcu_is_watching() validation check optional
Andrii Nakryiko
maintainerPATCH v4 1/2 ftrace: make extra rcu_is_watching() validation check optional
Introduce CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING config option to control whether ftrace low-level code performs additional rcu_is_watching()-based validation logic in an attempt to catch noinstr violations.
This check is expected to never be true and is mostly useful for low-level validation of ftrace subsystem invariants. For most users it should probably be kept disabled to eliminate unnecessary runtime overhead.
This improves BPF multi-kretprobe (relying on ftrace and rethook infrastructure) runtime throughput by 2%, according to BPF benchmarks ([0]).
[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/
--- include/linux/trace_recursion.h | 2 +- kernel/trace/Kconfig | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
include/linux/trace_recursion.h +1 -1
@@ -135,7 +135,7 @@ extern void ftrace_record_recursion(unsigned long ip, unsigned long parent_ip);# define do_ftrace_record_recursion (ip, pip) do { } while (0)#endif -#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#ifdef CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING # define trace_warn_on_no_rcu (ip) \ ({ \ bool __ret = !rcu_is_watching(); \
kernel/trace/Kconfig +13 -0
@@ -974,6 +974,19 @@ config FTRACE_RECORD_RECURSION_SIZE This file can be reset, but the limit can not change in size at runtime. +config FTRACE_VALIDATE_RCU_IS_WATCHING + bool "Validate RCU is on during ftrace execution" + depends on FUNCTION_TRACER + depends on ARCH_WANTS_NO_INSTR + help + All callbacks that attach to the function tracing have some sort of + protection against recursion. This option is only to verify that + ftrace (and other users of ftrace_test_recursion_trylock()) are not + called outside of RCU, as if they are, it can cause a race. But it + also has a noticeable overhead when enabled. + + If unsure, say N + config RING_BUFFER_RECORD_RECURSION bool "Record functions that recurse in the ring buffer" depends on FTRACE_RECORD_RECURSION
Signature · 1 line
2.43.0
Andrii Nakryiko
maintainerPATCH v4 2/2 rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()
Take into account CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING when validating that RCU is watching when trying to setup rethooko on a function entry.
One notable exception when we force rcu_is_watching() check is CONFIG_KPROBE_EVENTS_ON_NOTRACE=y case, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.
This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).
[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/
--- kernel/trace/rethook.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/trace/rethook.c +2 -0
@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)if (unlikely (!handler))return NULL ; +#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING ) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE )/* * This expects the caller will set up a rethook on a function entry. * When the function returns, the rethook will eventually be reclaimed
@@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)*/ if (unlikely (!rcu_is_watching ()))return NULL ; +#endif return (struct rethook_node *)objpool_pop (&rh->pool); }
Signature · 1 line
2.43.0
Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()
Quoted 8 lines · Andrii Nakryiko
Take into account
CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHINGwhen validating that RCU is watching when trying to setup rethooko on a function entry.One notable exception when we force
rcu_is_watching()check isCONFIG_KPROBE_EVENTS_ON_NOTRACE=ycase, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.
Quoted 7 lines
This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).
[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/
Signed-off-by Andrii Nakryiko <[email protected]>
Quoted 27 lines
--- kernel/trace/rethook.c | 2 ++ 1 file changed, 2 insertions(+)kernel/trace/rethook.c +2
-0@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)if (unlikely (!handler))return NULL ; +#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING ) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE )/* * This expects the caller will set up a rethook on a function entry. * When the function returns, the rethook will eventually be reclaimed @@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)*/ if (unlikely (!rcu_is_watching ()))return NULL ; +#endif return (struct rethook_node *)objpool_pop (&rh->pool); }Signature · 1 line
2.43.0
Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()
In reply to Andrii NakryikoQuoted 8 lines · On Thu, 18 Apr 2024 12:09:09 -0700 Andrii Nakryiko <[email protected]> wrote:
Take into account
CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHINGwhen validating that RCU is watching when trying to setup rethooko on a function entry.One notable exception when we force
rcu_is_watching()check isCONFIG_KPROBE_EVENTS_ON_NOTRACE=ycase, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.
Quoted 7 lines
This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).
[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/
Signed-off-by Andrii Nakryiko <[email protected]>
Thanks for update! This looks good to me.
Thanks,
Quoted 27 lines
--- kernel/trace/rethook.c | 2 ++ 1 file changed, 2 insertions(+)kernel/trace/rethook.c +2
-0@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)if (unlikely (!handler))return NULL ; +#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING ) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE )/* * This expects the caller will set up a rethook on a function entry. * When the function returns, the rethook will eventually be reclaimed @@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)*/ if (unlikely (!rcu_is_watching ()))return NULL ; +#endif return (struct rethook_node *)objpool_pop (&rh->pool); }Signature · 1 line
2.43.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()
Quoted 16 lines · Masami Hiramatsu
Quoted 12 lines · On Thu, 18 Apr 2024 12:09:09 -0700 Andrii Nakryiko <[email protected]> wrote:
Take into account
CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHINGwhen validating that RCU is watching when trying to setup rethooko on a function entry.One notable exception when we force
rcu_is_watching()check isCONFIG_KPROBE_EVENTS_ON_NOTRACE=ycase, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).
Quoted 6 lines
Quoted 3 lines
[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/
Signed-off-by Andrii Nakryiko <[email protected]>Thanks for update! This looks good to me.
Thanks, Masami! Will you take it through your tree, or you'd like to route it through bpf-next?
Quoted 36 lines
Thanks,
Quoted 25 lines
--- kernel/trace/rethook.c | 2 ++ 1 file changed, 2 insertions(+)kernel/trace/rethook.c +2
-1@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)if (unlikely (!handler))return NULL ; +#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING ) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE )/* * This expects the caller will set up a rethook on a function entry. * When the function returns, the rethook will eventually be reclaimed @@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)*/ if (unlikely (!rcu_is_watching ()))return NULL ; +#endif return (struct rethook_node *)objpool_pop (&rh->pool); } --Quoted 2 lines
2.43.0
--
Masami Hiramatsu (Google) <[email protected]>
Masami Hiramatsu (Google)
maintainerRe: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()
Quoted 19 lines · On Fri, 19 Apr 2024 10:59:09 -0700 Andrii Nakryiko <[email protected]>…
Quoted 18 lines · Masami Hiramatsu
Quoted 14 lines · On Thu, 18 Apr 2024 12:09:09 -0700 Andrii Nakryiko <[email protected]> wrote:
Take into account
CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHINGwhen validating that RCU is watching when trying to setup rethooko on a function entry.One notable exception when we force
rcu_is_watching()check isCONFIG_KPROBE_EVENTS_ON_NOTRACE=ycase, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).
[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Andrii Nakryiko <[email protected]>Thanks for update! This looks good to me.
Thanks, Masami! Will you take it through your tree, or you'd like to route it through bpf-next?
OK, let me take it through linux-trace tree.
Thank you!
Quoted 37 lines
Quoted 37 lines
Thanks,
Quoted 25 lines
--- kernel/trace/rethook.c | 2 ++ 1 file changed, 2 insertions(+)kernel/trace/rethook.c +2
-1@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)if (unlikely (!handler))return NULL ; +#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING ) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE )/* * This expects the caller will set up a rethook on a function entry. * When the function returns, the rethook will eventually be reclaimed @@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)*/ if (unlikely (!rcu_is_watching ()))return NULL ; +#endif return (struct rethook_node *)objpool_pop (&rh->pool); } --Quoted 2 lines
2.43.0
--
Masami Hiramatsu (Google) <[email protected]>
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>