Skip to contentSkip to conversation

ftrace: make extra rcu_is_watching() validation check optional

PATCH In review Andrii Nakryiko 4 participants 6 messages Apr 18, 2024 – Apr 21, 2024

Subsystems (2)
lore
Share & tools
lore

Andrii Nakryiko

author maintainer
#0 lore raw

PATCH v4 1/2 ftrace: make extra rcu_is_watching() validation check optional

Introduce CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING config option to control whether ftrace low-level code performs additional rcu_is_watching()-based validation logic in an attempt to catch noinstr violations.

This check is expected to never be true and is mostly useful for low-level validation of ftrace subsystem invariants. For most users it should probably be kept disabled to eliminate unnecessary runtime overhead.

This improves BPF multi-kretprobe (relying on ftrace and rethook infrastructure) runtime throughput by 2%, according to BPF benchmarks ([0]).

[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/

---
 include/linux/trace_recursion.h |  2 +-
 kernel/trace/Kconfig            | 13 +++++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
include/linux/trace_recursion.h +1 -1
index d48cd92d2364..24ea8ac049b4 100644
@@ -135,7 +135,7 @@ extern void ftrace_record_recursion(unsigned long ip, unsigned long parent_ip);
 # define do_ftrace_record_recursion(ip, pip)	do { } while (0)
 #endif
 
-#ifdef CONFIG_ARCH_WANTS_NO_INSTR
+#ifdef CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING
 # define trace_warn_on_no_rcu(ip)					\
 	({								\
 		bool __ret = !rcu_is_watching();			\
kernel/trace/Kconfig +13 -0
index 61c541c36596..7aebd1b8f93e 100644
@@ -974,6 +974,19 @@ config FTRACE_RECORD_RECURSION_SIZE
 	  This file can be reset, but the limit can not change in
 	  size at runtime.
 
+config FTRACE_VALIDATE_RCU_IS_WATCHING
+	bool "Validate RCU is on during ftrace execution"
+	depends on FUNCTION_TRACER
+	depends on ARCH_WANTS_NO_INSTR
+	help
+	  All callbacks that attach to the function tracing have some sort of
+	  protection against recursion. This option is only to verify that
+	  ftrace (and other users of ftrace_test_recursion_trylock()) are not
+	  called outside of RCU, as if they are, it can cause a race. But it
+	  also has a noticeable overhead when enabled.
+
+	  If unsure, say N
+
 config RING_BUFFER_RECORD_RECURSION
 	bool "Record functions that recurse in the ring buffer"
 	depends on FTRACE_RECORD_RECURSION
Signature · 1 line
2.43.0
author maintainer
#1 lore raw

PATCH v4 2/2 rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()

Take into account CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING when validating that RCU is watching when trying to setup rethooko on a function entry.

One notable exception when we force rcu_is_watching() check is CONFIG_KPROBE_EVENTS_ON_NOTRACE=y case, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.

This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).

[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/

---
 kernel/trace/rethook.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/rethook.c +2 -0
index fa03094e9e69..a974605ad7a5 100644
@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
 	if (unlikely(!handler))
 		return NULL;
 
+#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
 	/*
 	 * This expects the caller will set up a rethook on a function entry.
 	 * When the function returns, the rethook will eventually be reclaimed
@@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
 	 */
 	if (unlikely(!rcu_is_watching()))
 		return NULL;
+#endif
 
 	return (struct rethook_node *)objpool_pop(&rh->pool);
 }
Signature · 1 line
2.43.0
Acked-by on v4 2/2
#2 lore raw

Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()

Quoted 8 lines · Andrii Nakryiko

Take into account CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING when validating that RCU is watching when trying to setup rethooko on a function entry.

One notable exception when we force rcu_is_watching() check is CONFIG_KPROBE_EVENTS_ON_NOTRACE=y case, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.

Quoted 7 lines

This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).

[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/

Quoted 27 lines
---
 kernel/trace/rethook.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/rethook.c +2 -0
index fa03094e9e69..a974605ad7a5 100644
@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
 	if (unlikely(!handler))
 		return NULL;
 
+#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
 	/*
 	 * This expects the caller will set up a rethook on a function entry.
 	 * When the function returns, the rethook will eventually be reclaimed
@@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
 	 */
 	if (unlikely(!rcu_is_watching()))
 		return NULL;
+#endif
 
 	return (struct rethook_node *)objpool_pop(&rh->pool);
 }
Signature · 1 line
2.43.0
maintainer Acked-by on v4 2/2
#3 lore raw

Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()

In reply to Andrii Nakryiko
Quoted 8 lines · On Thu, 18 Apr 2024 12:09:09 -0700 Andrii Nakryiko <[email protected]> wrote:

Take into account CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING when validating that RCU is watching when trying to setup rethooko on a function entry.

One notable exception when we force rcu_is_watching() check is CONFIG_KPROBE_EVENTS_ON_NOTRACE=y case, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.

Quoted 7 lines

This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).

[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/

Thanks for update! This looks good to me.

Thanks,

Quoted 27 lines
---
 kernel/trace/rethook.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/rethook.c +2 -0
index fa03094e9e69..a974605ad7a5 100644
@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
 	if (unlikely(!handler))
 		return NULL;
 
+#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
 	/*
 	 * This expects the caller will set up a rethook on a function entry.
 	 * When the function returns, the rethook will eventually be reclaimed
@@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
 	 */
 	if (unlikely(!rcu_is_watching()))
 		return NULL;
+#endif
 
 	return (struct rethook_node *)objpool_pop(&rh->pool);
 }
Signature · 1 line
2.43.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
#4 lore raw

Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()

Quoted 16 lines · Masami Hiramatsu
Quoted 12 lines · On Thu, 18 Apr 2024 12:09:09 -0700 Andrii Nakryiko <[email protected]> wrote:

Take into account CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING when validating that RCU is watching when trying to setup rethooko on a function entry.

One notable exception when we force rcu_is_watching() check is CONFIG_KPROBE_EVENTS_ON_NOTRACE=y case, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.

This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).

Quoted 6 lines
Quoted 3 lines

[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/

Thanks for update! This looks good to me.

Thanks, Masami! Will you take it through your tree, or you'd like to route it through bpf-next?

Quoted 36 lines

Thanks,

Quoted 25 lines
---
 kernel/trace/rethook.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/rethook.c +2 -1
index fa03094e9e69..a974605ad7a5 100644
@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
      if (unlikely(!handler))
              return NULL;
 
+#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
      /*
       * This expects the caller will set up a rethook on a function entry.
       * When the function returns, the rethook will eventually be reclaimed
@@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
       */
      if (unlikely(!rcu_is_watching()))
              return NULL;
+#endif
 
      return (struct rethook_node *)objpool_pop(&rh->pool);
 }
--
Quoted 2 lines

2.43.0

--
Masami Hiramatsu (Google) <[email protected]>

#5 lore raw

Re: [PATCH v4 2/2] rethook: honor CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING in rethook_try_get()

Quoted 19 lines · On Fri, 19 Apr 2024 10:59:09 -0700 Andrii Nakryiko <[email protected]>…
Quoted 18 lines · Masami Hiramatsu
Quoted 14 lines · On Thu, 18 Apr 2024 12:09:09 -0700 Andrii Nakryiko <[email protected]> wrote:

Take into account CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING when validating that RCU is watching when trying to setup rethooko on a function entry.

One notable exception when we force rcu_is_watching() check is CONFIG_KPROBE_EVENTS_ON_NOTRACE=y case, in which case kretprobes will use old-style int3-based workflow instead of relying on ftrace, making RCU watching check important to validate.

This further (in addition to improvements in the previous patch) improves BPF multi-kretprobe (which rely on rethook) runtime throughput by 2.3%, according to BPF benchmarks ([0]).

[0] https://lore.kernel.org/bpf/CAEf4BzauQ2WKMjZdc9s0rBWa01BYbgwHN6aNDXQSHYia47pQ-w@mail.gmail.com/

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Thanks for update! This looks good to me.

Thanks, Masami! Will you take it through your tree, or you'd like to route it through bpf-next?

OK, let me take it through linux-trace tree.

Thank you!

Quoted 37 lines
Quoted 37 lines

Thanks,

Quoted 25 lines
---
 kernel/trace/rethook.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/rethook.c +2 -1
index fa03094e9e69..a974605ad7a5 100644
@@ -166,6 +166,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
      if (unlikely(!handler))
              return NULL;
 
+#if defined(CONFIG_FTRACE_VALIDATE_RCU_IS_WATCHING) || defined(CONFIG_KPROBE_EVENTS_ON_NOTRACE)
      /*
       * This expects the caller will set up a rethook on a function entry.
       * When the function returns, the rethook will eventually be reclaimed
@@ -174,6 +175,7 @@ struct rethook_node *rethook_try_get(struct rethook *rh)
       */
      if (unlikely(!rcu_is_watching()))
              return NULL;
+#endif
 
      return (struct rethook_node *)objpool_pop(&rh->pool);
 }
--
Quoted 2 lines

2.43.0

--
Masami Hiramatsu (Google) <[email protected]>

Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>