Skip to contentSkip to conversation

tracing: generate RCU warnings even when tracepoints are disabled

PATCH In review Dave Hansen 3 participants 4 messages Aug 7, 2014 – Aug 8, 2014 Tracing

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tracing: generate RCU warnings even when tracepoints are disabled

From: Dave Hansen <[email protected]>

Dave Jones reported seeing a bug from one of my TLB tracepoints:

	http://lkml.kernel.org/r/[email protected]

I've been running these patches for months and never saw this. But, a big chunk of my testing, especially with all the debugging enabled, was in a vm where intel_idle doesn't work. On the systems where I was using intel_idle, I never had lockdep enabled and this tracepoint on at the same time.

This patch ensures that whenever we have lockdep available, we do _some_ RCU activity at the site of the tracepoint, despite whether the tracepoint's condition matches or even if the tracepoint itself is completely disabled. This is a bit of a hack, but it is pretty self-contained.

I confirmed that with this patch plus lockdep I get the same splat as Dave Jones did, but without enabling the tracepoint explicitly.

Cc Dave Jones <[email protected]>,
Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>

---

 b/include/linux/tracepoint.h |   11 +++++++++++
 1 file changed, 11 insertions(+)
include/linux/tracepoint.h renamed from include/linux/tracepoint.h~easier-rcu-splat +11 -0
diff -puN include/linux/tracepoint.h~easier-rcu-splat include/linux/tracepoint.h
@@ -157,6 +157,12 @@ extern void syscall_unregfunc(void);
  * Make sure the alignment of the structure in the __tracepoints section will
  * not add unwanted padding between the beginning of the section and the
  * structure. Force alignment to the same alignment as the section start.
+ *
+ * When lockdep is enabled, we make sure to always do the RCU portions of
+ * the tracepoint code, regardless of whether tracing is on or we match the
+ * condition.  This lets us find RCU issues triggered with tracepoints even
+ * when this tracepoint is off.  This code has no purpose other than poking
+ * RCU a bit.
  */
 #define __DECLARE_TRACE(name, proto, args, cond, data_proto, data_args) \
 	extern struct tracepoint __tracepoint_##name;			\
@@ -167,6 +173,11 @@ extern void syscall_unregfunc(void);
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args),			\
 				TP_CONDITION(cond),,);			\
+		if (IS_ENABLED(CONFIG_LOCKDEP)) {			\
+			rcu_read_lock_sched_notrace();			\
+			rcu_dereference_sched(__tracepoint_##name.funcs);\
+			rcu_read_unlock_sched_notrace();		\
+		}							\
 	}								\
 	__DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args),		\
 		PARAMS(cond), PARAMS(data_proto), PARAMS(data_args))	\

_

#1 lore raw
Quoted 23 lines · Dave Hansen

From: Dave Hansen <[email protected]>

Dave Jones reported seeing a bug from one of my TLB tracepoints:

	http://lkml.kernel.org/r/[email protected]

I've been running these patches for months and never saw this. But, a big chunk of my testing, especially with all the debugging enabled, was in a vm where intel_idle doesn't work. On the systems where I was using intel_idle, I never had lockdep enabled and this tracepoint on at the same time.

This patch ensures that whenever we have lockdep available, we do _some_ RCU activity at the site of the tracepoint, despite whether the tracepoint's condition matches or even if the tracepoint itself is completely disabled. This is a bit of a hack, but it is pretty self-contained.

I confirmed that with this patch plus lockdep I get the same splat as Dave Jones did, but without enabling the tracepoint explicitly.

Quoted 5 lines
Cc Dave Jones <[email protected]>,
Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>

Looks good to me, but I must defer to Steven on this one.

Thanx, Paul

Quoted 35 lines

---

 b/include/linux/tracepoint.h |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff -puN include/linux/tracepoint.h~easier-rcu-splat include/linux/tracepoint.h

include/linux/tracepoint.h renamed from include/linux/tracepoint.h~easier-rcu-splat +11 -0
@@ -157,6 +157,12 @@ extern void syscall_unregfunc(void);
  * Make sure the alignment of the structure in the __tracepoints section will
  * not add unwanted padding between the beginning of the section and the
  * structure. Force alignment to the same alignment as the section start.
+ *
+ * When lockdep is enabled, we make sure to always do the RCU portions of
+ * the tracepoint code, regardless of whether tracing is on or we match the
+ * condition.  This lets us find RCU issues triggered with tracepoints even
+ * when this tracepoint is off.  This code has no purpose other than poking
+ * RCU a bit.
  */
 #define __DECLARE_TRACE(name, proto, args, cond, data_proto, data_args) \
 	extern struct tracepoint __tracepoint_##name;			\
@@ -167,6 +173,11 @@ extern void syscall_unregfunc(void);
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args),			\
 				TP_CONDITION(cond),,);			\
+		if (IS_ENABLED(CONFIG_LOCKDEP)) {			\
+			rcu_read_lock_sched_notrace();			\
+			rcu_dereference_sched(__tracepoint_##name.funcs);\
+			rcu_read_unlock_sched_notrace();		\
+		}							\
 	}								\
 	__DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args),		\
 		PARAMS(cond), PARAMS(data_proto), PARAMS(data_args))	\

_

maintainer
#2 lore raw
Quoted 24 lines · On Thu, 7 Aug 2014 11:13:56 -0700 "Paul E. McKenney"…
Quoted 23 lines · Dave Hansen

From: Dave Hansen <[email protected]>

Dave Jones reported seeing a bug from one of my TLB tracepoints:

	http://lkml.kernel.org/r/[email protected]

I've been running these patches for months and never saw this. But, a big chunk of my testing, especially with all the debugging enabled, was in a vm where intel_idle doesn't work. On the systems where I was using intel_idle, I never had lockdep enabled and this tracepoint on at the same time.

This patch ensures that whenever we have lockdep available, we do _some_ RCU activity at the site of the tracepoint, despite whether the tracepoint's condition matches or even if the tracepoint itself is completely disabled. This is a bit of a hack, but it is pretty self-contained.

I confirmed that with this patch plus lockdep I get the same splat as Dave Jones did, but without enabling the tracepoint explicitly.

Quoted 8 lines
Quoted 5 lines
Cc Dave Jones <[email protected]>,
Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>

Looks good to me, but I must defer to Steven on this one.

Looks fine. I can add it to my 3.18 queue.

Paul, want to send me an "Acked-by"?

Thanks,

-- Steve

#3 lore raw
Quoted 33 lines · Steven Rostedt
Quoted 30 lines · On Thu, 7 Aug 2014 11:13:56 -0700 "Paul E. McKenney"…
Quoted 28 lines · Dave Hansen

From: Dave Hansen <[email protected]>

Dave Jones reported seeing a bug from one of my TLB tracepoints:

	http://lkml.kernel.org/r/[email protected]

I've been running these patches for months and never saw this. But, a big chunk of my testing, especially with all the debugging enabled, was in a vm where intel_idle doesn't work. On the systems where I was using intel_idle, I never had lockdep enabled and this tracepoint on at the same time.

This patch ensures that whenever we have lockdep available, we do _some_ RCU activity at the site of the tracepoint, despite whether the tracepoint's condition matches or even if the tracepoint itself is completely disabled. This is a bit of a hack, but it is pretty self-contained.

I confirmed that with this patch plus lockdep I get the same splat as Dave Jones did, but without enabling the tracepoint explicitly.

Cc Dave Jones <[email protected]>,
Cc Steven Rostedt <[email protected]>
Cc Ingo Molnar <[email protected]>
Quoted 5 lines
Quoted 1 line

Looks good to me, but I must defer to Steven on this one.

Looks fine. I can add it to my 3.18 queue.

Paul, want to send me an "Acked-by"?

Here you go:

Thanx, Paul