Skip to contentSkip to conversation

tracing: Add conditional to tracepoints

RFC PATCH Superseded Steven Rostedt 4 participants 7 messages Dec 2, 2010 – Dec 3, 2010

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing: Add conditional to tracepoints

This is an RFC that adds the TP_CONDITION() to the TRACE_EVENT() code.

There are certain cases that a tracepoint only makes sense if a specific condition is met. But because we do not want to dirty the fast path (the non tracing case) with if statements that are there only to avoid tracing, we just pass the condition variables to the tracepoint, and let the user filter them out if needed.

A perfect example is the tracepoint sched_wakeup. It traces all calls to try_to_wake_up() even if it fails to wake up. But if we add:

	if (success)
		trace_sched_wakeup(p);

We have that "if (success)" tested for every time we call try_to_wake_up(). Even when tracing is not (or never will be) enabled.

This patch set adds a variant TRACE_EVENT_CONDITIONAL()
(and DECLARE_EVENT_CLASS_CONDITIONAL()) that has a "cond" argument. This argument is encapsulated with "TP_CONDITIONAL()" which turns into:

	if (!cond)
		return;

This is placed inside the called tracepoint routine, and is only tested when the trace is enabled. Otherwise it is a nop as tracepoints normally are when disabled.

Note, another variant of this, is to move the test directly into the _DO_TRACE() macro, and not call any registered event callbacks. This would even speed it up faster when tracing is enabled. I did not do this orginially because I just thought of it now as I wrote this change log ;-) I'm posting this version now just in case people prefer it instead.

The following patches are in:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git

branch: rfc/trace-conditional

Steven Rostedt (2):
tracing: Add TRACE_EVENT_CONDITIONAL()
tracing: Only trace sched_wakeup if it actually work something up

----

 include/linux/tracepoint.h   |    3 ++
 include/trace/define_trace.h |    6 +++
 include/trace/events/sched.h |   12 +++---
 include/trace/ftrace.h       |   70 +++++++++++++++++++++++++++++++----------
 4 files changed, 68 insertions(+), 23 deletions(-)
author maintainer
#1 lore raw

RFC PATCH 1/2 tracing: Add TRACE_EVENT_CONDITIONAL()

From: Steven Rostedt <[email protected]>

There are instances in the kernel that we only want to trace a tracepoint when a certain condition is set. But we do not want to test for that condition in the core kernel.
If we test for that condition before calling the tracepoin, then we will be performing that test even when tracing is not enabled. This is 99.99% of the time.

We currently can just filter out on that condition, but that happens after we write to the trace buffer. We just wasted time writing to the ring buffer for an event we never cared about.

This patch adds:

TRACE_EVENT_CONDITION() and DECLARE_TRACE_CLASS_CONDITION()

These have a new TP_CONDITION() argument that comes right after the TP_ARGS(). This condition can use the parameters of the TRACE_EVENT() to determine if the tracepoint should be traced or not. The TP_CONDITION() will be placed in a if (cond) trace;

For example, for the tracepoint sched_wakeup, it is useless to trace an wakeup event where the caller never actually work anything up (success = 0). So adding:

TP_CONDITION(success),

which uses the "success" parameter of the wakeup tracepoint will have it only trace when we have successfully woken up a task.

Cc Arjan van de Ven <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/tracepoint.h   |    3 ++
 include/trace/define_trace.h |    6 +++
 include/trace/events/sched.h |    2 +-
 include/trace/ftrace.h       |   70 +++++++++++++++++++++++++++++++----------
 4 files changed, 63 insertions(+), 18 deletions(-)
include/linux/tracepoint.h +3 -0
index 5a6074f..380c807 100644
@@ -344,12 +344,15 @@ do_trace:								\
  * TRACE_EVENT_FN to perform any (un)registration work.
  */
 
+#define DECLARE_EVENT_CLASS_CONDITION(name, proto, args, cond, tstruct, assign, print)
 #define DECLARE_EVENT_CLASS(name, proto, args, tstruct, assign, print)
 #define DEFINE_EVENT(template, name, proto, args)		\
 	DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
 #define DEFINE_EVENT_PRINT(template, name, proto, args, print)	\
 	DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
 
+#define TRACE_EVENT_CONDITION(name, proto, args, cond, struct, assign, print) \
+	DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
 #define TRACE_EVENT(name, proto, args, struct, assign, print)	\
 	DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
 #define TRACE_EVENT_FN(name, proto, args, struct,		\
include/trace/define_trace.h +6 -0
index 1dfab54..89cf844 100644
@@ -26,6 +26,10 @@
 #define TRACE_EVENT(name, proto, args, tstruct, assign, print)	\
 	DEFINE_TRACE(name)
 
+#undef TRACE_EVENT_CONDITION
+#define TRACE_EVENT_CONDITION(name, proto, args, cond, tstruct, assign, print) \
+	DEFINE_TRACE(name)
+
 #undef TRACE_EVENT_FN
 #define TRACE_EVENT_FN(name, proto, args, tstruct,		\
 		assign, print, reg, unreg)			\
@@ -75,7 +79,9 @@
 
 #undef TRACE_EVENT
 #undef TRACE_EVENT_FN
+#undef TRACE_EVENT_CONDITION
 #undef DECLARE_EVENT_CLASS
+#undef DECLARE_EVENT_CLASS_CONDITION
 #undef DEFINE_EVENT
 #undef DEFINE_EVENT_PRINT
 #undef TRACE_HEADER_MULTI_READ
include/trace/events/sched.h +1 -1
index f633478..67456f0 100644
@@ -10,7 +10,7 @@
 /*
  * Tracepoint for calling kthread_stop, performed to end a kthread:
  */
-TRACE_EVENT(sched_kthread_stop,
+TRACE_EVENT_CONDITION(sched_kthread_stop,
 
 	TP_PROTO(struct task_struct *t),
 
include/trace/ftrace.h +53 -17

143 lines not on this page · raw message

Signature · 1 line
1.7.2.3
#2 lore raw

Re: [RFC][PATCH 1/2] [PATCH 1/2] tracing: Add TRACE_EVENT_CONDITIONAL()

Quoted 17 lines · Steven Rostedt

From: Steven Rostedt<[email protected]>

There are instances in the kernel that we only want to trace a tracepoint when a certain condition is set. But we do not want to test for that condition in the core kernel.
If we test for that condition before calling the tracepoin, then we will be performing that test even when tracing is not enabled. This is 99.99% of the time.

We currently can just filter out on that condition, but that happens after we write to the trace buffer. We just wasted time writing to the ring buffer for an event we never cared about.

This patch adds:

TRACE_EVENT_CONDITION() and DECLARE_TRACE_CLASS_CONDITION()

Quoted 8 lines

These have a new TP_CONDITION() argument that comes right after the TP_ARGS(). This condition can use the parameters of the TRACE_EVENT() to determine if the tracepoint should be traced or not. The TP_CONDITION() will be placed in a if (cond) trace;

For example, for the tracepoint sched_wakeup, it is useless to trace an wakeup event where the caller never actually work anything up (success = 0). So adding:

                      ^^^
                   s/=/==/

As much as I hate to be a pedant, I would suggest changing this change log for the sake of clarity.

David Daney

author maintainer
#3 lore raw

Re: [RFC][PATCH 1/2] [PATCH 1/2] tracing: Add TRACE_EVENT_CONDITIONAL()

Quoted 10 lines · David Daney
Quoted 3 lines · Steven Rostedt

For example, for the tracepoint sched_wakeup, it is useless to trace an wakeup event where the caller never actually work anything up (success = 0). So adding:

                       ^^^
                    s/=/==/

As much as I hate to be a pedant, I would suggest changing this change log for the sake of clarity.

Eek! Yeah I'll fix that, along with s/work anything/woke anything/.

Thanks,

-- Steve

author maintainer
#4 lore raw

RFC PATCH 2/2 tracing: Only trace sched_wakeup if it actually work something up

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Currently the tracepoint sched_wakeup traces the wakeup event even if the wakeup failed to wake anything up. This is quite stupid but it happens because we did not want to add a conditional to the core kernel code that would just slow down the wakeup events.

This patch changes the wakeup tracepoints to use the

DECLARE_EVENT_CLASS_CONDITIONAL()

to test the "success" parameter and will only trace the event if the wakeup was successfull.

The success field in the tracepoint is removed since it is no longer needed.

---
 include/trace/events/sched.h |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
include/trace/events/sched.h +5 -5
index 67456f0..0807a25 100644
@@ -52,17 +52,18 @@ TRACE_EVENT(sched_kthread_stop_ret,
 /*
  * Tracepoint for waking up a task:
  */
-DECLARE_EVENT_CLASS(sched_wakeup_template,
+DECLARE_EVENT_CLASS_CONDITION(sched_wakeup_template,
 
 	TP_PROTO(struct task_struct *p, int success),
 
 	TP_ARGS(p, success),
 
+	TP_CONDITION(success),
+
 	TP_STRUCT__entry(
 		__array(	char,	comm,	TASK_COMM_LEN	)
 		__field(	pid_t,	pid			)
 		__field(	int,	prio			)
-		__field(	int,	success			)
 		__field(	int,	target_cpu		)
 	),
 
@@ -70,13 +71,12 @@ DECLARE_EVENT_CLASS(sched_wakeup_template,
 		memcpy(__entry->comm, p->comm, TASK_COMM_LEN);
 		__entry->pid		= p->pid;
 		__entry->prio		= p->prio;
-		__entry->success	= success;
 		__entry->target_cpu	= task_cpu(p);
 	),
 
-	TP_printk("comm=%s pid=%d prio=%d success=%d target_cpu=%03d",
+	TP_printk("comm=%s pid=%d prio=%d target_cpu=%03d",
 		  __entry->comm, __entry->pid, __entry->prio,
-		  __entry->success, __entry->target_cpu)
+		  __entry->target_cpu)
 );
 
 DEFINE_EVENT(sched_wakeup_template, sched_wakeup,
Signature · 1 line
1.7.2.3
#5 lore raw
In reply to Steven Rostedt
Quoted 25 lines · Steven Rostedt

This is an RFC that adds the TP_CONDITION() to the TRACE_EVENT() code.

There are certain cases that a tracepoint only makes sense if a specific condition is met. But because we do not want to dirty the fast path (the non tracing case) with if statements that are there only to avoid tracing, we just pass the condition variables to the tracepoint, and let the user filter them out if needed.

A perfect example is the tracepoint sched_wakeup. It traces all calls to try_to_wake_up() even if it fails to wake up. But if we add:

	if (success)
		trace_sched_wakeup(p);

We have that "if (success)" tested for every time we call try_to_wake_up(). Even when tracing is not (or never will be) enabled.

This patch set adds a variant TRACE_EVENT_CONDITIONAL()
(and DECLARE_EVENT_CLASS_CONDITIONAL()) that has a "cond" argument. This argument is encapsulated with "TP_CONDITIONAL()" which turns into:

	if (!cond)
		return;
Quoted 8 lines

This is placed inside the called tracepoint routine, and is only tested when the trace is enabled. Otherwise it is a nop as tracepoints normally are when disabled.

Note, another variant of this, is to move the test directly into the _DO_TRACE() macro, and not call any registered event callbacks. This would even speed it up faster when tracing is enabled. I did not do this orginially because I just thought of it now as I wrote this change log ;-)

Hehe :)

Yeah indeed. And that looks fairly possible.

Quoted 19 lines

I'm posting this version now just in case people prefer it instead.

The following patches are in:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git

branch: rfc/trace-conditional

Steven Rostedt (2):
tracing: Add TRACE_EVENT_CONDITIONAL()
tracing: Only trace sched_wakeup if it actually work something up

----

 include/linux/tracepoint.h   |    3 ++
 include/trace/define_trace.h |    6 +++
 include/trace/events/sched.h |   12 +++---
 include/trace/ftrace.h       |   70 +++++++++++++++++++++++++++++++----------
 4 files changed, 68 insertions(+), 23 deletions(-)
#6 lore raw
Quoted 10 lines · * Frederic Weisbecker ([email protected]) wrote:
Quoted 5 lines · Steven Rostedt

Note, another variant of this, is to move the test directly into the _DO_TRACE() macro, and not call any registered event callbacks. This would even speed it up faster when tracing is enabled. I did not do this orginially because I just thought of it now as I wrote this change log ;-)

Hehe :)

Yeah indeed. And that looks fairly possible.

I'd very much prefer if the test is performed before the call, within the block that contains the stack setup and the tracepoint function call. Having an utterly low performance impact for the events that are filtered out is very important for my client's use-cases. Also, moving it outside of the tracepoint probe function allows us to filter only once for all the registered handlers.

All it would require is to skip over the function call rather than doing a "return".

For the rest, it looks nice. :-)

Thanks,

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com