Skip to contentSkip to conversation

ftrace: Add function names to dangling } in function graph tracer

GIT PULL Steven Rostedt 3 participants 7 messages Feb 27, 2010 – Feb 28, 2010 Tracing

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

Ingo,

Please pull the latest tip/tracing/core tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/core

Steven Rostedt (1):
ftrace: Add function names to dangling } in function graph tracer

----

 kernel/trace/trace_functions_graph.c |   52 ++++++++++++++++++++++++++++------
 1 files changed, 43 insertions(+), 9 deletions(-)
---------------------------
commit f1c7f517a5dc23bce07efa5ed55e2c074ed9d4ba
Author: Steven Rostedt <[email protected]>
Date:   Fri Feb 26 17:08:16 2010 -0500

ftrace: Add function names to dangling } in function graph tracer

The function graph tracer is currently the most invasive tracer in the ftrace family. It can easily overflow the buffer even with 10megs per CPU. This means that events can often be lost.

On start up, or after events are lost, if the function return is recorded but the function enter was lost, all we get to see is the exiting '}'.

Here is how a typical trace output starts:

[tracing] cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) + 91.897 us | }
0) ! 567.961 us | }
0) <========== |
0) ! 579.083 us | _raw_spin_lock_irqsave();
0) 4.694 us | _raw_spin_unlock_irqrestore();
0) ! 594.862 us | }
0) ! 603.361 us | }
0) ! 613.574 us | }
0) ! 623.554 us | }
0) 3.653 us | fget_light();
0) | sock_poll() {

There are a series of '}' with no matching "func() {". There's no information to what functions these ending brackets belong to.

This patch adds a stack on the per cpu structure used in outputting the function graph tracer to keep track of what function was outputted. Then on a function exit event, it checks the depth to see if the function exit has a matching entry event. If it does, then it only prints the '}', otherwise it adds the function name after the '}'.

This allows function exit events to show what function they belong to at trace output startup, when the entry was lost due to ring buffer overflow, or even after a new task is scheduled in.

Here is what the above trace will look like after this patch:

[tracing] cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) + 91.897 us | } (irq_exit)
0) ! 567.961 us | } (smp_apic_timer_interrupt)
0) <========== |
0) ! 579.083 us | _raw_spin_lock_irqsave();
0) 4.694 us | _raw_spin_unlock_irqrestore();
0) ! 594.862 us | } (add_wait_queue)
0) ! 603.361 us | } (__pollwait)
0) ! 613.574 us | } (tcp_poll)
0) ! 623.554 us | } (sock_poll)
0) 3.653 us | fget_light();
0) | sock_poll() {

kernel/trace/trace_functions_graph.c +43 -9
index 112561d..e998a82 100644
@@ -18,6 +18,7 @@ struct fgraph_cpu_data {
 	pid_t		last_pid;
 	int		depth;
 	int		ignore;
+	unsigned long	enter_funcs[FTRACE_RETFUNC_DEPTH];
 };
 
 struct fgraph_data {
@@ -670,15 +671,21 @@ print_graph_entry_leaf(struct trace_iterator *iter,
 	duration = graph_ret->rettime - graph_ret->calltime;
 
 	if (data) {
+		struct fgraph_cpu_data *cpu_data;
 		int cpu = iter->cpu;
-		int *depth = &(per_cpu_ptr(data->cpu_data, cpu)->depth);
+
+		cpu_data = per_cpu_ptr(data->cpu_data, cpu);
 
 		/*
 		 * Comments display at + 1 to depth. Since
 		 * this is a leaf function, keep the comments
 		 * equal to this depth.
 		 */
-		*depth = call->depth - 1;
+		cpu_data->depth = call->depth - 1;
+
+		/* No need to keep this function around for this depth */
+		if (call->depth < FTRACE_RETFUNC_DEPTH)
+			cpu_data->enter_funcs[call->depth] = 0;
 	}
 
 	/* Overhead */
@@ -718,10 +725,15 @@ print_graph_entry_nested(struct trace_iterator *iter,
 	int i;
 
 	if (data) {
+		struct fgraph_cpu_data *cpu_data;
 		int cpu = iter->cpu;
-		int *depth = &(per_cpu_ptr(data->cpu_data, cpu)->depth);
 
-		*depth = call->depth;
+		cpu_data = per_cpu_ptr(data->cpu_data, cpu);
+		cpu_data->depth = call->depth;
+
+		/* Save this function pointer to see if the exit matches */
+		if (call->depth < FTRACE_RETFUNC_DEPTH)
+			cpu_data->enter_funcs[call->depth] = call->func;
 	}
 
 	/* No overhead */
@@ -851,18 +863,28 @@ print_graph_return(struct ftrace_graph_ret *trace, struct trace_seq *s,
 	struct fgraph_data *data = iter->private;
 	pid_t pid = ent->pid;
 	int cpu = iter->cpu;
+	int func_match = 1;
 	int ret;
 	int i;
 
 	if (data) {
-		int *depth = &(per_cpu_ptr(data->cpu_data, cpu)->depth);
+		struct fgraph_cpu_data *cpu_data;
+		int cpu = iter->cpu;
+
+		cpu_data = per_cpu_ptr(data->cpu_data, cpu);
 
 		/*
 		 * Comments display at + 1 to depth. This is the
 		 * return from a function, we now want the comments
 		 * to display at the same level of the bracket.
 		 */
-		*depth = trace->depth - 1;
+		cpu_data->depth = trace->depth - 1;
+
+		if (trace->depth < FTRACE_RETFUNC_DEPTH) {
+			if (cpu_data->enter_funcs[trace->depth] != trace->func)
+				func_match = 0;
+			cpu_data->enter_funcs[trace->depth] = 0;
+		}
 	}
 
 	if (print_graph_prologue(iter, s, 0, 0))
@@ -887,9 +909,21 @@ print_graph_return(struct ftrace_graph_ret *trace, struct trace_seq *s,
 			return TRACE_TYPE_PARTIAL_LINE;
 	}
 
-	ret = trace_seq_printf(s, "}\n");
-	if (!ret)
-		return TRACE_TYPE_PARTIAL_LINE;
+	/*
+	 * If the return function does not have a matching entry,
+	 * then the entry was lost. Instead of just printing
+	 * the '}' and letting the user guess what function this
+	 * belongs to, write out the function name.
+	 */
+	if (func_match) {
+		ret = trace_seq_printf(s, "}\n");
+		if (!ret)
+			return TRACE_TYPE_PARTIAL_LINE;
+	} else {
+		ret = trace_seq_printf(s, "} (%ps)\n", (void *)trace->func);
+		if (!ret)
+			return TRACE_TYPE_PARTIAL_LINE;
+	}
 
 	/* Overrun */
 	if (tracer_flags.val & TRACE_GRAPH_PRINT_OVERRUN) {
#1 lore raw
Quoted 9 lines · Steven Rostedt

Ingo,

Please pull the latest tip/tracing/core tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/core

Quoted 6 lines

Steven Rostedt (1):
ftrace: Add function names to dangling } in function graph tracer

----

 kernel/trace/trace_functions_graph.c |   52 ++++++++++++++++++++++++++++------
 1 files changed, 43 insertions(+), 9 deletions(-)

Pulled, thanks Steve!

Quoted 8 lines

[tracing] cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) + 91.897 us | } (irq_exit)
0) ! 567.961 us | } (smp_apic_timer_interrupt)
0) <========== |

Quoted 8 lines

0) ! 579.083 us | _raw_spin_lock_irqsave();
0) 4.694 us | _raw_spin_unlock_irqrestore();
0) ! 594.862 us | } (add_wait_queue)
0) ! 603.361 us | } (__pollwait)
0) ! 613.574 us | } (tcp_poll)
0) ! 623.554 us | } (sock_poll)
0) 3.653 us | fget_light();
0) | sock_poll() {

Neat! I think it would read even more C-ish if it was something non-syntactic, like:

Quoted 4 lines
      0) ! 594.862 us  |              } /* add_wait_queue */
      0) ! 603.361 us  |            } /* __pollwait */
      0) ! 613.574 us  |          } /* tcp_poll */
      0) ! 623.554 us  |        } /* sock_poll */

hm?

Ingo

#2 lore raw
Quoted 20 lines · Ingo Molnar
Quoted 8 lines

[tracing] cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) + 91.897 us | } (irq_exit)
0) ! 567.961 us | } (smp_apic_timer_interrupt)
0) <========== |

Quoted 8 lines

0) ! 579.083 us | _raw_spin_lock_irqsave();
0) 4.694 us | _raw_spin_unlock_irqrestore();
0) ! 594.862 us | } (add_wait_queue)
0) ! 603.361 us | } (__pollwait)
0) ! 613.574 us | } (tcp_poll)
0) ! 623.554 us | } (sock_poll)
0) 3.653 us | fget_light();
0) | sock_poll() {

Neat! I think it would read even more C-ish if it was something non-syntactic, like:

Quoted 6 lines
Quoted 4 lines
      0) ! 594.862 us  |              } /* add_wait_queue */
      0) ! 603.361 us  |            } /* __pollwait */
      0) ! 613.574 us  |          } /* tcp_poll */
      0) ! 623.554 us  |        } /* sock_poll */

hm?

Agreed. It won't even be confusing with other events that are printed as comments, since we have the closing brace before.

Quoted 2 lines

Ingo

author maintainer
#3 lore raw
In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

* Steven Rostedt <[email protected]> wrote:

Quoted 8 lines

Neat! I think it would read even more C-ish if it was something non-syntactic, like:

Quoted 4 lines
      0) ! 594.862 us  |              } /* add_wait_queue */
      0) ! 603.361 us  |            } /* __pollwait */
      0) ! 613.574 us  |          } /* tcp_poll */
      0) ! 623.554 us  |        } /* sock_poll */

Yeah, that looks better. I'll write up a patch when I get a chance.

Thanks,

-- Steve

#4 lore raw
In reply to Steven Rostedt
Quoted 48 lines · Steven Rostedt

commit f1c7f517a5dc23bce07efa5ed55e2c074ed9d4ba

Author: Steven Rostedt <[email protected]>
Date:   Fri Feb 26 17:08:16 2010 -0500

ftrace: Add function names to dangling } in function graph tracer

The function graph tracer is currently the most invasive tracer in the ftrace family. It can easily overflow the buffer even with 10megs per CPU. This means that events can often be lost.

On start up, or after events are lost, if the function return is recorded but the function enter was lost, all we get to see is the exiting '}'.

Here is how a typical trace output starts:

[tracing] cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) + 91.897 us | }
0) ! 567.961 us | }
0) <========== |
0) ! 579.083 us | _raw_spin_lock_irqsave();
0) 4.694 us | _raw_spin_unlock_irqrestore();
0) ! 594.862 us | }
0) ! 603.361 us | }
0) ! 613.574 us | }
0) ! 623.554 us | }
0) 3.653 us | fget_light();
0) | sock_poll() {

There are a series of '}' with no matching "func() {". There's no information to what functions these ending brackets belong to.

This patch adds a stack on the per cpu structure used in outputting the function graph tracer to keep track of what function was outputted. Then on a function exit event, it checks the depth to see if the function exit has a matching entry event. If it does, then it only prints the '}', otherwise it adds the function name after the '}'.

This allows function exit events to show what function they belong to at trace output startup, when the entry was lost due to ring buffer overflow, or even after a new task is scheduled in.

Here is what the above trace will look like after this patch:

Quoted 16 lines

[tracing] cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) + 91.897 us | } (irq_exit)
0) ! 567.961 us | } (smp_apic_timer_interrupt)
0) <========== |
0) ! 579.083 us | _raw_spin_lock_irqsave();
0) 4.694 us | _raw_spin_unlock_irqrestore();
0) ! 594.862 us | } (add_wait_queue)
0) ! 603.361 us | } (__pollwait)
0) ! 613.574 us | } (tcp_poll)
0) ! 623.554 us | } (sock_poll)
0) 3.653 us | fget_light();
0) | sock_poll() {

That's a good idea.

Quoted 25 lines
kernel/trace/trace_functions_graph.c +4 -1
index 112561d..e998a82 100644
@@ -18,6 +18,7 @@ struct fgraph_cpu_data {
 	pid_t		last_pid;
 	int		depth;
 	int		ignore;
+	unsigned long	enter_funcs[FTRACE_RETFUNC_DEPTH];
 };
 
 struct fgraph_data {
@@ -670,15 +671,21 @@ print_graph_entry_leaf(struct trace_iterator *iter,
 	duration = graph_ret->rettime - graph_ret->calltime;
 
 	if (data) {
+		struct fgraph_cpu_data *cpu_data;
 		int cpu = iter->cpu;
-		int *depth = &(per_cpu_ptr(data->cpu_data, cpu)->depth);
+
+		cpu_data = per_cpu_ptr(data->cpu_data, cpu);
 

kernel/trace/trace_functions_graph.c print_graph_entry_leaf(struct trace_iterator *iter, · line 687

Quoted 10 lines
kernel/trace/trace_functions_graph.c +4 -1
 		/*
 		 * Comments display at + 1 to depth. Since
 		 * this is a leaf function, keep the comments
 		 * equal to this depth.
 		 */
-		*depth = call->depth - 1;
+		cpu_data->depth = call->depth - 1;
+
+		/* No need to keep this function around for this depth */
+		if (call->depth < FTRACE_RETFUNC_DEPTH)

Do you really need to check that? call->depth >= FTRACE_RETFUNC_DEPTH are not recorded.

Quoted 70 lines
kernel/trace/trace_functions_graph.c +28 -7
+			cpu_data->enter_funcs[call->depth] = 0;
 	}
 
 	/* Overhead */
@@ -718,10 +725,15 @@ print_graph_entry_nested(struct trace_iterator *iter,
 	int i;
 
 	if (data) {
+		struct fgraph_cpu_data *cpu_data;
 		int cpu = iter->cpu;
-		int *depth = &(per_cpu_ptr(data->cpu_data, cpu)->depth);
 
-		*depth = call->depth;
+		cpu_data = per_cpu_ptr(data->cpu_data, cpu);
+		cpu_data->depth = call->depth;
+
+		/* Save this function pointer to see if the exit matches */
+		if (call->depth < FTRACE_RETFUNC_DEPTH)
+			cpu_data->enter_funcs[call->depth] = call->func;
 	}
 
 	/* No overhead */
@@ -851,18 +863,28 @@ print_graph_return(struct ftrace_graph_ret *trace, struct trace_seq *s,
 	struct fgraph_data *data = iter->private;
 	pid_t pid = ent->pid;
 	int cpu = iter->cpu;
+	int func_match = 1;
 	int ret;
 	int i;
 
 	if (data) {
-		int *depth = &(per_cpu_ptr(data->cpu_data, cpu)->depth);
+		struct fgraph_cpu_data *cpu_data;
+		int cpu = iter->cpu;
+
+		cpu_data = per_cpu_ptr(data->cpu_data, cpu);
 
 		/*
 		 * Comments display at + 1 to depth. This is the
 		 * return from a function, we now want the comments
 		 * to display at the same level of the bracket.
 		 */
-		*depth = trace->depth - 1;
+		cpu_data->depth = trace->depth - 1;
+
+		if (trace->depth < FTRACE_RETFUNC_DEPTH) {
+			if (cpu_data->enter_funcs[trace->depth] != trace->func)
+				func_match = 0;
+			cpu_data->enter_funcs[trace->depth] = 0;
+		}
 	}
 
 	if (print_graph_prologue(iter, s, 0, 0))
@@ -887,9 +909,21 @@ print_graph_return(struct ftrace_graph_ret *trace, struct trace_seq *s,
 			return TRACE_TYPE_PARTIAL_LINE;
 	}
 
-	ret = trace_seq_printf(s, "}\n");
-	if (!ret)
-		return TRACE_TYPE_PARTIAL_LINE;
+	/*
+	 * If the return function does not have a matching entry,
+	 * then the entry was lost. Instead of just printing
+	 * the '}' and letting the user guess what function this
+	 * belongs to, write out the function name.
+	 */
+	if (func_match) {
+		ret = trace_seq_printf(s, "}\n");

kernel/trace/trace_functions_graph.c print_graph_return(struct ftrace_graph_ret *trace, struct trace_seq *s, · line 929

Quoted 10 lines
kernel/trace/trace_functions_graph.c +7 -0
+		if (!ret)
+			return TRACE_TYPE_PARTIAL_LINE;
+	} else {
+		ret = trace_seq_printf(s, "} (%ps)\n", (void *)trace->func);
+		if (!ret)
+			return TRACE_TYPE_PARTIAL_LINE;
+	}
 
 	/* Overrun */
 	if (tracer_flags.val & TRACE_GRAPH_PRINT_OVERRUN) {

Very nice!

author maintainer
#5 lore raw
Quoted 7 lines · Frederic Weisbecker
Quoted 7 lines
kernel/trace/trace_functions_graph.c +1 -1
 		/*
 		 * Comments display at + 1 to depth. Since
 		 * this is a leaf function, keep the comments
 		 * equal to this depth.
 		 */
-		*depth = call->depth - 1;
+		cpu_data->depth = call->depth - 1;
Quoted 10 lines

kernel/trace/trace_functions_graph.c print_graph_entry_leaf(struct trace_iterator *iter, · line 687

Quoted 3 lines
kernel/trace/trace_functions_graph.c +3 -0
+
+		/* No need to keep this function around for this depth */
+		if (call->depth < FTRACE_RETFUNC_DEPTH)

Do you really need to check that? call->depth >= FTRACE_RETFUNC_DEPTH are not recorded.

Call me paranoid, but working inside the kernel makes me paranoid. If for some reason a trace gets corrupted here, not doing this check can cause a kernel oops.

Quoted 2 lines

kernel/trace/trace_functions_graph.c print_graph_entry_leaf(struct trace_iterator *iter, · line 688

Quoted 2 lines
kernel/trace/trace_functions_graph.c +1 -0
+			cpu_data->enter_funcs[call->depth] = 0;

cpu_data->enter_funcs[102340320211] = 0;

would be bad ;-)

Hmm, I should also make sure depth is not less than zero. I'll send a new patch to do that too.

-- Steve

Quoted 2 lines
Quoted 2 lines

}

#6 lore raw
Quoted 20 lines · Steven Rostedt
Quoted 7 lines · Frederic Weisbecker
Quoted 7 lines
kernel/trace/trace_functions_graph.c +1 -1
 		/*
 		 * Comments display at + 1 to depth. Since
 		 * this is a leaf function, keep the comments
 		 * equal to this depth.
 		 */
-		*depth = call->depth - 1;
+		cpu_data->depth = call->depth - 1;
Quoted 10 lines

kernel/trace/trace_functions_graph.c print_graph_entry_leaf(struct trace_iterator *iter, · line 687

Quoted 3 lines
kernel/trace/trace_functions_graph.c +3 -0
+
+		/* No need to keep this function around for this depth */
+		if (call->depth < FTRACE_RETFUNC_DEPTH)

Do you really need to check that? call->depth >= FTRACE_RETFUNC_DEPTH are not recorded.

Quoted 3 lines

Call me paranoid, but working inside the kernel makes me paranoid. If for some reason a trace gets corrupted here, not doing this check can cause a kernel oops.

Ok but this may also hide a bug.

Could it be a WARN_ON_ONCE?

Quoted 10 lines
Quoted 2 lines

kernel/trace/trace_functions_graph.c print_graph_entry_leaf(struct trace_iterator *iter, · line 688

Quoted 2 lines
kernel/trace/trace_functions_graph.c +1 -0
+			cpu_data->enter_funcs[call->depth] = 0;

cpu_data->enter_funcs[102340320211] = 0;

would be bad ;-)

Hmm, I should also make sure depth is not less than zero. I'll send a new patch to do that too.

With a WARN_ON_ONCE? :)

Thanks.