Skip to contentSkip to conversation

tracing/function_graph fixes for 2.6.31

PATCH In review Frederic Weisbecker 2 participants 5 messages Jul 28, 2009 – Aug 4, 2009 Tracing

lore
Share & tools
lore
#0 lore raw

cover tracing/function_graph fixes for 2.6.31

Ingo, Thomas,

Those are two fixes for the function graph tracer for .31

The first one fixes a possible race condition between a function graph trace reader and ring buffer writers, leading to a mixup inside an event entry, or worst.

The second one fixes some issues in the output of the function graph tracer while using splice on the trace_pipe file or ftrace_dump().

Both fixes should also be backported on .30 for the -stable branch.

Thanks,
Frederic.

The following changes since commit 4733fd328f14280900435d9dbae1487d110a4d56:
Benjamin Herrenschmidt (1):
mm: Remove duplicate definitions in MIPS and SH

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/fixes

Lai Jiangshan (2):
tracing: Fix invalid function_graph entry
tracing: Fix missing function_graph events when we splice_read from trace_pipe

 kernel/trace/trace.c                 |   10 +++++++---
 kernel/trace/trace_functions_graph.c |   11 +++++++++--
 2 files changed, 16 insertions(+), 5 deletions(-)
#1 lore raw

PATCH 1/2 tracing: Fix invalid function_graph entry

From: Lai Jiangshan <[email protected]>

When print_graph_entry() computes a function call entry event, it needs to also check the next entry to guess if it matches the return event of the current function entry.
In order to look at this next event, it needs to consume the current entry before going ahead in the ring buffer.

However, if the current event that gets consumed is the last one in the ring buffer head page, the ring_buffer may reuse the page for writers. The consumed entry will then become invalid because of possible racy overwriting.

Me must then handle this entry by making a copy of it.

The fix also applies on 2.6.30

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_functions_graph.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
kernel/trace/trace_functions_graph.c +9 -2
index d2249ab..420ec34 100644
@@ -843,9 +843,16 @@ print_graph_function(struct trace_iterator *iter)
 
 	switch (entry->type) {
 	case TRACE_GRAPH_ENT: {
-		struct ftrace_graph_ent_entry *field;
+		/*
+		 * print_graph_entry() may consume the current event,
+		 * thus @field may become invalid, so we need to save it.
+		 * sizeof(struct ftrace_graph_ent_entry) is very small,
+		 * it can be safely saved at the stack.
+		 */
+		struct ftrace_graph_ent_entry *field, saved;
 		trace_assign_type(field, entry);
-		return print_graph_entry(field, s, iter);
+		saved = *field;
+		return print_graph_entry(&saved, s, iter);
 	}
 	case TRACE_GRAPH_RET: {
 		struct ftrace_graph_ret_entry *field;
Signature · 1 line
1.6.2.3
#2 lore raw

PATCH 2/2 tracing: Fix missing function_graph events when we splice_read from trace_pipe

In reply to Frederic Weisbecker

From: Lai Jiangshan <[email protected]>

About a half events are missing when we splice_read from trace_pipe. They are unexpectedly consumed because we ignore the TRACE_TYPE_NO_CONSUME return value used by the function graph tracer when it needs to consume the events by itself to walk on the ring buffer.

The same problem appears with ftrace_dump()

Example of an output before this patch:

1)               |      ktime_get_real() {
1)   2.846 us    |          read_hpet();
1)   4.558 us    |        }
1)   6.195 us    |      }

After this patch:

0)               |      ktime_get_real() {
0)               |        getnstimeofday() {
0)   1.960 us    |          read_hpet();
0)   3.597 us    |        }
0)   5.196 us    |      }

The fix also applies on 2.6.30

LKML-Reference: <[email protected]>

---
 kernel/trace/trace.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)
kernel/trace/trace.c +7 -3
index 8bc8d8a..da984ad 100644
@@ -3085,7 +3085,8 @@ tracing_fill_pipe_page(size_t rem, struct trace_iterator *iter)
 			break;
 		}
 
-		trace_consume(iter);
+		if (ret != TRACE_TYPE_NO_CONSUME)
+			trace_consume(iter);
 		rem -= count;
 		if (!find_next_entry_inc(iter))	{
 			rem = 0;
@@ -4233,8 +4234,11 @@ static void __ftrace_dump(bool disable_tracing)
 		iter.pos = -1;
 
 		if (find_next_entry_inc(&iter) != NULL) {
-			print_trace_line(&iter);
-			trace_consume(&iter);
+			int ret;
+
+			ret = print_trace_line(&iter);
+			if (ret != TRACE_TYPE_NO_CONSUME)
+				trace_consume(&iter);
 		}
 
 		trace_printk_seq(&iter.seq);
Signature · 1 line
1.6.2.3
#3 lore raw
In reply to Frederic Weisbecker
Quoted 8 lines · Frederic Weisbecker

Ingo, Thomas,

Those are two fixes for the function graph tracer for .31

The first one fixes a possible race condition between a function graph trace reader and ring buffer writers, leading to a mixup inside an event entry, or worst.

Quoted 7 lines

The second one fixes some issues in the output of the function graph tracer while using splice on the trace_pipe file or ftrace_dump().

Both fixes should also be backported on .30 for the -stable branch.

Thanks,
Frederic.

(Changing the title with the [GIT PULL] prefix, in case you miss it.)

Thanks.

Quoted 15 lines

The following changes since commit 4733fd328f14280900435d9dbae1487d110a4d56:
Benjamin Herrenschmidt (1):
mm: Remove duplicate definitions in MIPS and SH

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/fixes

Lai Jiangshan (2):
tracing: Fix invalid function_graph entry
tracing: Fix missing function_graph events when we splice_read from trace_pipe

 kernel/trace/trace.c                 |   10 +++++++---
 kernel/trace/trace_functions_graph.c |   11 +++++++++--
 2 files changed, 16 insertions(+), 5 deletions(-)
#4 lore raw
In reply to Frederic Weisbecker
Quoted 26 lines · Frederic Weisbecker

Ingo, Thomas,

Those are two fixes for the function graph tracer for .31

The first one fixes a possible race condition between a function graph trace reader and ring buffer writers, leading to a mixup inside an event entry, or worst.

The second one fixes some issues in the output of the function graph tracer while using splice on the trace_pipe file or ftrace_dump().

Both fixes should also be backported on .30 for the -stable branch.

Thanks,
Frederic.

The following changes since commit 4733fd328f14280900435d9dbae1487d110a4d56:
Benjamin Herrenschmidt (1):
mm: Remove duplicate definitions in MIPS and SH

are available in the git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git tracing/fixes

Quoted 7 lines

Lai Jiangshan (2):
tracing: Fix invalid function_graph entry
tracing: Fix missing function_graph events when we splice_read from trace_pipe

 kernel/trace/trace.c                 |   10 +++++++---
 kernel/trace/trace_functions_graph.c |   11 +++++++++--
 2 files changed, 16 insertions(+), 5 deletions(-)

Pulled, thanks a lot Frederic!

Ingo