Skip to contentSkip to conversation

ftrace: ftrace_printk fixes

PATCH In review Steven Rostedt 3 participants 7 messages Sep 3, 2008 – Sep 4, 2008 Tracing

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover ftrace: ftrace_printk fixes

Peter Zijlstra is starting to use ftrace_printk to debug the scheduler. He quickly came across a few issues.

These two patches address some of the things he has concerns about.

I'm expecting that Peter will be sending some more patches to help with the interface.

-- Steve

author maintainer
#1 lore raw

PATCH 1/2 ftrace: binary and not logical for continue test

Peter Zijlstra provided me with a nice brown paper bag while letting me know that I was doing a logical AND and not a binary one, making a condition true more often than it should be.

Luckily, a false true is handled by the calling function and no harm is done. But this needs to be fixed regardless.

---
 kernel/trace/trace.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/trace/trace.c +3 -3
Index: linux-tip.git/kernel/trace/trace.c
===================================================================
@@ -1655,7 +1655,7 @@ print_lat_fmt(struct trace_iterator *ite
 	case TRACE_PRINT:
 		seq_print_ip_sym(s, field->print.ip, sym_flags);
 		trace_seq_printf(s, ": %s", field->print.buf);
-		if (field->flags && TRACE_FLAG_CONT)
+		if (field->flags & TRACE_FLAG_CONT)
 			trace_seq_print_cont(s, iter);
 		break;
 	default:
@@ -1768,7 +1768,7 @@ static int print_trace_fmt(struct trace_
 	case TRACE_PRINT:
 		seq_print_ip_sym(s, field->print.ip, sym_flags);
 		trace_seq_printf(s, ": %s", field->print.buf);
-		if (field->flags && TRACE_FLAG_CONT)
+		if (field->flags & TRACE_FLAG_CONT)
 			trace_seq_print_cont(s, iter);
 		break;
 	}
@@ -1833,7 +1833,7 @@ static int print_raw_fmt(struct trace_it
 	case TRACE_PRINT:
 		trace_seq_printf(s, "# %lx %s",
 				 field->print.ip, field->print.buf);
-		if (field->flags && TRACE_FLAG_CONT)
+		if (field->flags & TRACE_FLAG_CONT)
 			trace_seq_print_cont(s, iter);
 		break;
 	}
author maintainer
#2 lore raw

PATCH 2/2 ftrace: print continue index fix

In reply to Steven Rostedt

An item in the trace buffer that is bigger than one entry may be split up using the TRACE_CONT entry. This makes it a virtual single entry. The current code increments the iterator index even while traversing TRACE_CONT entries, making it look like the iterator is further than it actually is.

This patch adds code to not increment the iterator index while skipping over TRACE_CONT entries.

---
 kernel/trace/trace.c |   15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
kernel/trace/trace.c +10 -5
Index: linux-tip.git/kernel/trace/trace.c
===================================================================
@@ -1117,9 +1117,8 @@ trace_entry_idx(struct trace_array *tr, 
 }
 
 /* Increment the index counter of an iterator by one */
-static void trace_iterator_increment(struct trace_iterator *iter, int cpu)
+static void __trace_iterator_increment(struct trace_iterator *iter, int cpu)
 {
-	iter->idx++;
 	iter->next_idx[cpu]++;
 	iter->next_page_idx[cpu]++;
 
@@ -1132,6 +1131,12 @@ static void trace_iterator_increment(str
 	}
 }
 
+static void trace_iterator_increment(struct trace_iterator *iter, int cpu)
+{
+	iter->idx++;
+	__trace_iterator_increment(iter, cpu);
+}
+
 static struct trace_entry *
 trace_entry_next(struct trace_array *tr, struct trace_array_cpu *data,
 		 struct trace_iterator *iter, int cpu)
@@ -1153,7 +1158,7 @@ trace_entry_next(struct trace_array *tr,
 
 	/* find a real entry */
 	do {
-		trace_iterator_increment(iter, cpu);
+		__trace_iterator_increment(iter, cpu);
 		ent = trace_entry_idx(tr, tr->data[cpu], iter, cpu);
 	} while (ent && ent->type != TRACE_CONT);
 
@@ -1187,7 +1192,7 @@ __find_next_entry(struct trace_iterator 
 				ent = trace_entry_next(tr, data, iter, cpu);
 			else {
 				while (ent && ent->type == TRACE_CONT) {
-					trace_iterator_increment(iter, cpu);
+					__trace_iterator_increment(iter, cpu);
 					ent = trace_entry_idx(tr, tr->data[cpu],
 							      iter, cpu);
 				}
@@ -1566,7 +1571,7 @@ trace_seq_print_cont(struct trace_seq *s
 
 	do {
 		trace_seq_printf(s, "%s", ent->cont.buf);
-		trace_iterator_increment(iter, iter->cpu);
+		__trace_iterator_increment(iter, iter->cpu);
 		ent = trace_entry_idx(tr, data, iter, iter->cpu);
 	} while (ent && ent->type == TRACE_CONT);
 }
Acked-by on the v1 series
#3 lore raw
In reply to Steven Rostedt
Quoted 4 lines · Steven Rostedt

Peter Zijlstra is starting to use ftrace_printk to debug the scheduler. He quickly came across a few issues.

These two patches address some of the things he has concerns about.

Yay - works now!

Quoted 2 lines

I'm expecting that Peter will be sending some more patches to help with the interface.

Yep, I will.. let me extract that from the massive diff I have here.. :-)

#4 lore raw

PATCH 1/1 ftrace: make ftrace_printk usable with the other tracers

In reply to Steven Rostedt

Subject: ftrace: make ftrace_printk usable with the other tracers
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 02 21:12:13 CEST 2008

Currently ftrace_printk only works with the ftrace tracer, switch it to an iter_ctrl setting so we can make us of them with other tracers too.

[[email protected]: tweak to the disable condition]
---
 kernel/trace/trace.c |   41 +++++++++++++++++++++++++++--------------
 kernel/trace/trace.h |    2 ++
 2 files changed, 29 insertions(+), 14 deletions(-)
kernel/trace/trace.c +5 -3
Index: linux-2.6/kernel/trace/trace.c
===================================================================
@@ -235,6 +235,7 @@ static const char *trace_options[] = {
 	"block",
 	"stacktrace",
 	"sched-tree",
+	"ftrace_printk",
 	NULL
 };
 
@@ -3086,9 +3098,10 @@ static __init void tracer_init_debugfs(v
 
 int __ftrace_printk(unsigned long ip, const char *fmt, ...)
 {
-	struct trace_array *tr = &global_trace;
 	static DEFINE_SPINLOCK(trace_buf_lock);
 	static char trace_buf[TRACE_BUF_SIZE];
+
+	struct trace_array *tr = &global_trace;
 	struct trace_array_cpu *data;
 	struct trace_entry *entry;
 	unsigned long flags;
@@ -3096,7 +3109,7 @@ int __ftrace_printk(unsigned long ip, co
 	va_list ap;
 	int cpu, len = 0, write, written = 0;
 
-	if (likely(!ftrace_function_enabled))
+	if (!(trace_flags & TRACE_ITER_PRINTK) || !tr->ctrl || tracing_disabled)
 		return 0;
 
 	local_irq_save(flags);
@@ -3104,7 +3117,7 @@ int __ftrace_printk(unsigned long ip, co
 	data = tr->data[cpu];
 	disabled = atomic_inc_return(&data->disabled);
 
-	if (unlikely(disabled != 1 || !ftrace_function_enabled))
+	if (unlikely(disabled != 1))
 		goto out;
 
 	spin_lock(&trace_buf_lock);
kernel/trace/trace.h +1 -0
Index: linux-2.6/kernel/trace/trace.h
===================================================================
@@ -356,6 +357,7 @@ enum trace_iterator_flags {
 	TRACE_ITER_BLOCK		= 0x80,
 	TRACE_ITER_STACKTRACE		= 0x100,
 	TRACE_ITER_SCHED_TREE		= 0x200,
+	TRACE_ITER_PRINTK		= 0x400,
 };
 
 #endif /* _LINUX_KERNEL_TRACE_H */
#5 lore raw

PATCH 1/1 ftrace: sched_switch: show the wakee's cpu

In reply to Steven Rostedt

Subject: ftrace: sched_switch: show the wakee's cpu
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 02 21:12:13 CEST 2008

While profiling the smp behaviour of the scheduler it was needed to know to which cpu a task got woken.

---
 kernel/trace/trace.c |   41 +++++++++++++++++++++++++++--------------
 kernel/trace/trace.h |    2 ++
 2 files changed, 29 insertions(+), 14 deletions(-)
kernel/trace/trace.c +9 -3
Index: linux-2.6/kernel/trace/trace.c
===================================================================
@@ -976,6 +977,7 @@ tracing_sched_switch_trace(struct trace_
 	entry->field.ctx.next_pid	= next->pid;
 	entry->field.ctx.next_prio	= next->prio;
 	entry->field.ctx.next_state	= next->state;
+	entry->field.ctx.next_cpu	= task_cpu(next);
 	__trace_stack(tr, data, flags, 5);
 	__raw_spin_unlock(&data->lock);
 	raw_local_irq_restore(irq_flags);
@@ -1002,6 +1004,7 @@ tracing_sched_wakeup_trace(struct trace_
 	entry->field.ctx.next_pid	= wakee->pid;
 	entry->field.ctx.next_prio	= wakee->prio;
 	entry->field.ctx.next_state	= wakee->state;
+	entry->field.ctx.next_cpu	= task_cpu(wakee);
 	__trace_stack(tr, data, flags, 6);
 	__raw_spin_unlock(&data->lock);
 	raw_local_irq_restore(irq_flags);
@@ -1630,10 +1638,11 @@ print_lat_fmt(struct trace_iterator *ite
 			__ffs(field->ctx.prev_state) + 1 : 0;
 		S = state < sizeof(state_to_char) - 1 ? state_to_char[state] : 'X';
 		comm = trace_find_cmdline(field->ctx.next_pid);
-		trace_seq_printf(s, " %5d:%3d:%c %s %5d:%3d:%c %s\n",
+		trace_seq_printf(s, " %5d:%3d:%c %s [%03d] %5d:%3d:%c %s\n",
 				 field->ctx.prev_pid,
 				 field->ctx.prev_prio,
 				 S, entry->type == TRACE_CTX ? "==>" : "  +",
+				 field->ctx.next_cpu,
 				 field->ctx.next_pid,
 				 field->ctx.next_prio,
 				 T, comm);
@@ -1730,11 +1739,12 @@ static int print_trace_fmt(struct trace_
 			state_to_char[field->ctx.prev_state] : 'X';
 		T = field->ctx.next_state < sizeof(state_to_char) ?
 			state_to_char[field->ctx.next_state] : 'X';
-		ret = trace_seq_printf(s, " %5d:%3d:%c %s %5d:%3d:%c\n",
+		ret = trace_seq_printf(s, " %5d:%3d:%c %s [%03d] %5d:%3d:%c\n",
 				       field->ctx.prev_pid,
 				       field->ctx.prev_prio,
 				       S,
 				       entry->type == TRACE_CTX ? "==>" : "  +",
+				       field->ctx.next_cpu,
 				       field->ctx.next_pid,
 				       field->ctx.next_prio,
 				       T);
@@ -1811,10 +1821,11 @@ static int print_raw_fmt(struct trace_it
 			state_to_char[field->ctx.next_state] : 'X';
 		if (entry->type == TRACE_WAKE)
 			S = '+';
-		ret = trace_seq_printf(s, "%d %d %c %d %d %c\n",
+		ret = trace_seq_printf(s, "%d %d %c %d %d %d %c\n",
 				       field->ctx.prev_pid,
 				       field->ctx.prev_prio,
 				       S,
+				       field->ctx.next_cpu,
 				       field->ctx.next_pid,
 				       field->ctx.next_prio,
 				       T);
@@ -1887,6 +1898,7 @@ static int print_hex_fmt(struct trace_it
 		SEQ_PUT_HEX_FIELD_RET(s, field->ctx.prev_pid);
 		SEQ_PUT_HEX_FIELD_RET(s, field->ctx.prev_prio);
 		SEQ_PUT_HEX_FIELD_RET(s, S);
+		SEQ_PUT_HEX_FIELD_RET(s, field->ctx.next_cpu);
 		SEQ_PUT_HEX_FIELD_RET(s, field->ctx.next_pid);
 		SEQ_PUT_HEX_FIELD_RET(s, field->ctx.next_prio);
 		SEQ_PUT_HEX_FIELD_RET(s, T);
kernel/trace/trace.h +1 -0
Index: linux-2.6/kernel/trace/trace.h
===================================================================
@@ -41,6 +41,7 @@ struct ctx_switch_entry {
 	unsigned int		next_pid;
 	unsigned char		next_prio;
 	unsigned char		next_state;
+	unsigned int		next_cpu;
 };
 
 /*
Applied
#6 lore raw

Re: [PATCH] ftrace: sched_switch: show the wakee's cpu

Quoted 6 lines · Peter Zijlstra

Subject: ftrace: sched_switch: show the wakee's cpu
From: Peter Zijlstra <[email protected]>
Date: Tue Sep 02 21:12:13 CEST 2008

While profiling the smp behaviour of the scheduler it was needed to know to which cpu a task got woken.

applied this and the other patches in this thread to tip/tracing/ftrace:

0d32728: ftrace: sched_switch: show the wakee's cpu 33fbc35: ftrace: make ftrace_printk usable with the other tracers 32b6a7b: ftrace: print continue index fix
3340e7e: ftrace: binary and not logical for continue test

thanks!

Ingo