Skip to contentSkip to conversation

Tracing configuration review

PATCH Applied Chase Douglas 9 participants 23 messages May 25, 2010 – Jun 14, 2010 Tracing

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm going through our Ubuntu kernel configuration for our next release to ensure we have all the trace options enabled that may be useful. I have a few questions about what tracer options we should have enabled.

Our guiding principle in regards to these options is: if an option can be turned on and has no performance impact unless explicitly enabled on the kernel command line or at runtime, we are happy to enable it. Secondarily, we don't want to enable options that are headed for deprecation.

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

# CONFIG_IRQSOFF_TRACER is not set (performance impact by default) # CONFIG_SYSPROF_TRACER is not set (don't know much about this) # CONFIG_SCHED_TRACER is not set (headed for deprecation?) CONFIG_FTRACE_SYSCALLS=y (no performance impact by default) CONFIG_BOOT_TRACER=y (no performance impact by default) CONFIG_KSYM_TRACER=y (no performance impact by default) # CONFIG_STACK_TRACER is not set (Kconfig says N if unsure) # CONFIG_KMEMTRACE is not set (Kconfig says N if unsure) CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

Lastly, what options are safe for performance when HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me through what's in Documentation/trace/* and the Kconfig entries what options could cause a performance decrease due to the inability to dynamically enable and disable tracing at runtime.

Thanks,

-- Chase

maintainer
#1 lore raw
Quoted 12 lines · Chase Douglas

Hi all,

I'm going through our Ubuntu kernel configuration for our next release to ensure we have all the trace options enabled that may be useful. I have a few questions about what tracer options we should have enabled.

Our guiding principle in regards to these options is: if an option can be turned on and has no performance impact unless explicitly enabled on the kernel command line or at runtime, we are happy to enable it. Secondarily, we don't want to enable options that are headed for deprecation.

Quoted 6 lines

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

# CONFIG_IRQSOFF_TRACER is not set (performance impact by default)

Correct, keep that off.

Quoted 1 line

# CONFIG_SYSPROF_TRACER is not set (don't know much about this)

Neither do I ;-)

Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Although it is headed for deprecation, I think it still gets set by other tracers, since it has the code to initiate the comm reader.

Quoted 1 line

CONFIG_FTRACE_SYSCALLS=y (no performance impact by default)

Correct

Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

But this tracer is pretty useless. It gives no more information than debug_initcalls.

Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

Yep

Quoted 1 line

# CONFIG_STACK_TRACER is not set (Kconfig says N if unsure)

I would set this if you already have the function tracer. It gives no more overhead than that, and it is very useful.

Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Don't know.

Quoted 7 lines

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

Lastly, what options are safe for performance when HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me through what's in Documentation/trace/* and the Kconfig entries what options could cause a performance decrease due to the inability to dynamically enable and disable tracing at runtime.

HAVE_DYNAMIC_FTRACE affects the function tracer. If you do not have that, then do not enable FUNCTION_TRACER or anything that depends on it.

Also note, FUNCTION_TRACER depends on FRAME_POINTERS. Your millage may vary with that. If you already have FRAME_POINTERS on, and the arch supports DYNAMIC_FTRACE, then its fine to have FUNCTION_TRACER and all those that are built on top (STACK_TRACER, FUNCTION_GRAPH, etc)

-- Steve

#2 lore raw
Quoted 62 lines · Steven Rostedt
Quoted 12 lines · Chase Douglas

Hi all,

I'm going through our Ubuntu kernel configuration for our next release to ensure we have all the trace options enabled that may be useful. I have a few questions about what tracer options we should have enabled.

Our guiding principle in regards to these options is: if an option can be turned on and has no performance impact unless explicitly enabled on the kernel command line or at runtime, we are happy to enable it. Secondarily, we don't want to enable options that are headed for deprecation.

Quoted 6 lines

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

# CONFIG_IRQSOFF_TRACER is not set (performance impact by default)

Correct, keep that off.

Quoted 1 line

# CONFIG_SYSPROF_TRACER is not set (don't know much about this)

Neither do I ;-)

Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Although it is headed for deprecation, I think it still gets set by other tracers, since it has the code to initiate the comm reader.

Quoted 1 line

CONFIG_FTRACE_SYSCALLS=y (no performance impact by default)

Correct

Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

But this tracer is pretty useless. It gives no more information than debug_initcalls.

Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

Yep

Quoted 1 line

# CONFIG_STACK_TRACER is not set (Kconfig says N if unsure)

I would set this if you already have the function tracer. It gives no more overhead than that, and it is very useful.

Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Don't know.

Quoted 7 lines

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

Lastly, what options are safe for performance when HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me through what's in Documentation/trace/* and the Kconfig entries what options could cause a performance decrease due to the inability to dynamically enable and disable tracing at runtime.

Quoted 7 lines

HAVE_DYNAMIC_FTRACE affects the function tracer. If you do not have that, then do not enable FUNCTION_TRACER or anything that depends on it.

Also note, FUNCTION_TRACER depends on FRAME_POINTERS. Your millage may vary with that. If you already have FRAME_POINTERS on, and the arch supports DYNAMIC_FTRACE, then its fine to have FUNCTION_TRACER and all those that are built on top (STACK_TRACER, FUNCTION_GRAPH, etc)

(I would have snipped much of the above, but since I've added CC for the Ubuntu list I wanted to leave it as is for this first reply)

After enabling KSYM_TRACER, I was presented with PROFILE_KSYM_TRACER. This is also "Say N if unsure," so I could use some guidance on whether we should turn it on as well.

Thanks,

-- Chase

#3 lore raw
Quoted 71 lines · Chase Douglas
Quoted 62 lines · Steven Rostedt
Quoted 12 lines · Chase Douglas

Hi all,

I'm going through our Ubuntu kernel configuration for our next release to ensure we have all the trace options enabled that may be useful. I have a few questions about what tracer options we should have enabled.

Our guiding principle in regards to these options is: if an option can be turned on and has no performance impact unless explicitly enabled on the kernel command line or at runtime, we are happy to enable it. Secondarily, we don't want to enable options that are headed for deprecation.

Quoted 6 lines

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

# CONFIG_IRQSOFF_TRACER is not set (performance impact by default)

Correct, keep that off.

Quoted 1 line

# CONFIG_SYSPROF_TRACER is not set (don't know much about this)

Neither do I ;-)

Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Although it is headed for deprecation, I think it still gets set by other tracers, since it has the code to initiate the comm reader.

Quoted 1 line

CONFIG_FTRACE_SYSCALLS=y (no performance impact by default)

Correct

Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

But this tracer is pretty useless. It gives no more information than debug_initcalls.

Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

Yep

Quoted 1 line

# CONFIG_STACK_TRACER is not set (Kconfig says N if unsure)

I would set this if you already have the function tracer. It gives no more overhead than that, and it is very useful.

Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Don't know.

Quoted 7 lines

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

Lastly, what options are safe for performance when HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me through what's in Documentation/trace/* and the Kconfig entries what options could cause a performance decrease due to the inability to dynamically enable and disable tracing at runtime.

Quoted 7 lines

HAVE_DYNAMIC_FTRACE affects the function tracer. If you do not have that, then do not enable FUNCTION_TRACER or anything that depends on it.

Also note, FUNCTION_TRACER depends on FRAME_POINTERS. Your millage may vary with that. If you already have FRAME_POINTERS on, and the arch supports DYNAMIC_FTRACE, then its fine to have FUNCTION_TRACER and all those that are built on top (STACK_TRACER, FUNCTION_GRAPH, etc)

Quoted 6 lines

(I would have snipped much of the above, but since I've added CC for the Ubuntu list I wanted to leave it as is for this first reply)

After enabling KSYM_TRACER, I was presented with PROFILE_KSYM_TRACER. This is also "Say N if unsure," so I could use some guidance on whether we should turn it on as well.

KSYM_TRACER traces memory accesses (using breakpoints) and display each of these in the trace/trace_pipe files. PROFILE_KSYM_TRACER produces basic statistics on top of these traces.

But both are deprecated really, the perf tools interface is much better for this job.

#4 lore raw
In reply to Steven Rostedt
Quoted 5 lines · Steven Rostedt
Quoted 1 line · Chase Douglas

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Although it is headed for deprecation, I think it still gets set by other tracers, since it has the code to initiate the comm reader.

Doesn't actually CONTEXT_SWITCH_TRACER has the code for the comm things?

Quoted 4 lines
Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

But this tracer is pretty useless. It gives no more information than debug_initcalls.

Yeah it's pretty useless. And these informations can also be displayed through printk on boot so...

maintainer
#5 lore raw
Quoted 4 lines · Frederic Weisbecker
Quoted 3 lines · Steven Rostedt
Quoted 1 line · Chase Douglas

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Quoted 8 lines
Quoted 2 lines

Although it is headed for deprecation, I think it still gets set by other tracers, since it has the code to initiate the comm reader.

Doesn't actually CONTEXT_SWITCH_TRACER has the code for the comm things?

Ah, I was confused. But then the SCHED_TRACER is the wakeup tracer, and that is heading for deprecation??

-- Steve

#6 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 9 lines · Frederic Weisbecker
Quoted 5 lines · Steven Rostedt
Quoted 1 line · Chase Douglas

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Although it is headed for deprecation, I think it still gets set by other tracers, since it has the code to initiate the comm reader.

Quoted 6 lines
Quoted 3 lines

Doesn't actually CONTEXT_SWITCH_TRACER has the code for the comm things?

Ah, I was confused. But then the SCHED_TRACER is the wakeup tracer, and that is heading for deprecation??

It would be nice if we can make it a ftrace_event_call, so that we can control it like any other trace event.

But for now we don't have any replacement, so it is not yet deprecated. That's why I said "in the long term" ;)

#7 lore raw

PATCH 1/1 tracing: remove boot tracer

In reply to Frederic Weisbecker
Quoted 10 lines · Frederic Weisbecker
Quoted 4 lines · Steven Rostedt
Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

But this tracer is pretty useless. It gives no more information than debug_initcalls.

Yeah it's pretty useless. And these informations can also be displayed through printk on boot so...

Then just remove it.

---

kernel/trace/Kconfig +0 -17
index 8b1797c..572992a 100644
@@ -229,23 +229,6 @@ config FTRACE_SYSCALLS
 	help
 	  Basic tracer to catch the syscall entry and exit events.
 
-config BOOT_TRACER
-	bool "Trace boot initcalls"
-	select GENERIC_TRACER
-	select CONTEXT_SWITCH_TRACER
-	help
-	  This tracer helps developers to optimize boot times: it records
-	  the timings of the initcalls and traces key events and the identity
-	  of tasks that can cause boot delays, such as context-switches.
-
-	  Its aim is to be parsed by the scripts/bootgraph.pl tool to
-	  produce pretty graphics about boot inefficiencies, giving a visual
-	  representation of the delays during initcalls - but the raw
-	  /debug/tracing/trace text output is readable too.
-
-	  You must pass in initcall_debug and ftrace=initcall to the kernel
-	  command line to enable this on bootup.
-
 config TRACE_BRANCH_PROFILING
 	bool
 	select GENERIC_TRACER
kernel/trace/Makefile +0 -1
index ffb1a5b..c3aaeba 100644
@@ -38,7 +38,6 @@ obj-$(CONFIG_SCHED_TRACER) += trace_sched_wakeup.o
 obj-$(CONFIG_NOP_TRACER) += trace_nop.o
 obj-$(CONFIG_STACK_TRACER) += trace_stack.o
 obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o
-obj-$(CONFIG_BOOT_TRACER) += trace_boot.o
 obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += trace_functions_graph.o
 obj-$(CONFIG_TRACE_BRANCH_PROFILING) += trace_branch.o
 obj-$(CONFIG_KMEMTRACE) += kmemtrace.o
kernel/trace/trace.c +0 -3
index 8a76339..687c487 100644
@@ -4602,9 +4602,6 @@ __init static int tracer_alloc_buffers(void)
 
 	register_tracer(&nop_trace);
 	current_trace = &nop_trace;
-#ifdef CONFIG_BOOT_TRACER
-	register_tracer(&boot_tracer);
-#endif
 	/* All seems OK, enable tracing */
 	tracing_disabled = 0;
 
kernel/trace/trace_boot.c deleted +0 -185

185 lines not on this page · raw message

#8 lore raw

Re: [Patch] tracing: remove boot tracer

Quoted 9 lines · Américo Wang
Quoted 8 lines · Frederic Weisbecker
Quoted 4 lines · Steven Rostedt
Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

But this tracer is pretty useless. It gives no more information than debug_initcalls.

Quoted 7 lines
Quoted 2 lines

Yeah it's pretty useless. And these informations can also be displayed through printk on boot so...

Then just remove it.

Queued, thanks!

#9 lore raw
In reply to Chase Douglas
Quoted 11 lines · Chase Douglas

Hi all,

I'm going through our Ubuntu kernel configuration for our next release to ensure we have all the trace options enabled that may be useful. I have a few questions about what tracer options we should have enabled.

Our guiding principle in regards to these options is: if an option can be turned on and has no performance impact unless explicitly enabled on the kernel command line or at runtime, we are happy to enable it. Secondarily, we don't want to enable options that are headed for deprecation.

Ok.

Quoted 7 lines

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

# CONFIG_IRQSOFF_TRACER is not set (performance impact by default)

Indeed.

Quoted 1 line

# CONFIG_SYSPROF_TRACER is not set (don't know much about this)

It is the upstream kernel implementation for sysprof:
http://www.daimi.au.dk/~sandmann/sysprof/

But I suspect it is not used widely. I think the users prefer the out of tree module.

And IIRC, sysprof now can use the perf interface instead. I guess we can think about it as deprecated.

Soeren could tell more about it.

Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

We want to deprecate it in the long term, but for now we don't have any replacement. Cool for RT latency tracing.

Quoted 1 line

CONFIG_FTRACE_SYSCALLS=y (no performance impact by default)

Yeah, this one is fine, and nice to have.

Quoted 1 line

CONFIG_BOOT_TRACER=y (no performance impact by default)

Yep.
It is targeted for deprecation in the long term but we don't have the replacement yet.

Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

IMO, it is deprecated. The perf interface is much more powerful and flexible. Prasad, do you agree if I remove this ftrace plugin?

Quoted 1 line

# CONFIG_STACK_TRACER is not set (Kconfig says N if unsure)

Can be useful to track places that eats most the stack. No overhead if off and CONFIG_DYNAMIC_FTRACE.

Again, it is targeted for deprecation in the long term but we don't have any replacement yet.

Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Deprecated. We have the kmem trace event that are a full replacement now. Pekka, Gabriel, can we remove it now?

Quoted 1 line

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

In the way for deprecation.

Quoted 5 lines

Lastly, what options are safe for performance when HAVE_DYNAMIC_FTRACE=n, like on our ARM kernels. It is not clear to me through what's in Documentation/trace/* and the Kconfig entries what options could cause a performance decrease due to the inability to dynamically enable and disable tracing at runtime.

So, if you have HAVE_DYNAMIC_FTRACE=n, you want to avoid any of the following:

CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_STACK_TRACER=y
CONFIG_FUNCTION_PROFILER=y

Because these will have a noticeable overhead even when they are disabled. Otherwise if you have CONFIG_DYNAMIC_FTRACE=y, the four above are safe wrt performance when they are =y but disabled.

And they are nice features. We want to make them use the trace events interface, which means we'll probably deprecate them in the long term, but that will be only to change their interface (like most of the other tracing features targeted for deprecation).

Thanks.

#10 lore raw
Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines · Chase Douglas

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

<snip>

Quoted 5 lines
Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

We want to deprecate it in the long term, but for now we don't have any replacement. Cool for RT latency tracing.

I thought that the functionality is the same as what you get by:

echo 1 > (debufs)/tracing/events/sched/enable

Quoted 5 lines
Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

IMO, it is deprecated. The perf interface is much more powerful and flexible. Prasad, do you agree if I remove this ftrace plugin?

If there isn't any use in enabling it due to perf's features, then we can turn it off. However, if there's any use to be gained by this over perf's features, then I'd prefer to leave it on. Thoughts?

Quoted 4 lines
Quoted 1 line

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

In the way for deprecation.

Is this like the KMEM_TRACER where trace events have superseded it?

Thanks,

-- Chase

#11 lore raw
Quoted 12 lines · Chase Douglas
Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines · Chase Douglas

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

<snip>

Quoted 3 lines
Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

Quoted 6 lines
Quoted 2 lines

We want to deprecate it in the long term, but for now we don't have any replacement. Cool for RT latency tracing.

I thought that the functionality is the same as what you get by:

echo 1 > (debufs)/tracing/events/sched/enable

No, enabling every sched events will simply dump every events related to the scheduler. It's then up to the user to make sense of these traces through post-processing.

The wakeup tracer hooks the scheduler events for the specific purpose of tracing the scheduler latencies: it measures the time between a task is woken up and its actual scheduling to a cpu. If you have the function tracer built, you'll also have a function trace of everything that happened in-between.

So the wakeup tracer brings a kind of brain on top of the sched events, but for very specific purposes.

Quoted 9 lines
Quoted 5 lines
Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

IMO, it is deprecated. The perf interface is much more powerful and flexible. Prasad, do you agree if I remove this ftrace plugin?

If there isn't any use in enabling it due to perf's features, then we can turn it off. However, if there's any use to be gained by this over perf's features, then I'd prefer to leave it on. Thoughts?

No, perf does much more:

- stacktraces recording
- "top" alike view with perf top
- stat with perf stat, etc...
- userspace memory accesses

Here is a quick example:

$ cat test.c
int var;

void func_c(void)

{
	var++;
}

void func_b(void)

{
	func_c();
}

void func_a(void)

{
	func_c();
}

int main(int argc, char **argv)

{
	int i;

	for (i = 0; i < 1000; i++)
		if (i % 2)
			func_a();
		else
			func_b();

	return 0;
}

//end test.c

$ gcc test.c -fno-omit-frame-pointer -o test
$ readelf -s test | grep var
    74: 000000000060102c     4 OBJECT  GLOBAL DEFAULT   25 var
$ perf record -g -c 1 -e mem:0x000000000060102c:w ./test
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.069 MB perf.data (~3020 samples) ]
$ perf report

# Events: 1K cycles
#

# Overhead  Command      Shared Object  Symbol
# ........  .......  .................  ......
#
    99.90%     test  test               [.] func_c
               |
               --- func_c
                  |          
                  |--49.95%-- func_a
                  |          |          
                  |          |--99.60%-- main
                  |          |          __libc_start_main
                  |           --0.40%-- [...]
                  |          
                  |--49.85%-- func_b
                  |          main
                  |          |          
                  |          |--99.60%-- __libc_start_main
                  |           --0.40%-- [...]
                   --0.20%-- [...]

To sum up, there is nothing the ksym tracer does that perf can't.

Well, may be perf doesn't offer the time ordered view of memory accesses, I must confess. Although this is still something we can easily provide if people want it.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

In the way for deprecation.

Is this like the KMEM_TRACER where trace events have superseded it?

It's a bit more complicated. This is a tracer that is able to produce statistics on top of workqueue events. You'll get the number of events queued and executed per workqueues. This gives some clues about their load. Past patches brought the ground to get the average/max time of execution, the works that took most time to complete, etc... But they didn't make it because they were growing too much the size and complexity of the code while a post processing in userspace would be better suited for that.

So the current version only displays the very basic informations of the number of works queued and executed.

This is something we could replace with a script in perf tools that analyse the workqueue events, but I'm not even sure it's worth now that the new cmwq workqueues may come upstream. The workqueue tracing code was there to analyse the latencies induced by works that block, which wouldn't be a problem anymore with cmwq.

So, what I think I'm going to do is to remove this workqueue statistics code from the kernel and if people complain for the loss, I'll write this script for perf as a replacement.

So you can expect CONFIG_WORKQUEUE_TRACER will be removed for 2.6.36 or so.

#12 lore raw
Quoted 3 lines · Frederic Weisbecker
Quoted 2 lines · Chase Douglas
Quoted 1 line · Frederic Weisbecker

On Tue, May 25, 2010 at 03:31:46PM -0400, Chase Douglas wrote:

<snipped>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

IMO, it is deprecated. The perf interface is much more powerful and flexible. Prasad, do you agree if I remove this ftrace plugin?

Sure, go ahead.

Quoted 85 lines
Quoted 4 lines

If there isn't any use in enabling it due to perf's features, then we can turn it off. However, if there's any use to be gained by this over perf's features, then I'd prefer to leave it on. Thoughts?

No, perf does much more:

- stacktraces recording
- "top" alike view with perf top
- stat with perf stat, etc...
- userspace memory accesses

Here is a quick example:

$ cat test.c
int var;

void func_c(void)

{
	var++;
}

void func_b(void)

{
	func_c();
}

void func_a(void)

{
	func_c();
}

int main(int argc, char **argv)

{
	int i;

	for (i = 0; i < 1000; i++)
		if (i % 2)
			func_a();
		else
			func_b();

	return 0;
}

//end test.c

$ gcc test.c -fno-omit-frame-pointer -o test
$ readelf -s test | grep var
    74: 000000000060102c     4 OBJECT  GLOBAL DEFAULT   25 var
$ perf record -g -c 1 -e mem:0x000000000060102c:w ./test
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.069 MB perf.data (~3020 samples) ]
$ perf report

# Events: 1K cycles
#

# Overhead  Command      Shared Object  Symbol
# ........  .......  .................  ......
#
    99.90%     test  test               [.] func_c
               |
               --- func_c
                  |          
                  |--49.95%-- func_a
                  |          |          
                  |          |--99.60%-- main
                  |          |          __libc_start_main
                  |           --0.40%-- [...]
                  |          
                  |--49.85%-- func_b
                  |          main
                  |          |          
                  |          |--99.60%-- __libc_start_main
                  |           --0.40%-- [...]
                   --0.20%-- [...]
Quoted 2 lines

To sum up, there is nothing the ksym tracer does that perf can't.

I second Frederic's opinion on this.

Thanks,
K.Prasad

Quoted 4 lines

Well, may be perf doesn't offer the time ordered view of memory accesses, I must confess. Although this is still something we can easily provide if people want it.

#13 lore raw
Quoted 11 lines · K.Prasad
Quoted 3 lines · Frederic Weisbecker
Quoted 2 lines · Chase Douglas
Quoted 1 line · Frederic Weisbecker

On Tue, May 25, 2010 at 03:31:46PM -0400, Chase Douglas wrote:

<snipped>

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

IMO, it is deprecated. The perf interface is much more powerful and flexible. Prasad, do you agree if I remove this ftrace plugin?

Sure, go ahead.

Great, I'll do that soon.

Thanks.

#14 lore raw
In reply to Chase Douglas
Quoted 30 lines · On Tue, 25 May 2010 17:09:59 -0400 Chase Douglas wrote:
Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines · Chase Douglas

The following options are what I am looking to set for our x86 configurations. I've only included those that I am not 100% sure of. Comments are what I could gather from documentation and Kconfig, but they may not be accurate:

<snip>

Quoted 5 lines
Quoted 1 line

# CONFIG_SCHED_TRACER is not set (headed for deprecation?)

We want to deprecate it in the long term, but for now we don't have any replacement. Cool for RT latency tracing.

I thought that the functionality is the same as what you get by:

echo 1 > (debufs)/tracing/events/sched/enable

Quoted 5 lines
Quoted 1 line

CONFIG_KSYM_TRACER=y (no performance impact by default)

IMO, it is deprecated. The perf interface is much more powerful and flexible. Prasad, do you agree if I remove this ftrace plugin?

If there isn't any use in enabling it due to perf's features, then we can turn it off. However, if there's any use to be gained by this over perf's features, then I'd prefer to leave it on. Thoughts?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

CONFIG_WORKQUEUE_TRACER=y (no performance impact by default)

In the way for deprecation.

Is this like the KMEM_TRACER where trace events have superseded it?

Hi Chase,

Would you mind sharing your final tracing config options?

thanks,
---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#15 lore raw
Quoted 3 lines · Randy Dunlap

Hi Chase,

Would you mind sharing your final tracing config options?

Sure.

Our source tree for Maverick encompasses all our major supported platforms in one git branch. Thus, we have one common config file and a bunch of arch and flavour (generic, server, virtual, pae) specific config files. One note about the common configs: if a config option would not exist because an option in a specific config file is set a certain way, then you can ignore it for that specific arch/flavour. For example, the common config has CONFIG_DYNAMIC_FTRACE=y, but the armel config has "CONFIG_FUNCTION_TRACER is not set" so the former would not have any affect on armel builds.

You can find the source branch at
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-maverick.git;a=summary

$ grep -Ir "TRAC[EI]" debian.master/config
./amd64/config.common.amd64:CONFIG_FUNCTION_TRACER=y
./amd64/config.common.amd64:CONFIG_STACK_TRACER=y
./i386/config.common.i386:CONFIG_FUNCTION_TRACER=y
./i386/config.common.i386:CONFIG_STACK_TRACER=y
./armel/config.common.armel:# CONFIG_FUNCTION_TRACER is not set
./armel/config.common.armel:# CONFIG_STACK_TRACER is not set
./config.common.ubuntu:# CONFIG_BACKTRACE_SELF_TEST is not set
./config.common.ubuntu:CONFIG_BLK_DEV_IO_TRACE=y
./config.common.ubuntu:# CONFIG_BOOT_TRACER is not set
./config.common.ubuntu:CONFIG_CAN_PM_TRACE=y
./config.common.ubuntu:CONFIG_CAPI_TRACE=y
./config.common.ubuntu:CONFIG_CONTEXT_SWITCH_TRACER=y
./config.common.ubuntu:CONFIG_DYNAMIC_FTRACE=y
./config.common.ubuntu:CONFIG_EVENT_TRACING=y
./config.common.ubuntu:CONFIG_FTRACE=y
./config.common.ubuntu:CONFIG_FTRACE_MCOUNT_RECORD=y
./config.common.ubuntu:CONFIG_FTRACE_NMI_ENTER=y
./config.common.ubuntu:# CONFIG_FTRACE_STARTUP_TEST is not set
./config.common.ubuntu:CONFIG_FTRACE_SYSCALLS=y
./config.common.ubuntu:CONFIG_FUNCTION_GRAPH_TRACER=y
./config.common.ubuntu:CONFIG_GENERIC_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_ARCH_TRACEHOOK=y
./config.common.ubuntu:CONFIG_HAVE_DYNAMIC_FTRACE=y
./config.common.ubuntu:CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
./config.common.ubuntu:CONFIG_HAVE_FTRACE_NMI_ENTER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
./config.common.ubuntu:CONFIG_HAVE_MMIOTRACE_SUPPORT=y
./config.common.ubuntu:CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
./config.common.ubuntu:# CONFIG_IRQSOFF_TRACER is not set
./config.common.ubuntu:CONFIG_IWLWIFI_DEVICE_TRACING=y
./config.common.ubuntu:# CONFIG_KMEMTRACE is not set
./config.common.ubuntu:# CONFIG_KSYM_TRACER is not set
./config.common.ubuntu:CONFIG_MMIOTRACE=y
./config.common.ubuntu:# CONFIG_MMIOTRACE_TEST is not set
./config.common.ubuntu:CONFIG_NETFILTER_XT_TARGET_TRACE=m
./config.common.ubuntu:CONFIG_NOP_TRACER=y
./config.common.ubuntu:CONFIG_PM_TRACE=y
./config.common.ubuntu:CONFIG_PM_TRACE_RTC=y
./config.common.ubuntu:# CONFIG_RCU_TRACE is not set
./config.common.ubuntu:CONFIG_SCHED_TRACER=y
./config.common.ubuntu:# CONFIG_SCSI_IPR_TRACE is not set
./config.common.ubuntu:CONFIG_STACKTRACE=y
./config.common.ubuntu:CONFIG_STACKTRACE_SUPPORT=y
./config.common.ubuntu:# CONFIG_SYSPROF_TRACER is not set
./config.common.ubuntu:CONFIG_TRACEPOINTS=y
./config.common.ubuntu:CONFIG_TRACER_MAX_TRACE=y
./config.common.ubuntu:CONFIG_TRACE_IRQFLAGS_SUPPORT=y
./config.common.ubuntu:CONFIG_TRACING=y
./config.common.ubuntu:CONFIG_TRACING_SUPPORT=y
./config.common.ubuntu:# CONFIG_TREE_RCU_TRACE is not set
./config.common.ubuntu:CONFIG_USER_STACKTRACE_SUPPORT=y
./config.common.ubuntu:# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
./config.common.ubuntu:# CONFIG_WORKQUEUE_TRACER is not set

Some of the above is not FTRACE. I also snipped the matches in config.common.ports because they are slightly different due to what the ports architectures support, but they should be pretty similar.

Please let me know if anything stands out at you as incorrect.

Thanks,

-- Chase

#16 lore raw
Quoted 77 lines · On Tue, 08 Jun 2010 18:00:34 -0400 Chase Douglas wrote:
Quoted 3 lines · Randy Dunlap

Hi Chase,

Would you mind sharing your final tracing config options?

Sure.

Our source tree for Maverick encompasses all our major supported platforms in one git branch. Thus, we have one common config file and a bunch of arch and flavour (generic, server, virtual, pae) specific config files. One note about the common configs: if a config option would not exist because an option in a specific config file is set a certain way, then you can ignore it for that specific arch/flavour. For example, the common config has CONFIG_DYNAMIC_FTRACE=y, but the armel config has "CONFIG_FUNCTION_TRACER is not set" so the former would not have any affect on armel builds.

You can find the source branch at
http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-maverick.git;a=summary

$ grep -Ir "TRAC[EI]" debian.master/config
./amd64/config.common.amd64:CONFIG_FUNCTION_TRACER=y
./amd64/config.common.amd64:CONFIG_STACK_TRACER=y
./i386/config.common.i386:CONFIG_FUNCTION_TRACER=y
./i386/config.common.i386:CONFIG_STACK_TRACER=y
./armel/config.common.armel:# CONFIG_FUNCTION_TRACER is not set
./armel/config.common.armel:# CONFIG_STACK_TRACER is not set
./config.common.ubuntu:# CONFIG_BACKTRACE_SELF_TEST is not set
./config.common.ubuntu:CONFIG_BLK_DEV_IO_TRACE=y
./config.common.ubuntu:# CONFIG_BOOT_TRACER is not set
./config.common.ubuntu:CONFIG_CAN_PM_TRACE=y
./config.common.ubuntu:CONFIG_CAPI_TRACE=y
./config.common.ubuntu:CONFIG_CONTEXT_SWITCH_TRACER=y
./config.common.ubuntu:CONFIG_DYNAMIC_FTRACE=y
./config.common.ubuntu:CONFIG_EVENT_TRACING=y
./config.common.ubuntu:CONFIG_FTRACE=y
./config.common.ubuntu:CONFIG_FTRACE_MCOUNT_RECORD=y
./config.common.ubuntu:CONFIG_FTRACE_NMI_ENTER=y
./config.common.ubuntu:# CONFIG_FTRACE_STARTUP_TEST is not set
./config.common.ubuntu:CONFIG_FTRACE_SYSCALLS=y
./config.common.ubuntu:CONFIG_FUNCTION_GRAPH_TRACER=y
./config.common.ubuntu:CONFIG_GENERIC_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_ARCH_TRACEHOOK=y
./config.common.ubuntu:CONFIG_HAVE_DYNAMIC_FTRACE=y
./config.common.ubuntu:CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
./config.common.ubuntu:CONFIG_HAVE_FTRACE_NMI_ENTER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_TRACER=y
./config.common.ubuntu:CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
./config.common.ubuntu:CONFIG_HAVE_MMIOTRACE_SUPPORT=y
./config.common.ubuntu:CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
./config.common.ubuntu:# CONFIG_IRQSOFF_TRACER is not set
./config.common.ubuntu:CONFIG_IWLWIFI_DEVICE_TRACING=y
./config.common.ubuntu:# CONFIG_KMEMTRACE is not set
./config.common.ubuntu:# CONFIG_KSYM_TRACER is not set
./config.common.ubuntu:CONFIG_MMIOTRACE=y
./config.common.ubuntu:# CONFIG_MMIOTRACE_TEST is not set
./config.common.ubuntu:CONFIG_NETFILTER_XT_TARGET_TRACE=m
./config.common.ubuntu:CONFIG_NOP_TRACER=y
./config.common.ubuntu:CONFIG_PM_TRACE=y
./config.common.ubuntu:CONFIG_PM_TRACE_RTC=y
./config.common.ubuntu:# CONFIG_RCU_TRACE is not set
./config.common.ubuntu:CONFIG_SCHED_TRACER=y
./config.common.ubuntu:# CONFIG_SCSI_IPR_TRACE is not set
./config.common.ubuntu:CONFIG_STACKTRACE=y
./config.common.ubuntu:CONFIG_STACKTRACE_SUPPORT=y
./config.common.ubuntu:# CONFIG_SYSPROF_TRACER is not set
./config.common.ubuntu:CONFIG_TRACEPOINTS=y
./config.common.ubuntu:CONFIG_TRACER_MAX_TRACE=y
./config.common.ubuntu:CONFIG_TRACE_IRQFLAGS_SUPPORT=y
./config.common.ubuntu:CONFIG_TRACING=y
./config.common.ubuntu:CONFIG_TRACING_SUPPORT=y
./config.common.ubuntu:# CONFIG_TREE_RCU_TRACE is not set
./config.common.ubuntu:CONFIG_USER_STACKTRACE_SUPPORT=y
./config.common.ubuntu:# CONFIG_VXGE_DEBUG_TRACE_ALL is not set
./config.common.ubuntu:# CONFIG_WORKQUEUE_TRACER is not set
Quoted 5 lines

Some of the above is not FTRACE. I also snipped the matches in config.common.ports because they are slightly different due to what the ports architectures support, but they should be pretty similar.

Please let me know if anything stands out at you as incorrect.

I'm a bit surprised by one (family) of kconfig symbols here:

PM_TRACE=y, PM_TRACE_RTC=y

That enables low-level kernel developer debugging code.

Other than that, the rest make sense to me.

thanks. [I don't know about that "armel" stuff.]

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

#17 lore raw
Quoted 7 lines · Randy Dunlap

I'm a bit surprised by one (family) of kconfig symbols here:

PM_TRACE=y, PM_TRACE_RTC=y

That enables low-level kernel developer debugging code.

Other than that, the rest make sense to me.

We try to focus a lot on the laptop use case, and so we do a bit of work on suspend/resume bugs. For those unaware, these options allow one to save the very last PM event point inside the RTC. If a suspend dies we may be able to get some good information on where it died using this tool. The functionality has to be manually enabled at runtime, otherwise the RTC would get scribbled over during every resume. Since it doesn't have any impact in the general use case, we leave it enabled by default. We already build too many test kernels for other types of bugs :).

Thanks,

-- Chase

#18 lore raw
In reply to Frederic Weisbecker
Quoted 4 lines · Frederic Weisbecker
Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Deprecated. We have the kmem trace event that are a full replacement now. Pekka, Gabriel, can we remove it now?

I don't think ftrace supports boot-time tracing which kmemtrace did. That said, I was probably the only one actually using the feature so maybe we can just nuke kmemtrace at this point...

#19 lore raw
Quoted 7 lines · Pekka Enberg
Quoted 3 lines · Frederic Weisbecker
Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Deprecated. We have the kmem trace event that are a full replacement now. Pekka, Gabriel, can we remove it now?

I don't think ftrace supports boot-time tracing which kmemtrace did.

We can do boot-time tracing by passing "trace_event=" kernel parameter.

By passing "ftrace=kmemtrace", kmemtrace can be started when calling tracer_alloc_buffer(), which is an early_initcall. While trace events are inititialized as a fs_initcall, it can be modified to an early_initcall.

Furthermore, I noticed the discussion on perf persistent events, which seems to enable perf trace at boot time, so I think perf-kmem can take advantage of this and will be a full-replacement of kmemtrace soon ?

Quoted 2 lines

That said, I was probably the only one actually using the feature so maybe we can just nuke kmemtrace at this point...

If you agree on removing kmemtrace now, I'll re-send the patch.

Acked-by
#20 lore raw
Quoted 4 lines · Li Zefan
Quoted 2 lines

That said, I was probably the only one actually using the feature so maybe we can just nuke kmemtrace at this point...

If you agree on removing kmemtrace now, I'll re-send the patch.

Sounds good to me.

#21 lore raw
In reply to Li Zefan
Quoted 16 lines · Li Zefan
Quoted 7 lines · Pekka Enberg
Quoted 3 lines · Frederic Weisbecker
Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Deprecated. We have the kmem trace event that are a full replacement now. Pekka, Gabriel, can we remove it now?

I don't think ftrace supports boot-time tracing which kmemtrace did.

We can do boot-time tracing by passing "trace_event=" kernel parameter.

By passing "ftrace=kmemtrace", kmemtrace can be started when calling tracer_alloc_buffer(), which is an early_initcall. While trace events are inititialized as a fs_initcall, it can be modified to an early_initcall.

Quoted 8 lines

Furthermore, I noticed the discussion on perf persistent events, which seems to enable perf trace at boot time, so I think perf-kmem can take advantage of this and will be a full-replacement of kmemtrace soon ?

Quoted 2 lines

That said, I was probably the only one actually using the feature so maybe we can just nuke kmemtrace at this point...

If you agree on removing kmemtrace now, I'll re-send the patch.

I don't understand... wasn't the old kmemtrace (the relayfs based stuff) removed a long time ago? Right now it's just the hooks and code that sets up tracing and printers.

Or does it happen that that code isn't necessary for 'perf kmem' to do its job?

And yeah, I'm all for the perf stuff, kmemtrace-user (the out-of-tree userspace tools) has been obsolete since the shift to ftrace.

As for early tracing, that was possible with the relayfs variant, it required only the SLAB layer to be up IIRC. But that stopped working once it got converted to ftrace (kmemtrace_init() is a nop).

Eduard

#22 lore raw
Quoted 21 lines · Eduard - Gabriel Munteanu
Quoted 20 lines · Li Zefan
Quoted 6 lines · Pekka Enberg
Quoted 3 lines · Frederic Weisbecker
Quoted 1 line

# CONFIG_KMEMTRACE is not set (Kconfig says N if unsure)

Deprecated. We have the kmem trace event that are a full replacement now. Pekka, Gabriel, can we remove it now?

I don't think ftrace supports boot-time tracing which kmemtrace did.

We can do boot-time tracing by passing "trace_event=" kernel parameter.

By passing "ftrace=kmemtrace", kmemtrace can be started when calling tracer_alloc_buffer(), which is an early_initcall. While trace events are inititialized as a fs_initcall, it can be modified to an early_initcall.

Furthermore, I noticed the discussion on perf persistent events, which seems to enable perf trace at boot time, so I think perf-kmem can take advantage of this and will be a full-replacement of kmemtrace soon ?

Quoted 2 lines

That said, I was probably the only one actually using the feature so maybe we can just nuke kmemtrace at this point...

Quoted 9 lines
Quoted 1 line

If you agree on removing kmemtrace now, I'll re-send the patch.

I don't understand... wasn't the old kmemtrace (the relayfs based stuff) removed a long time ago? Right now it's just the hooks and code that sets up tracing and printers.

Or does it happen that that code isn't necessary for 'perf kmem' to do its job?

Yes, actually the code has nothing to do with perf-kmem.

Quoted 7 lines

And yeah, I'm all for the perf stuff, kmemtrace-user (the out-of-tree userspace tools) has been obsolete since the shift to ftrace.

As for early tracing, that was possible with the relayfs variant, it required only the SLAB layer to be up IIRC. But that stopped working once it got converted to ftrace (kmemtrace_init() is a nop).

Ah, thanks for making this clear, so there's no such early tracing feature that acts as a barrier in removing kmemtrace.