Skip to contentSkip to conversation

Benchmarks of kernel tracing options 2 (ftrace, lttng and perf)

DISCUSSION Douglas Santos 3 participants 5 messages Nov 17, 2010

lore
Share & tools
lore
#0 lore raw

Hi all,

This is a response to a benchmark, submitted a few weeks ago, comparing kernel tracing options.
http://lkml.org/lkml/2010/10/28/422

We followed the methodology described in the link bellow, but using the shellscripts posted there to reproduce autotest scripts.
http://lkml.org/lkml/2010/10/28/261

We disabled the extra syscall tracing on lttng, for a fair comparison.
http://lkml.org/lkml/2010/10/28/290

Average results with tracing "on":

lttng:  220 ns
ftrace: 260 ns
perf:   740 ns

E5405 system
kernel 2.6.36

-lttng 0.239 + 0.19.2modules + sys_getuid tracepoint + sys_getuid probe
+ remove syscall_trace

-ftrace and perf + sys_getuid tracepoint

#1 lore raw
Quoted 10 lines · Douglas Santos

Hi all,

This is a response to a benchmark, submitted a few weeks ago, comparing kernel tracing options.
http://lkml.org/lkml/2010/10/28/422

We followed the methodology described in the link bellow, but using the shellscripts posted there to reproduce autotest scripts.
http://lkml.org/lkml/2010/10/28/261

Quoted 7 lines

We disabled the extra syscall tracing on lttng, for a fair comparison.
http://lkml.org/lkml/2010/10/28/290

Average results with tracing "on":

lttng:  220 ns
ftrace: 260 ns

Heh, so ftrace got worse with the new kernel?

-- Steve

Quoted 10 lines

perf: 740 ns

E5405 system
kernel 2.6.36

(quoted diff) +0 -2
-lttng 0.239 + 0.19.2modules + sys_getuid tracepoint + sys_getuid probe
 + remove syscall_trace
 
-ftrace and perf + sys_getuid tracepoint
#2 lore raw

Quoting Steven Rostedt <[email protected]>:

Quoted 15 lines · Douglas Santos
Quoted 3 lines

Hi all,

This is a response to a benchmark, submitted a few weeks ago, comparing

kernel

Quoted 10 lines

tracing options.
http://lkml.org/lkml/2010/10/28/422

We followed the methodology described in the link bellow, but using the shellscripts posted there to reproduce autotest scripts.
http://lkml.org/lkml/2010/10/28/261

We disabled the extra syscall tracing on lttng, for a fair comparison.
http://lkml.org/lkml/2010/10/28/290

Quoted 6 lines
Quoted 4 lines

Average results with tracing "on":

lttng:  220 ns
ftrace: 260 ns

Heh, so ftrace got worse with the new kernel?

The previous bench was doing tracing "on" minus "off" average results. They also used autotest scripts, not sure if it does exactly the same thing.

I'll check if we missed something.

#3 lore raw
Quoted 22 lines · Douglas Santos

Quoting Steven Rostedt <[email protected]>:

Quoted 21 lines · Douglas Santos
Quoted 3 lines

Hi all,

This is a response to a benchmark, submitted a few weeks ago, comparing

kernel

Quoted 7 lines

tracing options.
http://lkml.org/lkml/2010/10/28/422

We followed the methodology described in the link bellow, but using the shellscripts posted there to reproduce autotest scripts.
http://lkml.org/lkml/2010/10/28/261

Quoted 7 lines

We disabled the extra syscall tracing on lttng, for a fair comparison.
http://lkml.org/lkml/2010/10/28/290

Average results with tracing "on":

lttng:  220 ns
ftrace: 260 ns

Heh, so ftrace got worse with the new kernel?

Steve, can you explain how you're drawing that conclusion? Did Douglas run this benchmark before on a previous kernel (I didn't see it if so)?

- You can't directly compare to my results because of different hardware.
- The methodology for lttng is different (syscall tracing was removed).
- My results were also on 2.6.36

Quoted 3 lines

The previous bench was doing tracing "on" minus "off" average results. They also used autotest scripts, not sure if it does exactly the same thing.

I think the subtraction is important, or it is at least important to see what the "off" result is as a baseline of comparison. Otherwise, a huge portion of the measurement is the cost of making the syscall itself.

Quoted 5 lines

I'll check if we missed something.

#4 lore raw
Quoted 20 lines · David Sharp
Quoted 18 lines · Douglas Santos

Quoting Steven Rostedt <[email protected]>:

Quoted 17 lines · Douglas Santos
Quoted 3 lines

Hi all,

This is a response to a benchmark, submitted a few weeks ago, comparing

kernel

Quoted 12 lines

tracing options.
http://lkml.org/lkml/2010/10/28/422

We followed the methodology described in the link bellow, but using the shellscripts posted there to reproduce autotest scripts.
http://lkml.org/lkml/2010/10/28/261

We disabled the extra syscall tracing on lttng, for a fair comparison.
http://lkml.org/lkml/2010/10/28/290

Average results with tracing "on":

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
lttng:  220 ns
ftrace: 260 ns

Heh, so ftrace got worse with the new kernel?

Steve, can you explain how you're drawing that conclusion? Did Douglas run this benchmark before on a previous kernel (I didn't see it if so)?

Oops, no, I was thinking that this was from your tests. I remember asking you to try the new kernel. I think I got you and Douglas confused :-)

-- Steve