Skip to contentSkip to conversation

Reading perf counters at ftrace trace boundaries

DISCUSSION Karim Yaghmour 3 participants 8 messages Aug 12, 2013 – Aug 13, 2013

lore
Share & tools
lore
#0 lore raw

Wondering if there's a way for reading perf counters in the kernel. I'd like to read/record perf counters on ftrace function tracing entries/exits to provide a rundown of the value of various counters on function call boundaries.

[ Steven: apologies for sending you a duplicate here of what I somewhat already sent privately. ]

Signature · 3 lines
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour
#1 lore raw
Quoted 4 lines · Karim Yaghmour

Wondering if there's a way for reading perf counters in the kernel. I'd like to read/record perf counters on ftrace function tracing entries/exits to provide a rundown of the value of various counters on function call boundaries.

KVM does it, see arch/x86/kvm/pmu.c. Essentially it would be doing RDPMC.

But the overhead will be likely very high, some sampling approach is likely better.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#2 lore raw
Quoted 1 line · Andi Kleen

KVM does it, see arch/x86/kvm/pmu.c. Essentially it would be doing RDPMC.

Thx for the pointer, appreciated.

Quoted 2 lines

But the overhead will be likely very high, some sampling approach is likely better.

Indeed. It doesn't actually have to be at every single ftrace begin/exit. But possibly starting with some kind of every nth and then drilling down as the culprit is incrementally singled-out.

Signature · 3 lines
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour
#3 lore raw
Quoted 3 lines

Indeed. It doesn't actually have to be at every single ftrace begin/exit. But possibly starting with some kind of every nth and then drilling down as the culprit is incrementally singled-out.

That's what normal sampling already does.

If you're worried about systematic shadow effects just randomize a bit.

-Andi

#4 lore raw
Quoted 3 lines · Andi Kleen

That's what normal sampling already does.

If you're worried about systematic shadow effects just randomize a bit.

That's actually the point. I'd like to be able to study/compare both approaches. I could be completely off, but I'd like to see if a divide and conquer approach (i.e. based on ftrace) wouldn't take the guesswork out of smart randomization. Just a hunch.

Signature · 3 lines
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour
#5 lore raw
In reply to Karim Yaghmour
Quoted 9 lines · Karim Yaghmour

Wondering if there's a way for reading perf counters in the kernel. I'd like to read/record perf counters on ftrace function tracing entries/exits to provide a rundown of the value of various counters on function call boundaries.

[ Steven: apologies for sending you a duplicate here of what I somewhat already sent privately. ]

If you want to base on ftrace, below two approach maybe take into use:

- register_ftrace_function/unregister_ftrace_function

- perf_event_create_kernel_counter (function event id is 1)

the first one is simplest, IMO.

You need to write your own kernel module to use these approach.

jovi.

#6 lore raw
Quoted 7 lines · zhangwei(Jovi)

If you want to base on ftrace, below two approach maybe take into use:

- register_ftrace_function/unregister_ftrace_function

- perf_event_create_kernel_counter (function event id is 1)

the first one is simplest, IMO.

Thx for the pointers.

Quoted 1 line

You need to write your own kernel module to use these approach.

As a proof-of-concept, sure. For something more permanent it would make more sense to adapt the various perf/ftrace tools to make this available on the command line with other options. But we're far away from that for the moment.

Signature · 3 lines
Karim Yaghmour
CEO - Opersys inc. / www.opersys.com
http://twitter.com/karimyaghmour
#7 lore raw
Quoted 10 lines · Karim Yaghmour
Quoted 7 lines · zhangwei(Jovi)

If you want to base on ftrace, below two approach maybe take into use:

- register_ftrace_function/unregister_ftrace_function

- perf_event_create_kernel_counter (function event id is 1)

the first one is simplest, IMO.

Quoted 9 lines

Thx for the pointers.

Quoted 1 line

You need to write your own kernel module to use these approach.

As a proof-of-concept, sure. For something more permanent it would make more sense to adapt the various perf/ftrace tools to make this available on the command line with other options. But we're far away from that for the moment.

If you want to embed pmu reading into ftrace/perf permanently, perhaps make pmu reading as clock source would be a nice way to go, actually the question you raised is very common for all of us in practical, sometimes we want to read cpu cycles both in kvm host and kvm guest, then in that kvm case, pmu reading for each event is very valuable.

(one issue need to handle specially for making pmu as tracing clock, is the pmu number overflow, because it relate with ftrace per-cpu ring buffer sync)

Thanks.

jovi