perf probe: Support globbing/regex in -a
Hi Masami,
In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:
[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#
And get the same result that I get from:
[root@zoo ~]# perf probe -x ~/bin/perf perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u Added new event:
probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)
You can now use it in all perf tools, such as:
perf record -e probe_perf:call_path__new -aR sleep 1
Added new event:
probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)
You can now use it in all perf tools, such as:
perf record -e probe_perf:call_path_root__new -aR sleep 1
Added new event:
probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
<SNIP>
I.e. that I end up with:
[root@zoo ~]# perf probe -l 2>&1 | grep probe_perf:.*__new
probe_perf:call_path__new (on call_path__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:call_path_root__new (on call_path_root__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:call_return_processor__new (on call_return_processor__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:comm__new (on comm__new@util/comm.c in /home/acme/bin/perf) probe_perf:cpu_map__new (on cpu_map__new@util/cpumap.c in /home/acme/bin/perf) probe_perf:debuginfo__new (on debuginfo__new@util/probe-finder.c in /home/acme/bin/perf) probe_perf:disasm_line__new (on disasm_line__new@util/annotate.c in /home/acme/bin/perf) probe_perf:dso__new (on dso__new@util/dso.c in /home/acme/bin/perf)
probe_perf:dso__new_map (on dso__new_map@util/dso.c in /home/acme/bin/perf) probe_perf:fdarray__new (on fdarray__new@fd/array.c in /home/acme/bin/perf) probe_perf:hist_browser__new (on hist_browser__new@ui/browsers/hists.c in /home/acme/bin/perf)
<SNIP>
probe_perf:thread_map__new_by_uid (on thread_map__new_by_uid@util/thread_map.c in /home/acme/bin/perf) probe_perf:thread_map__new_dummy (on thread_map__new_dummy@util/thread_map.c in /home/acme/bin/perf) probe_perf:thread_map__new_str (on thread_map__new_str@util/thread_map.c in /home/acme/bin/perf)
[root@zoo ~]#
Some more goodies to have, please consider better compressing the output of successfully installed probes, Something like:
[root@zoo ~]# perf probe -x ~/bin/perf perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u Added 46 new events:
probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf) probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf) probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
<SNIP>
You can now use them in all perf tools, e.g.:
perf record -e probe_perf:call_path_root__new -aR sleep 1
------
Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:
[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
[root@zoo ~]#
Fails, but:
[root@zoo ~]# perf list *:*switch*
List of pre-defined events (to be used in -e):
cfg80211:cfg80211_ch_switch_notify [Tracepoint event] cfg80211:rdev_channel_switch [Tracepoint event] mac80211:api_chswitch_done [Tracepoint event] mac80211:drv_channel_switch [Tracepoint event] mac80211:drv_channel_switch_beacon [Tracepoint event] mac80211:drv_switch_vif_chanctx [Tracepoint event] sched:sched_switch [Tracepoint event]
[root@zoo ~]#
Works :-)
- Arnaldo
Hi Arnaldo,
Quoted 10 lines · Arnaldo Carvalho de Melo
Hi Masami,
In
tools/perfconstructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:[root@zoo ~]# perf probe -x ~/bin/perf -a "
*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#
Actually, I had sent it :)
https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213
I'll update and resend the wildcard support.
Quoted 58 lines
And get the same result that I get from:
[root@zoo ~]# perf probe -x ~/bin/perf
perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -uAdded new event:
probe_perf:call_path__new (oncall_path__newin/home/acme/bin/perf)You can now use it in all perf tools, such as:
perf record -e probe_perf:call_path__new -aR sleep 1
Added new event:
probe_perf:call_path_root__new (oncall_path_root__newin/home/acme/bin/perf)You can now use it in all perf tools, such as:
perf record -e probe_perf:call_path_root__new -aR sleep 1
Added new event:
probe_perf:call_return_processor__new (oncall_return_processor__newin/home/acme/bin/perf)<SNIP>
I.e. that I end up with:
[root@zoo ~]# perf probe -l 2>&1 | grep probe_perf:.*__new
probe_perf:call_path__new (oncall_path__new@util/thread-stack.cin/home/acme/bin/perf) probe_perf:call_path_root__new (oncall_path_root__new@util/thread-stack.cin/home/acme/bin/perf) probe_perf:call_return_processor__new (oncall_return_processor__new@util/thread-stack.cin/home/acme/bin/perf) probe_perf:comm__new (oncomm__new@util/comm.cin/home/acme/bin/perf) probe_perf:cpu_map__new (oncpu_map__new@util/cpumap.cin/home/acme/bin/perf) probe_perf:debuginfo__new (ondebuginfo__new@util/probe-finder.cin/home/acme/bin/perf) probe_perf:disasm_line__new (ondisasm_line__new@util/annotate.cin/home/acme/bin/perf) probe_perf:dso__new (ondso__new@util/dso.cin/home/acme/bin/perf)
probe_perf:dso__new_map (ondso__new_map@util/dso.cin/home/acme/bin/perf) probe_perf:fdarray__new (onfdarray__new@fd/array.cin/home/acme/bin/perf) probe_perf:hist_browser__new (onhist_browser__new@ui/browsers/hists.cin/home/acme/bin/perf)
<SNIP>
probe_perf:thread_map__new_by_uid (onthread_map__new_by_uid@util/thread_map.cin/home/acme/bin/perf) probe_perf:thread_map__new_dummy (onthread_map__new_dummy@util/thread_map.cin/home/acme/bin/perf) probe_perf:thread_map__new_str (onthread_map__new_str@util/thread_map.cin/home/acme/bin/perf)
[root@zoo ~]#Some more goodies to have, please consider better compressing the output of successfully installed probes, Something like:
[root@zoo ~]# perf probe -x ~/bin/perf
perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -uAdded 46 new events:
probe_perf:call_path__new (oncall_path__newin/home/acme/bin/perf) probe_perf:call_path_root__new (oncall_path_root__newin/home/acme/bin/perf) probe_perf:call_return_processor__new (oncall_return_processor__newin/home/acme/bin/perf)
<SNIP>You can now use them in all perf tools, e.g.:
perf record -e probe_perf:call_path_root__new -aR sleep 1
Quoted 8 lines
------
Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:
[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
Hmm, right. This should be fixed.
Thank you!
Quoted 22 lines
[root@zoo ~]#
Fails, but:
[root@zoo ~]# perf list *:*switch*
List of pre-defined events (to be used in -e):
cfg80211:cfg80211_ch_switch_notify [Tracepoint event] cfg80211:rdev_channel_switch [Tracepoint event] mac80211:api_chswitch_done [Tracepoint event] mac80211:drv_channel_switch [Tracepoint event] mac80211:drv_channel_switch_beacon [Tracepoint event] mac80211:drv_switch_vif_chanctx [Tracepoint event] sched:sched_switch [Tracepoint event][root@zoo ~]#
Works :-)
- Arnaldo
Signature · 5 lines
Masami HIRAMATSU Linux Technology Research Center, System Productivity Research Dept. Center for Technology Innovation - Systems Engineering Hitachi, Ltd., Research & Development Group E-mail: [email protected]
Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:
Quoted 14 lines
Hi Arnaldo,
Quoted 10 lines · Arnaldo Carvalho de Melo
Hi Masami,
In
tools/perfconstructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:[root@zoo ~]# perf probe -x ~/bin/perf -a "
*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#
Quoted 6 lines
Actually, I had sent it :)
https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213I'll update and resend the wildcard support.
Oops, but it was together with that cache thing, could it be sent separately?
<SNIP>
Quoted 8 lines
Quoted 6 lines
Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:
[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.Hmm, right. This should be fixed.
Thanks!
- Arnaldo
Quoted 17 lines · Arnaldo Carvalho de Melo
Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:
Quoted 16 lines
Hi Arnaldo,
Quoted 10 lines · Arnaldo Carvalho de Melo
Hi Masami,
In
tools/perfconstructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:[root@zoo ~]# perf probe -x ~/bin/perf -a "
*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#Actually, I had sent it :)
Quoted 7 lines
Quoted 4 lines
https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213I'll update and resend the wildcard support.
Oops, but it was together with that cache thing, could it be sent separately?
Yes, it can be applied separately. However, it may open the Pandora's box, which we have to face to someday anyway :)
- "perf probe -a \*" can try to put probes on all functions in the kernel including the inline functions. The total number of probes will be more than one hundred-thousand (100,000)...
- Currently ftrace-kprobe doesn't limit the number of dynamic events, so it can cause out of memory on some machine.
- Also, enabling such events at once, could cause performance problem, which I had tackled last year. See below slide.
http://events.linuxfoundation.org/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
(note that this series is not applied yet)
So, currently perf probe limits the maximum number of definable events at once you can expand it with --max-probes anyway. And also, I've introduced --no-inlines option to perf probe which reduces the number of matched functions when user puts the probes with wild cards. (I'm considering to apply this option automatically if the function name includes wild cards)
Thank you,
Quoted 16 lines
<SNIP>
Quoted 8 lines
Quoted 6 lines
Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:
[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.Hmm, right. This should be fixed.
Thanks!
- Arnaldo
Signature · 5 lines
Masami HIRAMATSU Linux Technology Research Center, System Productivity Research Dept. Center for Technology Innovation - Systems Engineering Hitachi, Ltd., Research & Development Group E-mail: [email protected]
Em Sat, Apr 11, 2015 at 09:25:54AM +0900, Masami Hiramatsu escreveu:
Quoted 17 lines
Quoted 15 lines · Arnaldo Carvalho de Melo
Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:
Quoted 14 lines
Quoted 8 lines · Arnaldo Carvalho de Melo
In
tools/perfconstructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:[root@zoo ~]# perf probe -x ~/bin/perf -a "
*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#Actually, I had sent it :)
https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I'll update and resend the wildcard support.
Oops, but it was together with that cache thing, could it be sent separately?
Yes, it can be applied separately. However, it may open the Pandora's box, which we have to face to someday anyway :)
yeah!
Quoted 6 lines
- "perf probe -a \*" can try to put probes on all functions in the kernel including the inline functions. The total number of probes will be more than one hundred-thousand (100,000)...
- Currently ftrace-kprobe doesn't limit the number of dynamic events, so it can cause out of memory on some machine.
- Also, enabling such events at once, could cause performance problem,
So don't do it?
Quoted 9 lines
which I had tackled last year. See below slide.
http://events.linuxfoundation.org/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
(note that this series is not applied yet)So, currently perf probe limits the maximum number of definable events at once you can expand it with --max-probes anyway. And also, I've introduced --no-inlines option to perf probe which reduces the number of matched functions when user puts the probes with wild cards. (I'm considering to apply this option automatically if the function name includes wild cards)
I.e. we have some mechanisms in place to limit the number of probes, user gets warned when that limit is hit, no problem, right?
And on top of that, you already have a patchkit that tries to reduce the overhead, good deal! :-)
I'll look again at those patches and see if they apply.
Thanks for all the info about this topic, appreciated!
- Arnaldo
Quoted 27 lines
Thank you,
Quoted 16 lines
<SNIP>
Quoted 8 lines
Quoted 6 lines
Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:
[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.Hmm, right. This should be fixed.
Thanks!
- Arnaldo
Signature · 5 lines
Masami HIRAMATSU Linux Technology Research Center, System Productivity Research Dept. Center for Technology Innovation - Systems Engineering Hitachi, Ltd., Research & Development Group E-mail: [email protected]