Skip to contentSkip to conversation

perf probe: Support globbing/regex in -a

RFE Arnaldo Carvalho de Melo 2 participants 5 messages Apr 8, 2015 – Apr 11, 2015

lore
Share & tools
lore
#0 lore raw

Hi Masami,

In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:

[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#

And get the same result that I get from:

[root@zoo ~]# perf probe -x ~/bin/perf perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u Added new event:
probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)

You can now use it in all perf tools, such as:

perf record -e probe_perf:call_path__new -aR sleep 1

Added new event:
probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)

You can now use it in all perf tools, such as:

perf record -e probe_perf:call_path_root__new -aR sleep 1

Added new event:
probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)

<SNIP>

I.e. that I end up with:

[root@zoo ~]# perf probe -l 2>&1 | grep probe_perf:.*__new
probe_perf:call_path__new (on call_path__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:call_path_root__new (on call_path_root__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:call_return_processor__new (on call_return_processor__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:comm__new (on comm__new@util/comm.c in /home/acme/bin/perf) probe_perf:cpu_map__new (on cpu_map__new@util/cpumap.c in /home/acme/bin/perf) probe_perf:debuginfo__new (on debuginfo__new@util/probe-finder.c in /home/acme/bin/perf) probe_perf:disasm_line__new (on disasm_line__new@util/annotate.c in /home/acme/bin/perf) probe_perf:dso__new (on dso__new@util/dso.c in /home/acme/bin/perf)
probe_perf:dso__new_map (on dso__new_map@util/dso.c in /home/acme/bin/perf) probe_perf:fdarray__new (on fdarray__new@fd/array.c in /home/acme/bin/perf) probe_perf:hist_browser__new (on hist_browser__new@ui/browsers/hists.c in /home/acme/bin/perf)
<SNIP>
probe_perf:thread_map__new_by_uid (on thread_map__new_by_uid@util/thread_map.c in /home/acme/bin/perf) probe_perf:thread_map__new_dummy (on thread_map__new_dummy@util/thread_map.c in /home/acme/bin/perf) probe_perf:thread_map__new_str (on thread_map__new_str@util/thread_map.c in /home/acme/bin/perf)
[root@zoo ~]#

Some more goodies to have, please consider better compressing the output of successfully installed probes, Something like:

[root@zoo ~]# perf probe -x ~/bin/perf perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u Added 46 new events:
probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf) probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf) probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
<SNIP>

You can now use them in all perf tools, e.g.:

perf record -e probe_perf:call_path_root__new -aR sleep 1

------

Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:

[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.
[root@zoo ~]#

Fails, but:

[root@zoo ~]# perf list *:*switch*

List of pre-defined events (to be used in -e):

  cfg80211:cfg80211_ch_switch_notify                 [Tracepoint event]
  cfg80211:rdev_channel_switch                       [Tracepoint event]
  mac80211:api_chswitch_done                         [Tracepoint event]
  mac80211:drv_channel_switch                        [Tracepoint event]
  mac80211:drv_channel_switch_beacon                 [Tracepoint event]
  mac80211:drv_switch_vif_chanctx                    [Tracepoint event]
  sched:sched_switch                                 [Tracepoint event]

[root@zoo ~]#

Works :-)

- Arnaldo

#1 lore raw

Hi Arnaldo,

Quoted 10 lines · Arnaldo Carvalho de Melo

Hi Masami,

In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:

[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#

Actually, I had sent it :)

https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213

I'll update and resend the wildcard support.

Quoted 58 lines

And get the same result that I get from:

[root@zoo ~]# perf probe -x ~/bin/perf perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u Added new event:
probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf)

You can now use it in all perf tools, such as:

perf record -e probe_perf:call_path__new -aR sleep 1

Added new event:
probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf)

You can now use it in all perf tools, such as:

perf record -e probe_perf:call_path_root__new -aR sleep 1

Added new event:
probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)

<SNIP>

I.e. that I end up with:

[root@zoo ~]# perf probe -l 2>&1 | grep probe_perf:.*__new
probe_perf:call_path__new (on call_path__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:call_path_root__new (on call_path_root__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:call_return_processor__new (on call_return_processor__new@util/thread-stack.c in /home/acme/bin/perf) probe_perf:comm__new (on comm__new@util/comm.c in /home/acme/bin/perf) probe_perf:cpu_map__new (on cpu_map__new@util/cpumap.c in /home/acme/bin/perf) probe_perf:debuginfo__new (on debuginfo__new@util/probe-finder.c in /home/acme/bin/perf) probe_perf:disasm_line__new (on disasm_line__new@util/annotate.c in /home/acme/bin/perf) probe_perf:dso__new (on dso__new@util/dso.c in /home/acme/bin/perf)
probe_perf:dso__new_map (on dso__new_map@util/dso.c in /home/acme/bin/perf) probe_perf:fdarray__new (on fdarray__new@fd/array.c in /home/acme/bin/perf) probe_perf:hist_browser__new (on hist_browser__new@ui/browsers/hists.c in /home/acme/bin/perf)
<SNIP>
probe_perf:thread_map__new_by_uid (on thread_map__new_by_uid@util/thread_map.c in /home/acme/bin/perf) probe_perf:thread_map__new_dummy (on thread_map__new_dummy@util/thread_map.c in /home/acme/bin/perf) probe_perf:thread_map__new_str (on thread_map__new_str@util/thread_map.c in /home/acme/bin/perf)
[root@zoo ~]#

Some more goodies to have, please consider better compressing the output of successfully installed probes, Something like:

[root@zoo ~]# perf probe -x ~/bin/perf perf probe -x ~/bin/perf -F | grep __new | sed 's/^/-a /g' | sort -u Added 46 new events:
probe_perf:call_path__new (on call_path__new in /home/acme/bin/perf) probe_perf:call_path_root__new (on call_path_root__new in /home/acme/bin/perf) probe_perf:call_return_processor__new (on call_return_processor__new in /home/acme/bin/perf)
<SNIP>

You can now use them in all perf tools, e.g.:

perf record -e probe_perf:call_path_root__new -aR sleep 1

Quoted 8 lines

------

Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:

[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Hmm, right. This should be fixed.

Thank you!

Quoted 22 lines

[root@zoo ~]#

Fails, but:

[root@zoo ~]# perf list *:*switch*

List of pre-defined events (to be used in -e):

  cfg80211:cfg80211_ch_switch_notify                 [Tracepoint event]
  cfg80211:rdev_channel_switch                       [Tracepoint event]
  mac80211:api_chswitch_done                         [Tracepoint event]
  mac80211:drv_channel_switch                        [Tracepoint event]
  mac80211:drv_channel_switch_beacon                 [Tracepoint event]
  mac80211:drv_switch_vif_chanctx                    [Tracepoint event]
  sched:sched_switch                                 [Tracepoint event]

[root@zoo ~]#

Works :-)

- Arnaldo

Signature · 5 lines
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]
#2 lore raw

Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:

Quoted 14 lines

Hi Arnaldo,

Quoted 10 lines · Arnaldo Carvalho de Melo

Hi Masami,

In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:

[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#

Quoted 6 lines

Actually, I had sent it :)

https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213

I'll update and resend the wildcard support.

Oops, but it was together with that cache thing, could it be sent separately?

<SNIP>

Quoted 8 lines
Quoted 6 lines

Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:

[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Hmm, right. This should be fixed.

Thanks!

- Arnaldo

#3 lore raw
Quoted 17 lines · Arnaldo Carvalho de Melo

Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:

Quoted 16 lines

Hi Arnaldo,

Quoted 10 lines · Arnaldo Carvalho de Melo

Hi Masami,

In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:

[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#

Actually, I had sent it :)

Quoted 7 lines
Quoted 4 lines

https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213

I'll update and resend the wildcard support.

Oops, but it was together with that cache thing, could it be sent separately?

Yes, it can be applied separately. However, it may open the Pandora's box, which we have to face to someday anyway :)

- "perf probe -a \*" can try to put probes on all functions in the kernel including the inline functions. The total number of probes will be more than one hundred-thousand (100,000)...
- Currently ftrace-kprobe doesn't limit the number of dynamic events, so it can cause out of memory on some machine.
- Also, enabling such events at once, could cause performance problem, which I had tackled last year. See below slide.
http://events.linuxfoundation.org/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
(note that this series is not applied yet)

So, currently perf probe limits the maximum number of definable events at once you can expand it with --max-probes anyway. And also, I've introduced --no-inlines option to perf probe which reduces the number of matched functions when user puts the probes with wild cards. (I'm considering to apply this option automatically if the function name includes wild cards)

Thank you,

Quoted 16 lines

<SNIP>

Quoted 8 lines
Quoted 6 lines

Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:

[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Hmm, right. This should be fixed.

Thanks!

- Arnaldo

Signature · 5 lines
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]
#4 lore raw

Em Sat, Apr 11, 2015 at 09:25:54AM +0900, Masami Hiramatsu escreveu:

Quoted 17 lines
Quoted 15 lines · Arnaldo Carvalho de Melo

Em Thu, Apr 09, 2015 at 02:27:52PM +0900, Masami Hiramatsu escreveu:

Quoted 14 lines
Quoted 8 lines · Arnaldo Carvalho de Melo

In tools/perf constructors are named <CLASS>__new, and right now I want
to know hoe many of each <CLASS> objects are being allocated, so I expected to be able to do:

[root@zoo ~]# perf probe -x ~/bin/perf -a "*__new" Probe point '*__new' not found.
Error: Failed to add events.
[root@zoo ~]#

Actually, I had sent it :)

https://lkml.org/lkml/2014/10/31/207
https://lkml.org/lkml/2014/10/31/213

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I'll update and resend the wildcard support.

Oops, but it was together with that cache thing, could it be sent separately?

Yes, it can be applied separately. However, it may open the Pandora's box, which we have to face to someday anyway :)

yeah!

Quoted 6 lines

- "perf probe -a \*" can try to put probes on all functions in the kernel including the inline functions. The total number of probes will be more than one hundred-thousand (100,000)...
- Currently ftrace-kprobe doesn't limit the number of dynamic events, so it can cause out of memory on some machine.
- Also, enabling such events at once, could cause performance problem,

So don't do it?

Quoted 9 lines

which I had tackled last year. See below slide.
http://events.linuxfoundation.org/sites/events/files/slides/Handling%20the%20Massive%20Multiple%20Kprobes%20v2_1.pdf
(note that this series is not applied yet)

So, currently perf probe limits the maximum number of definable events at once you can expand it with --max-probes anyway. And also, I've introduced --no-inlines option to perf probe which reduces the number of matched functions when user puts the probes with wild cards. (I'm considering to apply this option automatically if the function name includes wild cards)

I.e. we have some mechanisms in place to limit the number of probes, user gets warned when that limit is hit, no problem, right?

And on top of that, you already have a patchkit that tries to reduce the overhead, good deal! :-)

I'll look again at those patches and see if they apply.

Thanks for all the info about this topic, appreciated!

- Arnaldo

Quoted 27 lines

Thank you,

Quoted 16 lines

<SNIP>

Quoted 8 lines
Quoted 6 lines

Also that -l should output to stdout, so that we can grep it, also it would be nice if it -l worked like in perf list:

[root@zoo ~]# perf probe -l probe_perf:* Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Hmm, right. This should be fixed.

Thanks!

- Arnaldo

Signature · 5 lines
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]