perfcounter: callchains with perf report
cover perfcounter: callchains with perf report
Hi,
Here is a first shot for the sorted callchains per entries handling with per report.
I'll continue to improve it:
- symbol resolution
- profit we have a tree to display a better graph hierarchy
- let the user provide a limit for hit percentage, depth, number of backtraces, etc...
- better output
- colors
- and so on....
Thanks.
Frederic Weisbecker (2):
perfcounter: prepare a small callchain framework
perfcounter: print sorted callchains per histogram entries
tools/perf/Makefile | 1 + tools/perf/builtin-report.c | 87 +++++++++++++++++---- tools/perf/perf.h | 5 + tools/perf/util/callchain.c | 174 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/callchain.h | 33 ++++++++ 5 files changed, 284 insertions(+), 16 deletions(-) create mode 100644 tools/perf/util/callchain.c create mode 100644 tools/perf/util/callchain.h
PATCH 1/2 perfcounter: prepare a small callchain framework
We plan to display the callchains depending on some user-configurable parameters.
To gather the callchains stats from the recorded stream in a fast way, this patch introduces an ad hoc radix tree adapted for callchains and also a rbtree to sort these callchains once we have gathered every events from the stream.
--- tools/perf/Makefile | 1 + tools/perf/builtin-report.c | 5 - tools/perf/perf.h | 5 + tools/perf/util/callchain.c | 174 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/callchain.h | 33 ++++++++ 5 files changed, 213 insertions(+), 5 deletions(-) create mode 100644 tools/perf/util/callchain.c create mode 100644 tools/perf/util/callchain.h
tools/perf/Makefile +1 -0
@@ -329,6 +329,7 @@ LIB_OBJS += util/symbol.o LIB_OBJS += util/color.o LIB_OBJS += util/pager.o LIB_OBJS += util/header.o +LIB_OBJS += util/callchain.o BUILTIN_OBJS += builtin-annotate.o BUILTIN_OBJS += builtin-help.o
tools/perf/builtin-report.c +0 -5
@@ -62,11 +62,6 @@ struct ip_event {unsigned char __more_data[]; }; -struct ip_callchain { -u64 nr; -u64 ips[0 ]; -}; -struct mmap_event {struct perf_event_header header;u32 pid, tid;
tools/perf/perf.h +5 -0
@@ -72,4 +72,9 @@ sys_perf_counter_open(struct perf_counter_attr *attr,#define MAX_COUNTERS 256#define MAX_NR_CPUS 256 +struct ip_callchain { +u64 nr; +u64 ips[0 ]; +}; +#endif
tools/perf/util/callchain.c new file +174 -0
174 lines not on this page · raw message
tools/perf/util/callchain.h new file +33 -0
@@ -0,0 +1,33 @@ +#ifndef __PERF_CALLCHAIN_H +#define __PERF_CALLCHAIN_H + +#include "../perf.h" +#include "list.h" +#include "rbtree.h" + + +struct callchain_node { +struct callchain_node *parent; +struct list_head brothers; +struct list_head children; +struct list_head val; +struct rb_node rb_node; +int val_nr; +int hit; +}; + +struct callchain_list { +unsigned long ip; +struct list_head list; +}; + +static inline void callchain_init (struct callchain_node *node) +{ +INIT_LIST_HEAD (&node->brothers); +INIT_LIST_HEAD (&node->children); +INIT_LIST_HEAD (&node->val); +} + +void append_chain (struct callchain_node *root,struct ip_callchain *chain); +void sort_chain_to_rbtree (struct rb_root *rb_root,struct callchain_node *node); +#endif
Signature · 1 line
1.6.2.3
[tip:perfcounters/urgent] perf_counter tools: Prepare a small callchain framework
Commit-ID: 8cb76d99d715741637b6d0884f389e17e9cb05d2 Gitweb: http://git.kernel.org/tip/8cb76d99d715741637b6d0884f389e17e9cb05d2 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 26 Jun 2009 16:28:00 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 26 Jun 2009 16:47:00 +0200
perf_counter tools: Prepare a small callchain framework
We plan to display the callchains depending on some user-configurable parameters.
To gather the callchains stats from the recorded stream in a fast way, this patch introduces an ad hoc radix tree adapted for callchains and also a rbtree to sort these callchains once we have gathered every events from the stream.
LKML-Reference: <[email protected]>
--- tools/perf/Makefile | 1 + tools/perf/builtin-report.c | 5 - tools/perf/perf.h | 5 + tools/perf/util/callchain.c | 174 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/callchain.h | 33 ++++++++ 5 files changed, 213 insertions(+), 5 deletions(-)
tools/perf/Makefile +1 -0
@@ -329,6 +329,7 @@ LIB_OBJS += util/symbol.o LIB_OBJS += util/color.o LIB_OBJS += util/pager.o LIB_OBJS += util/header.o +LIB_OBJS += util/callchain.o BUILTIN_OBJS += builtin-annotate.o BUILTIN_OBJS += builtin-help.o
tools/perf/builtin-report.c +0 -5
@@ -62,11 +62,6 @@ struct ip_event {unsigned char __more_data[]; }; -struct ip_callchain { -u64 nr; -u64 ips[0 ]; -}; -struct mmap_event {struct perf_event_header header;u32 pid, tid;
tools/perf/perf.h +5 -0
@@ -66,4 +66,9 @@ sys_perf_counter_open(struct perf_counter_attr *attr,#define MAX_COUNTERS 256#define MAX_NR_CPUS 256 +struct ip_callchain { +u64 nr; +u64 ips[0 ]; +}; +#endif
tools/perf/util/callchain.c new file +174 -0
174 lines not on this page · raw message
tools/perf/util/callchain.h new file +33 -0
@@ -0,0 +1,33 @@ +#ifndef __PERF_CALLCHAIN_H +#define __PERF_CALLCHAIN_H + +#include "../perf.h" +#include "list.h" +#include "rbtree.h" + + +struct callchain_node { +struct callchain_node *parent; +struct list_head brothers; +struct list_head children; +struct list_head val; +struct rb_node rb_node; +int val_nr; +int hit; +}; + +struct callchain_list { +unsigned long ip; +struct list_head list; +}; + +static inline void callchain_init (struct callchain_node *node) +{ +INIT_LIST_HEAD (&node->brothers); +INIT_LIST_HEAD (&node->children); +INIT_LIST_HEAD (&node->val); +} + +void append_chain (struct callchain_node *root,struct ip_callchain *chain); +void sort_chain_to_rbtree (struct rb_root *rb_root,struct callchain_node *node); +#endif
PATCH 2/2 perfcounter: print sorted callchains per histogram entries
In reply to Frederic WeisbeckerUse the newly created callchains radix tree to gather the chains stats from the recorded events and then print the callchains for all of them, sorted by hits, using the "-c" parameter with perf report.
Example:
66.15% [k] atm_clip_exit
63.08%
0xffffffffffffff80
0xffffffff810196a8
0xffffffff810c14c8
0xffffffff8101a79c
0xffffffff810194f3
0xffffffff8106ab7f
0xffffffff8106abe5
0xffffffff8106acde
0xffffffff8100d94b
0xffffffff8153e7ea
[...]
1.54%
0xffffffffffffff80
0xffffffff810196a8
0xffffffff810c14c8
0xffffffff8101a79c
[...]--- tools/perf/builtin-report.c | 82 +++++++++++++++++++++++++++++++++++++------ 1 files changed, 71 insertions(+), 11 deletions(-)
tools/perf/builtin-report.c +71 -11
@@ -15,6 +15,7 @@#include "util/rbtree.h" #include "util/symbol.h" #include "util/string.h" +#include "util/callchain.h" #include "perf.h" #include "util/header.h"
@@ -52,6 +53,7 @@ static char *parent_pattern = default_parent_pattern;static regex_t parent_regex;static int exclude_other =1 ; +static int callchain;static u64 sample_type;
@@ -488,17 +490,19 @@ static size_t threads__fprintf(FILE *fp)static struct rb_root hist;struct hist_entry { -struct rb_node rb_node; - -struct thread *thread; -struct map *map; -struct dso *dso; -struct symbol *sym; -struct symbol *parent; -u64 ip; -char level; - -u64 count; +struct rb_node rb_node; + +struct thread *thread; +struct map *map; +struct dso *dso; +struct symbol *sym; +struct symbol *parent; +u64 ip; +char level; +struct callchain_node callchain; +struct rb_root sorted_chain; + +u64 count; };/*
@@ -769,6 +773,48 @@ hist_entry__collapse(struct hist_entry *left, struct hist_entry *right) }static size_t +callchain__fprintf (FILE *fp,struct callchain_node *self,u64 total_samples) +{ +struct callchain_list *chain; +size_t ret =0 ; + +if (!self) +return 0 ; + + ret +=callchain__fprintf (fp, self->parent, total_samples); + + +list_for_each_entry (chain, &self->val, list) + ret +=fprintf (fp," %p\n" , (void *)chain->ip); + +return ret; +} + +static size_t +hist_entry_callchain__fprintf (FILE *fp,struct hist_entry *self, +u64 total_samples) +{ +struct rb_node *rb_node; +struct callchain_node *chain; +size_t ret =0 ; + + rb_node =rb_first (&self->sorted_chain); +while (rb_node) { +double percent; + + chain = rb_entry(rb_node, struct callchain_node, rb_node); + percent = chain->hit *100.0 / total_samples; + ret +=fprintf (fp," %6.2f%%\n" , percent); + ret +=callchain__fprintf (fp, chain, total_samples); + ret +=fprintf (fp,"\n" ); + rb_node =rb_next (rb_node); + } + +return ret; +} + + +static size_t hist_entry__fprintf (FILE *fp,struct hist_entry *self,u64 total_samples) {struct sort_entry *se;
@@ -808,6 +854,9 @@ hist_entry__fprintf(FILE *fp, struct hist_entry *self, u64 total_samples) ret +=fprintf (fp,"\n" ); +if (callchain) +hist_entry_callchain__fprintf (fp, self, total_samples); +return ret; }
@@ -892,6 +941,7 @@ hist_entry__add(struct thread *thread, struct map *map, struct dso *dso, .level = level, .count = count, .parent = NULL, + .sorted_chain = RB_ROOT };int cmp;
@@ -934,6 +984,8 @@ hist_entry__add(struct thread *thread, struct map *map, struct dso *dso,if (!cmp) { he->count += count; +if (callchain) +append_chain (&he->callchain, chain);return 0 ; }
@@ -947,6 +999,10 @@ hist_entry__add(struct thread *thread, struct map *map, struct dso *dso,if (!he)return -ENOMEM ; *he = entry; +if (callchain) { +callchain_init (&he->callchain); +append_chain (&he->callchain, chain); + }rb_link_node (&he->rb_node, parent, p);rb_insert_color (&he->rb_node, &hist);
@@ -1023,6 +1079,9 @@ static void output__insert_entry(struct hist_entry *he)struct rb_node *parent =NULL ;struct hist_entry *iter; +if (callchain) +sort_chain_to_rbtree (&he->sorted_chain, &he->callchain); +while (*p !=NULL ) { parent = *p; iter = rb_entry(parent, struct hist_entry, rb_node);
@@ -1599,6 +1658,7 @@ static const struct option options[] = { "regex filter to identify parent, see: '--sort parent'"),OPT_BOOLEAN ('x' ,"exclude-other" , &exclude_other,"Only display entries with parent-match" ), +OPT_BOOLEAN ('c' ,"callchain" , &callchain,"Display callchains" ), OPT_END() };
Signature · 1 line
1.6.2.3
[tip:perfcounters/urgent] perf report: Print sorted callchains per histogram entries
Commit-ID: f55c555226b1010b249730ec6b232e5470286950 Gitweb: http://git.kernel.org/tip/f55c555226b1010b249730ec6b232e5470286950 Author: Frederic Weisbecker <[email protected]> AuthorDate: Fri, 26 Jun 2009 16:28:01 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 26 Jun 2009 16:47:01 +0200
perf report: Print sorted callchains per histogram entries
Use the newly created callchains radix tree to gather the chains stats from the recorded events and then print the callchains for all of them, sorted by hits, using the "-c" parameter with perf report.
Example:
66.15% [k] atm_clip_exit
63.08%
0xffffffffffffff80
0xffffffff810196a8
0xffffffff810c14c8
0xffffffff8101a79c
0xffffffff810194f3
0xffffffff8106ab7f
0xffffffff8106abe5
0xffffffff8106acde
0xffffffff8100d94b
0xffffffff8153e7ea
[...]
1.54%
0xffffffffffffff80
0xffffffff810196a8
0xffffffff810c14c8
0xffffffff8101a79c
[...]Symbols are not yet resolved.
LKML-Reference: <[email protected]>
--- tools/perf/builtin-report.c | 82 +++++++++++++++++++++++++++++++++++++------ 1 files changed, 71 insertions(+), 11 deletions(-)
tools/perf/builtin-report.c +71 -11
@@ -15,6 +15,7 @@#include "util/rbtree.h" #include "util/symbol.h" #include "util/string.h" +#include "util/callchain.h" #include "perf.h" #include "util/header.h"
@@ -52,6 +53,7 @@ static char *parent_pattern = default_parent_pattern;static regex_t parent_regex;static int exclude_other =1 ; +static int callchain;static u64 sample_type;
@@ -488,17 +490,19 @@ static size_t threads__fprintf(FILE *fp)static struct rb_root hist;struct hist_entry { -struct rb_node rb_node; - -struct thread *thread; -struct map *map; -struct dso *dso; -struct symbol *sym; -struct symbol *parent; -u64 ip; -char level; - -u64 count; +struct rb_node rb_node; + +struct thread *thread; +struct map *map; +struct dso *dso; +struct symbol *sym; +struct symbol *parent; +u64 ip; +char level; +struct callchain_node callchain; +struct rb_root sorted_chain; + +u64 count; };/*
@@ -769,6 +773,48 @@ hist_entry__collapse(struct hist_entry *left, struct hist_entry *right) }static size_t +callchain__fprintf (FILE *fp,struct callchain_node *self,u64 total_samples) +{ +struct callchain_list *chain; +size_t ret =0 ; + +if (!self) +return 0 ; + + ret +=callchain__fprintf (fp, self->parent, total_samples); + + +list_for_each_entry (chain, &self->val, list) + ret +=fprintf (fp," %p\n" , (void *)chain->ip); + +return ret; +} + +static size_t +hist_entry_callchain__fprintf (FILE *fp,struct hist_entry *self, +u64 total_samples) +{ +struct rb_node *rb_node; +struct callchain_node *chain; +size_t ret =0 ; + + rb_node =rb_first (&self->sorted_chain); +while (rb_node) { +double percent; + + chain = rb_entry(rb_node, struct callchain_node, rb_node); + percent = chain->hit *100.0 / total_samples; + ret +=fprintf (fp," %6.2f%%\n" , percent); + ret +=callchain__fprintf (fp, chain, total_samples); + ret +=fprintf (fp,"\n" ); + rb_node =rb_next (rb_node); + } + +return ret; +} + + +static size_t hist_entry__fprintf (FILE *fp,struct hist_entry *self,u64 total_samples) {struct sort_entry *se;
@@ -808,6 +854,9 @@ hist_entry__fprintf(FILE *fp, struct hist_entry *self, u64 total_samples) ret +=fprintf (fp,"\n" ); +if (callchain) +hist_entry_callchain__fprintf (fp, self, total_samples); +return ret; }
@@ -892,6 +941,7 @@ hist_entry__add(struct thread *thread, struct map *map, struct dso *dso, .level = level, .count = count, .parent = NULL, + .sorted_chain = RB_ROOT };int cmp;
@@ -934,6 +984,8 @@ hist_entry__add(struct thread *thread, struct map *map, struct dso *dso,if (!cmp) { he->count += count; +if (callchain) +append_chain (&he->callchain, chain);return 0 ; }
@@ -947,6 +999,10 @@ hist_entry__add(struct thread *thread, struct map *map, struct dso *dso,if (!he)return -ENOMEM ; *he = entry; +if (callchain) { +callchain_init (&he->callchain); +append_chain (&he->callchain, chain); + }rb_link_node (&he->rb_node, parent, p);rb_insert_color (&he->rb_node, &hist);
@@ -1023,6 +1079,9 @@ static void output__insert_entry(struct hist_entry *he)struct rb_node *parent =NULL ;struct hist_entry *iter; +if (callchain) +sort_chain_to_rbtree (&he->sorted_chain, &he->callchain); +while (*p !=NULL ) { parent = *p; iter = rb_entry(parent, struct hist_entry, rb_node);
@@ -1599,6 +1658,7 @@ static const struct option options[] = { "regex filter to identify parent, see: '--sort parent'"),OPT_BOOLEAN ('x' ,"exclude-other" , &exclude_other,"Only display entries with parent-match" ), +OPT_BOOLEAN ('c' ,"callchain" , &callchain,"Display callchains" ), OPT_END() };
Quoted 4 lines · Frederic Weisbecker
Hi,
Here is a first shot for the sorted callchains per entries handling with per report.
ok, this is already useful at displaying the raw data.
Quoted 9 lines
I'll continue to improve it:
- symbol resolution
- profit we have a tree to display a better graph hierarchy
- let the user provide a limit for hit percentage, depth, number of backtraces, etc...
- better output
- colors
- and so on....
nice plans! :)
Ingo
Quoted 5 lines · Frederic Weisbecker
Here is a first shot for the sorted callchains per entries handling with per report.
I'll continue to improve it:
Quoted 7 lines
- symbol resolution
- profit we have a tree to display a better graph hierarchy
- let the user provide a limit for hit percentage, depth, number of backtraces, etc...
- better output
- colors
- and so on....
Nice!
I have just about finished doing the kernel piece of callchain support on powerpc. Because of the way function calls and returns work on powerpc, working out the first one or two return addresses can be tricky. We potentially have a valid return address in the link register (LR), or in the LR save area in the second stack frame, or both, and you need extra information such as DWARF unwind tables to work out which of those three possibilities you have, in general. This is the case at each point where an interrupt or signal has occurred.
Because I didn't want to go trawling through CFI tables at interrupt time, particularly for user code, I made the kernel save both possible return addresses in the callchain. For the kernel part of the callchain, I check those two addresses to see if they're valid kernel addresses and set them to 0 if not, or if they're equal.
That means I need to make some changes to builtin-report.c to ignore zero addresses. I may need to add stuff to look for and use unwind tables as well, if we want completely accurate call chains.
The other thing I did is to put PERF_CONTEXT_KERNEL markers in the callchain every time we find an interrupt frame, and PERF_CONTEXT_USER markers every time we find a signal frame, so that userspace knows when it needs to do the unwinding.
Oh, and a third point is that on powerpc the sampled IP recorded if you ask for PERF_SAMPLE_IP won't in general be the same as the first IP in the callchain. The reason is that the PERF_SAMPLE_IP value points to the instruction that caused the counter overflow whereas the first IP in the callchain tells you where the CPU took the interrupt. That is almost always a few instructions further on, and can be quite a way further on if interrupts were disabled when the counter overflow occur. In fact we regularly see the PERF_SAMPLE_IP value being in the hypervisor but the first IP in the callchain being in the kernel.
Paul.
Quoted 33 lines · Paul Mackerras
Quoted 5 lines · Frederic Weisbecker
Here is a first shot for the sorted callchains per entries handling with per report.
I'll continue to improve it:
Quoted 7 lines
- symbol resolution
- profit we have a tree to display a better graph hierarchy
- let the user provide a limit for hit percentage, depth, number of backtraces, etc...
- better output
- colors
- and so on....Nice!
I have just about finished doing the kernel piece of callchain support on powerpc. Because of the way function calls and returns work on powerpc, working out the first one or two return addresses can be tricky. We potentially have a valid return address in the link register (LR), or in the LR save area in the second stack frame, or both, and you need extra information such as DWARF unwind tables to work out which of those three possibilities you have, in general. This is the case at each point where an interrupt or signal has occurred.
Because I didn't want to go trawling through CFI tables at interrupt time, particularly for user code, I made the kernel save both possible return addresses in the callchain. For the kernel part of the callchain, I check those two addresses to see if they're valid kernel addresses and set them to 0 if not, or if they're equal.
Quoted 3 lines
That means I need to make some changes to
builtin-report.cto ignore zero addresses. I may need to add stuff to look for and use unwind tables as well, if we want completely accurate call chains.
Well, I guess I can ignore them in my further patches. But wouldn't it be better to discard them from the kernel? Unless it's somewhat useful to know we had an unknown entry?
Quoted 5 lines
The other thing I did is to put
PERF_CONTEXT_KERNELmarkers in the callchain every time we find an interrupt frame, andPERF_CONTEXT_USERmarkers every time we find a signal frame, so that userspace knows when it needs to do the unwinding.
Quoted 9 lines
Oh, and a third point is that on powerpc the sampled IP recorded if you ask for
PERF_SAMPLE_IPwon't in general be the same as the first IP in the callchain. The reason is that thePERF_SAMPLE_IPvalue points to the instruction that caused the counter overflow whereas the first IP in the callchain tells you where the CPU took the interrupt. That is almost always a few instructions further on, and can be quite a way further on if interrupts were disabled when the counter overflow occur. In fact we regularly see thePERF_SAMPLE_IPvalue being in the hypervisor but the first IP in the callchain being in the kernel.
Ok.
Quoted 2 lines
Paul.
Quoted 10 lines · Frederic Weisbecker
Quoted 3 lines · Paul Mackerras
That means I need to make some changes to
builtin-report.cto ignore zero addresses. I may need to add stuff to look for and use unwind tables as well, if we want completely accurate call chains.Well, I guess I can ignore them in my further patches. But wouldn't it be better to discard them from the kernel? Unless it's somewhat useful to know we had an unknown entry?
If we discard the entries then userspace doesn't know which entries were discarded, or whether any were discarded. As it is, userspace can know that the second value after PERF_CONTEXT_KERNEL/USER is a LR value or zero, and the third value is from the second stack frame (or zero). If we discarded the entries then userspace wouldn't know exactly where the second and third values came from, which would make it harder to use unwind or traceback tables to work out more accurately what the call chain was.
I would be open to replacing the bogus entries with some other distinguishable value rather than zero if you think that would be better.
Paul.