perf tools: callchain: Warn only once in empty node detection
PATCH 1/4 perf tools: callchain: Warn only once in empty node detection
When we fill a new node that is found to be empty (sign of a bug), we print a message each time that happens. Sometimes we reach thousands of lines printed.
Just warn only once in that case (and use stderr instead of stdout).
--- tools/perf/util/callchain.c | 3 ++- tools/perf/util/util.h | 3 +++ tools/perf/util/wrapper.c | 16 ++++++++++++++++ 3 files changed, 21 insertions(+), 1 deletions(-)
tools/perf/util/callchain.c +2 -1
@@ -13,6 +13,7 @@#include <stdio.h> #include <stdbool.h> #include <errno.h> +#include "util.h" #include "callchain.h"
@@ -203,7 +204,7 @@ fill_node(struct callchain_node *node, struct ip_callchain *chain, } node->val_nr = chain->nr - start;if (!node->val_nr) -printf ("Warning: empty node in callchain tree\n" ); +fprintf_once (stderr,"Warning: empty node in callchain tree\n" ); } static void
tools/perf/util/util.h +3 -0
@@ -148,6 +148,9 @@ static inline const char *skip_prefix(const char *str, const char *prefix)return strncmp (str, prefix, len) ?NULL : str + len; } +extern int fprintf_once (FILE *fp,const char *fmt, ...) + __attribute__((format (printf,2 ,3 ))); +#if defined(NO_MMAP ) || defined(USE_WIN32_MMAP )#ifndef PROT_READ
tools/perf/util/wrapper.c +16 -0
@@ -205,3 +205,19 @@ int xmkstemp(char *template)die ("Unable to create temporary file: %s" ,strerror (errno));return fd; } + +int fprintf_once (FILE *fp,const char *fmt, ...) +{ +va_list args; +int ret; +static int once; + +if (once++) +return 1 ; + +va_start (args, fmt); + ret =vfprintf (fp, fmt, args); +va_end (args); + +return ret; +}
Signature · 1 line
1.6.2.3
PATCH 2/4 perf tools: callchain: Ignore empty callchains
When the callchain tree comes to insert an empty backtrace, it raises a spurious warning about the fact we are inserting an empty. This is spurious because the radix tree assumes it did something wrong to reach this state. But it didn't, we just met an empty callchain that has to be ignored.
--- tools/perf/util/callchain.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
tools/perf/util/callchain.c +2 -0
@@ -337,5 +337,7 @@ __append_chain(struct callchain_node *root, struct ip_callchain *chain,void append_chain (struct callchain_node *root,struct ip_callchain *chain,struct symbol **syms) { +if (!chain->nr) +return ;__append_chain_children (root, chain, syms,0 ); }
Signature · 1 line
1.6.2.3
[tip:perfcounters/core] perf tools: callchain: Fix spurious 'perf report' warnings: ignore empty callchains
Commit-ID: d6bda845b99fbdfa2f88ac9f82d757297783b5d5 Gitweb: http://git.kernel.org/tip/d6bda845b99fbdfa2f88ac9f82d757297783b5d5 Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:23 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 8 Aug 2009 13:49:21 +0200
perf tools: callchain: Fix spurious 'perf report' warnings: ignore empty callchains
When the callchain tree comes to insert an empty backtrace, it raises a spurious warning about the fact we are inserting an empty. This is spurious because the radix tree assumes it did something wrong to reach this state. But it didn't, we just met an empty callchain that has to be ignored.
This happens occasionally with certain types of call-chain recordings. If it happens it's a big nuisance as perf report output starts with thousands of warning lines.
LKML-Reference: <[email protected]>
--- tools/perf/util/callchain.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
tools/perf/util/callchain.c +2 -0
@@ -336,5 +336,7 @@ __append_chain(struct callchain_node *root, struct ip_callchain *chain,void append_chain (struct callchain_node *root,struct ip_callchain *chain,struct symbol **syms) { +if (!chain->nr) +return ;__append_chain_children (root, chain, syms,0 ); }
[tip:perfcounters/core] perf tools: callchain: Fix spurious 'perf report' warnings: ignore empty callchains
In reply to Frederic WeisbeckerCommit-ID: b0efe213f84f7fd5ccfe07053e3d9fb827b7c188 Gitweb: http://git.kernel.org/tip/b0efe213f84f7fd5ccfe07053e3d9fb827b7c188 Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:23 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 9 Aug 2009 12:54:41 +0200
perf tools: callchain: Fix spurious 'perf report' warnings: ignore empty callchains
When the callchain tree comes to insert an empty backtrace, it raises a spurious warning about the fact we are inserting an empty. This is spurious because the radix tree assumes it did something wrong to reach this state. But it didn't, we just met an empty callchain that has to be ignored.
This happens occasionally with certain types of call-chain recordings. If it happens it's a big nuisance as perf report output starts with thousands of warning lines.
LKML-Reference: <[email protected]>
--- tools/perf/util/callchain.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
tools/perf/util/callchain.c +2 -0
@@ -336,5 +336,7 @@ __append_chain(struct callchain_node *root, struct ip_callchain *chain,void append_chain (struct callchain_node *root,struct ip_callchain *chain,struct symbol **syms) { +if (!chain->nr) +return ;__append_chain_children (root, chain, syms,0 ); }
PATCH 3/4 perf tools: callchain: Default display callchain from report if recorded with -g
In reply to Frederic WeisbeckerIf we recorded with -g option to record the callchain, let's display the callchain by default from perf report.
The user can override this default using "-g none" option from perf report.
--- tools/perf/builtin-report.c | 16 +++++++++++++++- tools/perf/util/callchain.h | 1 + 2 files changed, 16 insertions(+), 1 deletions(-)
tools/perf/builtin-report.c +15 -1
@@ -68,7 +68,7 @@ static int callchain;static struct callchain_param callchain_param = { - .mode =CHAIN_GRAPH_ABS , + .mode =CHAIN_GRAPH_REL , .min_percent =0.5 };
@@ -1836,6 +1836,13 @@ static int __cmd_report(void) " -g?\n");exit (-1 ); } + }else if (callchain_param.mode !=CHAIN_NONE && !callchain) { + callchain =1 ; +if (register_callchain_param (&callchain_param) <0 ) { +fprintf (stderr,"Can't register callchain" +" params\n" ); +exit (-1 ); + } }if (load_kernel () <0 ) {
@@ -1974,6 +1981,13 @@ parse_callchain_opt(const struct option *opt __used, const char *arg,else if (!strncmp (tok,"fractal" ,strlen (arg))) callchain_param.mode =CHAIN_GRAPH_REL ; +else if (!strncmp (tok,"none" ,strlen (arg))) { + callchain_param.mode =CHAIN_NONE ; + callchain =0 ; + +return 0 ; + } +else return -1 ;
tools/perf/util/callchain.h +1 -0
@@ -7,6 +7,7 @@#include "symbol.h" enum chain_mode { +CHAIN_NONE ,CHAIN_FLAT ,CHAIN_GRAPH_ABS ,CHAIN_GRAPH_REL
Signature · 1 line
1.6.2.3
[tip:perfcounters/core] perf tools: callchain: Fix 'perf report' display to be callchain by default
Commit-ID: 0b51a97cec68a953629cc388d762c61b2c05886e Gitweb: http://git.kernel.org/tip/0b51a97cec68a953629cc388d762c61b2c05886e Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 8 Aug 2009 13:49:21 +0200
perf tools: callchain: Fix 'perf report' display to be callchain by default
If we recorded with -g option to record the callchain, right now we require a -g option to perf report as well - and people reported this as unnecessary complication: the user already specified -g once, no need to require it a second time.
So if the recording includes call-chains, display the callchain by default from perf report.
( The user can override this default using "-g none" option from
perf report. )
LKML-Reference: <[email protected]>
--- tools/perf/builtin-report.c | 16 +++++++++++++++- tools/perf/util/callchain.h | 1 + 2 files changed, 16 insertions(+), 1 deletions(-)
tools/perf/builtin-report.c +15 -1
@@ -68,7 +68,7 @@ static int callchain;static struct callchain_param callchain_param = { - .mode =CHAIN_GRAPH_ABS , + .mode =CHAIN_GRAPH_REL , .min_percent =0.5 };
@@ -1836,6 +1836,13 @@ static int __cmd_report(void) " -g?\n");exit (-1 ); } + }else if (callchain_param.mode !=CHAIN_NONE && !callchain) { + callchain =1 ; +if (register_callchain_param (&callchain_param) <0 ) { +fprintf (stderr,"Can't register callchain" +" params\n" ); +exit (-1 ); + } }if (load_kernel () <0 ) {
@@ -1974,6 +1981,13 @@ parse_callchain_opt(const struct option *opt __used, const char *arg,else if (!strncmp (tok,"fractal" ,strlen (arg))) callchain_param.mode =CHAIN_GRAPH_REL ; +else if (!strncmp (tok,"none" ,strlen (arg))) { + callchain_param.mode =CHAIN_NONE ; + callchain =0 ; + +return 0 ; + } +else return -1 ;
tools/perf/util/callchain.h +1 -0
@@ -7,6 +7,7 @@#include "symbol.h" enum chain_mode { +CHAIN_NONE ,CHAIN_FLAT ,CHAIN_GRAPH_ABS ,CHAIN_GRAPH_REL
[tip:perfcounters/core] perf tools: callchain: Fix 'perf report' display to be callchain by default
In reply to Frederic WeisbeckerCommit-ID: b1a88349c37624755b28ac3b3152b48f52c1f487 Gitweb: http://git.kernel.org/tip/b1a88349c37624755b28ac3b3152b48f52c1f487 Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 9 Aug 2009 12:54:42 +0200
perf tools: callchain: Fix 'perf report' display to be callchain by default
If we recorded with -g option to record the callchain, right now we require a -g option to perf report as well - and people reported this as unnecessary complication: the user already specified -g once, no need to require it a second time.
So if the recording includes call-chains, display the callchain by default from perf report.
( The user can override this default using "-g none" option from
perf report. )
LKML-Reference: <[email protected]>
--- tools/perf/builtin-report.c | 16 +++++++++++++++- tools/perf/util/callchain.h | 1 + 2 files changed, 16 insertions(+), 1 deletions(-)
tools/perf/builtin-report.c +15 -1
@@ -68,7 +68,7 @@ static int callchain;static struct callchain_param callchain_param = { - .mode =CHAIN_GRAPH_ABS , + .mode =CHAIN_GRAPH_REL , .min_percent =0.5 };
@@ -1836,6 +1836,13 @@ static int __cmd_report(void) " -g?\n");exit (-1 ); } + }else if (callchain_param.mode !=CHAIN_NONE && !callchain) { + callchain =1 ; +if (register_callchain_param (&callchain_param) <0 ) { +fprintf (stderr,"Can't register callchain" +" params\n" ); +exit (-1 ); + } }if (load_kernel () <0 ) {
@@ -1974,6 +1981,13 @@ parse_callchain_opt(const struct option *opt __used, const char *arg,else if (!strncmp (tok,"fractal" ,strlen (arg))) callchain_param.mode =CHAIN_GRAPH_REL ; +else if (!strncmp (tok,"none" ,strlen (arg))) { + callchain_param.mode =CHAIN_NONE ; + callchain =0 ; + +return 0 ; + } +else return -1 ;
tools/perf/util/callchain.h +1 -0
@@ -7,6 +7,7 @@#include "symbol.h" enum chain_mode { +CHAIN_NONE ,CHAIN_FLAT ,CHAIN_GRAPH_ABS ,CHAIN_GRAPH_REL
PATCH 4/4 perf tools: callchain: Display amount of ignored chains in fractal mode
In reply to Frederic WeisbeckerWhen we filter the callchains below a given percentage, we ignore them and the end result only shows entries that have an upper percentage than the filter threshold. It seems then that we have an imbalance in the percentage, as if the sum inside a profiled branch doesn't reach 100%.
Fix this by displaying the remaining hits that have been filtered but without more detail than their amount in each branches. Example while filtering below 50%:
7.73% [k] delay_tsc
|
|--98.22%-- __const_udelay
| |
| |--86.37%-- ath5k_hw_register_timeout
| | ath5k_hw_noise_floor_calibration
| | ath5k_hw_reset
| | ath5k_reset
| | ath5k_config
| | ieee80211_hw_config
| | |
| | |--88.53%-- ieee80211_scan_work
| | | worker_thread
| | | kthread
| | | child_rip
| | --11.47%-- [...]
| --13.63%-- [...]
--1.78%-- [...]--- tools/perf/builtin-report.c | 47 ++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 44 insertions(+), 3 deletions(-)
tools/perf/builtin-report.c +44 -3
@@ -891,6 +891,21 @@ ipchain__fprintf_graph(FILE *fp, struct callchain_list *chain, int depth,return ret; } +static struct symbol *rem_sq_bracket; +static struct callchain_list rem_hits; + +static void init_rem_hits (void ) +{ + rem_sq_bracket =malloc (sizeof (*rem_sq_bracket) +6 ); +if (!rem_sq_bracket) { +fprintf (stderr,"Not enough memory to display remaining hits\n" ); +return ; + } + +strcpy (rem_sq_bracket->name,"[...]" ); + rem_hits.sym = rem_sq_bracket; +} +static size_t callchain__fprintf_graph (FILE *fp,struct callchain_node *self,u64 total_samples,int depth,int depth_mask)
@@ -900,6 +915,7 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self,struct callchain_list *chain;int new_depth_mask = depth_mask;u64 new_total; +u64 remaining;size_t ret =0 ;int i;
@@ -908,17 +924,25 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self,else new_total = total_samples; + remaining = new_total; + node =rb_first (&self->rb_root);while (node) { +u64 cumul; + child = rb_entry(node, struct callchain_node, rb_node); + cumul =cumul_hits (child); + remaining -= cumul;/* * The depth mask manages the output of pipes that show * the depth. We don't want to keep the pipes of the current -* level for the last child of this depth +* level for the last child of this depth. +* Except if we have remaining filtered hits. They will +* supersede the last child */ next =rb_next (node); -if (!next) +if (!next && (callchain_param.mode !=CHAIN_GRAPH_REL || !remaining)) new_depth_mask &= ~(1 << (depth -1 ));/*
@@ -933,7 +957,7 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self, ret +=ipchain__fprintf_graph (fp, chain, depth, new_depth_mask, i++, new_total, -cumul_hits (child)); + cumul); } ret += callchain__fprintf_graph(fp, child, new_total, depth + 1,
@@ -941,6 +965,19 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self, node = next; } +if (callchain_param.mode ==CHAIN_GRAPH_REL && + remaining && remaining != new_total) { + +if (!rem_sq_bracket) +return ret; + + new_depth_mask &= ~(1 << (depth -1 )); + + ret +=ipchain__fprintf_graph (fp, &rem_hits, depth, + new_depth_mask,0 , new_total, + remaining); + } +return ret; }
@@ -1361,6 +1398,8 @@ static size_t output__fprintf(FILE *fp, u64 total_samples)unsigned int width;char *col_width = col_width_list_str; +init_rem_hits (); +fprintf (fp,"# Samples: %Ld\n" , (u64 )total_samples);fprintf (fp,"#\n" );
@@ -1432,6 +1471,8 @@ print_entries: }fprintf (fp,"\n" ); +free (rem_sq_bracket); +return ret; }
Signature · 1 line
1.6.2.3
[tip:perfcounters/core] perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
Commit-ID: 84fce09c6b21ef675de079a724c13a205c89520b Gitweb: http://git.kernel.org/tip/84fce09c6b21ef675de079a724c13a205c89520b Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:25 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 8 Aug 2009 13:49:21 +0200
perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
When we filter the callchains below a given percentage, we ignore them and the end result only shows entries that have an upper percentage than the filter threshold.
It seems to users then that we have an imbalance in the percentage, as if the sum inside a profiled branch doesn't reach 100%.
Since in the past there have been real perf report bugs that showed the same sypmtom, it would be nice to assure the user that the data is perfect and trustable and it all sums up to 100.00%.
So fix this by displaying the remaining hits that have been filtered but without more detail than their amount in each branches. Example while filtering below 50%:
7.73% [k] delay_tsc
|
|--98.22%-- __const_udelay
| |
| |--86.37%-- ath5k_hw_register_timeout
| | ath5k_hw_noise_floor_calibration
| | ath5k_hw_reset
| | ath5k_reset
| | ath5k_config
| | ieee80211_hw_config
| | |
| | |--88.53%-- ieee80211_scan_work
| | | worker_thread
| | | kthread
| | | child_rip
| | --11.47%-- [...]
| --13.63%-- [...]
--1.78%-- [...]LKML-Reference: <[email protected]>
--- tools/perf/builtin-report.c | 47 ++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 44 insertions(+), 3 deletions(-)
tools/perf/builtin-report.c +44 -3
@@ -891,6 +891,21 @@ ipchain__fprintf_graph(FILE *fp, struct callchain_list *chain, int depth,return ret; } +static struct symbol *rem_sq_bracket; +static struct callchain_list rem_hits; + +static void init_rem_hits (void ) +{ + rem_sq_bracket =malloc (sizeof (*rem_sq_bracket) +6 ); +if (!rem_sq_bracket) { +fprintf (stderr,"Not enough memory to display remaining hits\n" ); +return ; + } + +strcpy (rem_sq_bracket->name,"[...]" ); + rem_hits.sym = rem_sq_bracket; +} +static size_t callchain__fprintf_graph (FILE *fp,struct callchain_node *self,u64 total_samples,int depth,int depth_mask)
@@ -900,6 +915,7 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self,struct callchain_list *chain;int new_depth_mask = depth_mask;u64 new_total; +u64 remaining;size_t ret =0 ;int i;
@@ -908,17 +924,25 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self,else new_total = total_samples; + remaining = new_total; + node =rb_first (&self->rb_root);while (node) { +u64 cumul; + child = rb_entry(node, struct callchain_node, rb_node); + cumul =cumul_hits (child); + remaining -= cumul;/* * The depth mask manages the output of pipes that show * the depth. We don't want to keep the pipes of the current -* level for the last child of this depth +* level for the last child of this depth. +* Except if we have remaining filtered hits. They will +* supersede the last child */ next =rb_next (node); -if (!next) +if (!next && (callchain_param.mode !=CHAIN_GRAPH_REL || !remaining)) new_depth_mask &= ~(1 << (depth -1 ));/*
@@ -933,7 +957,7 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self, ret +=ipchain__fprintf_graph (fp, chain, depth, new_depth_mask, i++, new_total, -cumul_hits (child)); + cumul); } ret += callchain__fprintf_graph(fp, child, new_total, depth + 1,
@@ -941,6 +965,19 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self, node = next; } +if (callchain_param.mode ==CHAIN_GRAPH_REL && + remaining && remaining != new_total) { + +if (!rem_sq_bracket) +return ret; + + new_depth_mask &= ~(1 << (depth -1 )); + + ret +=ipchain__fprintf_graph (fp, &rem_hits, depth, + new_depth_mask,0 , new_total, + remaining); + } +return ret; }
@@ -1361,6 +1398,8 @@ static size_t output__fprintf(FILE *fp, u64 total_samples)unsigned int width;char *col_width = col_width_list_str; +init_rem_hits (); +fprintf (fp,"# Samples: %Ld\n" , (u64 )total_samples);fprintf (fp,"#\n" );
@@ -1432,6 +1471,8 @@ print_entries: }fprintf (fp,"\n" ); +free (rem_sq_bracket); +return ret; }
Re: [tip:perfcounters/core] perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
Quoted 36 lines · tip-bot for Frederic Weisbecker
Commit-ID: 84fce09c6b21ef675de079a724c13a205c89520b Gitweb: http://git.kernel.org/tip/84fce09c6b21ef675de079a724c13a205c89520b Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:25 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 8 Aug 2009 13:49:21 +0200perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
When we filter the callchains below a given percentage, we ignore them and the end result only shows entries that have an upper percentage than the filter threshold.
It seems to users then that we have an imbalance in the percentage, as if the sum inside a profiled branch doesn't reach 100%.
Since in the past there have been real perf report bugs that showed the same sypmtom, it would be nice to assure the user that the data is perfect and trustable and it all sums up to 100.00%.
So fix this by displaying the remaining hits that have been filtered but without more detail than their amount in each branches. Example while filtering below 50%:
7.73% [k]
delay_tsc| |--98.22%-- __const_udelay | | | |--86.37%-- ath5k_hw_register_timeout | | ath5k_hw_noise_floor_calibration | | ath5k_hw_reset | | ath5k_reset | | ath5k_config | | ieee80211_hw_config
Quoted 8 lines
| | |
| | |--88.53%-- ieee80211_scan_work
| | | worker_thread
| | | kthread
| | | child_rip
| | --11.47%-- [...]
| --13.63%-- [...]
--1.78%-- [...]i dont seem to be able to get the bracketed numbers. I only get:
titan:~> perf record -f -g -a ./hackbench 10
titan:~> perf report -g fractal,10.0
# Samples: 26832
#
# Overhead Command Shared Object Symbol
# ........ .............. ........................ ......
#
8.69% hackbench [kernel] [k] copy_user_generic_string 6.84% hackbench [kernel] [k] unix_stream_recvmsg
6.07% hackbench [kernel] [k] sock_alloc_send_pskb
4.33% hackbench [kernel] [k] _spin_lock
3.77% hackbench [kernel] [k] __kmalloc_node_track_caller
3.74% hackbench [kernel] [k] __alloc_skb
|
|--80.00%-- sock_alloc_send_pskb
| sock_alloc_send_skb
| unix_stream_sendmsg
| __sock_sendmsg
| sock_aio_write
| do_sync_write
| vfs_write
| sys_write
| sysenter_dispatch
| 0xf7ffa430
| 0xffadeb0000000014
|
--20.00%-- sock_alloc_send_skb
unix_stream_sendmsg
__sock_sendmsg
sock_aio_write
do_sync_write
vfs_write
sys_write
sysenter_dispatch
0xf7ffa430
0xffadeb0000000014
3.53% hackbench [kernel] [k] ia32_sysenter_target
2.92% hackbench [vdso] [.] 0x000000f7ffa430
|
|--66.67%-- 0xf7ffa430
| 0xffadeb0000000014
|
--33.33%-- 0xf7ffa431
0xffadeb0000000014
2.56% hackbench [kernel] [k] virt_to_head_page
2.37% hackbench [kernel] [k] kmem_cache_alloc_node
2.33% hackbench [kernel] [k] _spin_lock_irqsave
|
|--78.57%-- remove_wait_queue
| poll_freewait
| do_sys_poll
| sys_poll
| sysenter_dispatch
| 0xf7ffa430
| 0x1ffadea3c
|
|--7.14%-- __up_read
| up_read
| do_page_fault
| page_fault
| 0xf7ffa430
| 0xa0df710000000a
|
|--7.14%-- unix_stream_sendmsg
| __sock_sendmsg
| sock_aio_write
| do_sync_write
| vfs_write
| sys_write
| sysenter_dispatch
| 0xf7ffa430
| 0xffadeb0000000014
|
--7.14%-- skb_queue_tail
unix_stream_sendmsg
__sock_sendmsg
sock_aio_write
do_sync_write
vfs_write
sys_write
sysenter_dispatch
0xf7ffa430
0xffadeb0000000014I dont see the brackets anymore, nor does the 10% limit seem to have been listened to for the 7.14% result above, right?
Ingo
Re: [tip:perfcounters/core] perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
Quoted 146 lines · Ingo Molnar
Quoted 36 lines · tip-bot for Frederic Weisbecker
Commit-ID: 84fce09c6b21ef675de079a724c13a205c89520b Gitweb: http://git.kernel.org/tip/84fce09c6b21ef675de079a724c13a205c89520b Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:25 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 8 Aug 2009 13:49:21 +0200perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
When we filter the callchains below a given percentage, we ignore them and the end result only shows entries that have an upper percentage than the filter threshold.
It seems to users then that we have an imbalance in the percentage, as if the sum inside a profiled branch doesn't reach 100%.
Since in the past there have been real perf report bugs that showed the same sypmtom, it would be nice to assure the user that the data is perfect and trustable and it all sums up to 100.00%.
So fix this by displaying the remaining hits that have been filtered but without more detail than their amount in each branches. Example while filtering below 50%:
7.73% [k]
delay_tsc| |--98.22%-- __const_udelay | | | |--86.37%-- ath5k_hw_register_timeout | | ath5k_hw_noise_floor_calibration | | ath5k_hw_reset | | ath5k_reset | | ath5k_config | | ieee80211_hw_configQuoted 8 lines
| | | | | |--88.53%-- ieee80211_scan_work | | | worker_thread | | | kthread | | | child_rip | | --11.47%-- [...] | --13.63%-- [...] --1.78%-- [...]i dont seem to be able to get the bracketed numbers. I only get:
titan:~> perf record -f -g -a ./hackbench 10
titan:~> perf report -g fractal,10.0# Samples: 26832
#
# Overhead Command Shared Object Symbol# ........ .............. ........................ ...... # 8.69% hackbench [kernel] [k] copy_user_generic_string6.84% hackbench [kernel] [k] unix_stream_recvmsg 6.07% hackbench [kernel] [k] sock_alloc_send_pskb 4.33% hackbench [kernel] [k] _spin_lock 3.77% hackbench [kernel] [k] __kmalloc_node_track_caller 3.74% hackbench [kernel] [k] __alloc_skb | |--80.00%-- sock_alloc_send_pskb | sock_alloc_send_skb | unix_stream_sendmsg | __sock_sendmsg | sock_aio_write | do_sync_write | vfs_write | sys_write | sysenter_dispatch | 0xf7ffa430 | 0xffadeb0000000014 | --20.00%-- sock_alloc_send_skb unix_stream_sendmsg __sock_sendmsg sock_aio_write do_sync_write vfs_write sys_write sysenter_dispatch 0xf7ffa430 0xffadeb0000000014 3.53% hackbench [kernel] [k] ia32_sysenter_target 2.92% hackbench [vdso] [.] 0x000000f7ffa430 | |--66.67%-- 0xf7ffa430 | 0xffadeb0000000014 | --33.33%-- 0xf7ffa431 0xffadeb0000000014 2.56% hackbench [kernel] [k] virt_to_head_page 2.37% hackbench [kernel] [k] kmem_cache_alloc_node 2.33% hackbench [kernel] [k] _spin_lock_irqsave | |--78.57%-- remove_wait_queue | poll_freewait | do_sys_poll | sys_poll | sysenter_dispatch | 0xf7ffa430 | 0x1ffadea3c | |--7.14%-- __up_read | up_read | do_page_fault | page_fault | 0xf7ffa430 | 0xa0df710000000a | |--7.14%-- unix_stream_sendmsg | __sock_sendmsg | sock_aio_write | do_sync_write | vfs_write | sys_write | sysenter_dispatch | 0xf7ffa430 | 0xffadeb0000000014 | --7.14%-- skb_queue_tail unix_stream_sendmsg __sock_sendmsg sock_aio_write do_sync_write vfs_write sys_write sysenter_dispatch 0xf7ffa430 0xffadeb0000000014
Quoted 4 lines
I dont see the brackets anymore, nor does the 10% limit seem to have been listened to for the 7.14% result above, right?
Ingo
Oh damn. I've been investigating for several hours in the wrong direction just because this bug triggered only when I wasn't using -s s
Actually, it's just that the other kind of sorting make this bug occur much more quickly: in the first results. It was then more obvious in these cases, but the issue was present in every kind of sorting.
Please find the fix below, thanks!
Frederic.
---
Quoted 1 line
From
52c89279744db58a68fa2339b3c8038e67113fcaMon Sep 17 00:00:00 2001
From: Frederic Weisbecker <[email protected]>
Date: Sun, 9 Aug 2009 04:19:15 +0200
Subject: [PATCH] perf tools: callchain: Fix bad rounding of minimum rate
Sometimes we get callchain branches that have a rate under the limit given by the user.
Say you launched:
perf record -f -g -a ./hackbench 10
perf report -g fractal,10.0
And you get:
2.33% hackbench [kernel] [k] _spin_lock_irqsave
|
|--78.57%-- remove_wait_queue
| poll_freewait
| do_sys_poll
| sys_poll
| sysenter_dispatch
| 0xf7ffa430
| 0x1ffadea3c
|
|--7.14%-- __up_read
| up_read
| do_page_fault
| page_fault
| 0xf7ffa430
| 0xa0df710000000a
...It is abnormal to get a 7.14% branch whereas we passed a 10% filter.
The problem is that we round down the minimum threshold. This happens mostly when we have very low number of events. If the total amount of your branch is 4 and you have a subranch of 3 events, filtering to 90% will be computed like follows:
limit = 4 * 0.9;
The result is about 3.6, but the cast to integer will round down to 3. It means that our filter is actually of 75%
We must then explicitly round up the minimum threshold.
--- tools/perf/util/callchain.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
tools/perf/util/callchain.c +3 -2
@@ -13,6 +13,7 @@#include <stdio.h> #include <stdbool.h> #include <errno.h> +#include <math.h> #include "callchain.h"
@@ -112,7 +113,7 @@ static void __sort_chain_graph_rel(struct callchain_node *node,u64 min_hit; node->rb_root =RB_ROOT ; - min_hit = node->children_hit * min_percent /100.0 ; + min_hit =ceil (node->children_hit * min_percent);chain_for_each_child (child, node) {__sort_chain_graph_rel (child, min_percent);
@@ -126,7 +127,7 @@ static void sort_chain_graph_rel(struct rb_root *rb_root, struct callchain_node *chain_root, u64 min_hit __used, struct callchain_param *param) { - __sort_chain_graph_rel(chain_root, param->min_percent); + __sort_chain_graph_rel(chain_root, param->min_percent / 100.0); rb_root->rb_node = chain_root->rb_root.rb_node; }
Signature · 1 line
1.6.2.3
[tip:perfcounters/urgent] perf tools: callchain: Fix bad rounding of minimum rate
Commit-ID: c0a8865e32c8d1a562db38e06ef31ef23282f646 Gitweb: http://git.kernel.org/tip/c0a8865e32c8d1a562db38e06ef31ef23282f646 Author: Frederic Weisbecker <[email protected]> AuthorDate: Sun, 9 Aug 2009 04:19:15 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 9 Aug 2009 13:07:46 +0200
perf tools: callchain: Fix bad rounding of minimum rate
Sometimes we get callchain branches that have a rate under the limit given by the user.
Say you launched:
perf record -f -g -a ./hackbench 10
perf report -g fractal,10.0
And you got:
2.33% hackbench [kernel] [k] _spin_lock_irqsave
|
|--78.57%-- remove_wait_queue
| poll_freewait
| do_sys_poll
| sys_poll
| sysenter_dispatch
| 0xf7ffa430
| 0x1ffadea3c
|
|--7.14%-- __up_read
| up_read
| do_page_fault
| page_fault
| 0xf7ffa430
| 0xa0df710000000a
...It is abnormal to get a 7.14% branch whereas we passed a 10% filter.
The problem is that we round down the minimum threshold. This happens mostly when we have very low number of events. If the total amount of your branch is 4 and you have a subranch of 3 events, filtering to 90% will be computed like follows:
limit = 4 * 0.9;
The result is about 3.6, but the cast to integer will round down to 3. It means that our filter is actually of 75%
We must then explicitly round up the minimum threshold.
LKML-Reference: <20090809024235.GA10146@nowhere>
--- tools/perf/util/callchain.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-)
tools/perf/util/callchain.c +3 -2
@@ -13,6 +13,7 @@#include <stdio.h> #include <stdbool.h> #include <errno.h> +#include <math.h> #include "callchain.h"
@@ -112,7 +113,7 @@ static void __sort_chain_graph_rel(struct callchain_node *node,u64 min_hit; node->rb_root =RB_ROOT ; - min_hit = node->children_hit * min_percent /100.0 ; + min_hit =ceil (node->children_hit * min_percent);chain_for_each_child (child, node) {__sort_chain_graph_rel (child, min_percent);
@@ -126,7 +127,7 @@ static void sort_chain_graph_rel(struct rb_root *rb_root, struct callchain_node *chain_root, u64 min_hit __used, struct callchain_param *param) { - __sort_chain_graph_rel(chain_root, param->min_percent); + __sort_chain_graph_rel(chain_root, param->min_percent / 100.0); rb_root->rb_node = chain_root->rb_root.rb_node; }
[tip:perfcounters/core] perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
In reply to Frederic WeisbeckerCommit-ID: 25446036cbfc2c89faacdb4fb4603943d2197dc6 Gitweb: http://git.kernel.org/tip/25446036cbfc2c89faacdb4fb4603943d2197dc6 Author: Frederic Weisbecker <[email protected]> AuthorDate: Sat, 8 Aug 2009 02:16:25 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 9 Aug 2009 12:54:43 +0200
perf tools: callchain: Fix sum of percentages to be 100% by displaying amount of ignored chains in fractal mode
When we filter the callchains below a given percentage, we ignore them and the end result only shows entries that have an upper percentage than the filter threshold.
It seems to users then that we have an imbalance in the percentage, as if the sum inside a profiled branch doesn't reach 100%.
Since in the past there have been real perf report bugs that showed the same sypmtom, it would be nice to assure the user that the data is perfect and trustable and it all sums up to 100.00%.
So fix this by displaying the remaining hits that have been filtered but without more detail than their amount in each branches. Example while filtering below 50%:
7.73% [k] delay_tsc
|
|--98.22%-- __const_udelay
| |
| |--86.37%-- ath5k_hw_register_timeout
| | ath5k_hw_noise_floor_calibration
| | ath5k_hw_reset
| | ath5k_reset
| | ath5k_config
| | ieee80211_hw_config
| | |
| | |--88.53%-- ieee80211_scan_work
| | | worker_thread
| | | kthread
| | | child_rip
| | --11.47%-- [...]
| --13.63%-- [...]
--1.78%-- [...]LKML-Reference: <[email protected]>
--- tools/perf/builtin-report.c | 47 ++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 44 insertions(+), 3 deletions(-)
tools/perf/builtin-report.c +44 -3
@@ -891,6 +891,21 @@ ipchain__fprintf_graph(FILE *fp, struct callchain_list *chain, int depth,return ret; } +static struct symbol *rem_sq_bracket; +static struct callchain_list rem_hits; + +static void init_rem_hits (void ) +{ + rem_sq_bracket =malloc (sizeof (*rem_sq_bracket) +6 ); +if (!rem_sq_bracket) { +fprintf (stderr,"Not enough memory to display remaining hits\n" ); +return ; + } + +strcpy (rem_sq_bracket->name,"[...]" ); + rem_hits.sym = rem_sq_bracket; +} +static size_t callchain__fprintf_graph (FILE *fp,struct callchain_node *self,u64 total_samples,int depth,int depth_mask)
@@ -900,6 +915,7 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self,struct callchain_list *chain;int new_depth_mask = depth_mask;u64 new_total; +u64 remaining;size_t ret =0 ;int i;
@@ -908,17 +924,25 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self,else new_total = total_samples; + remaining = new_total; + node =rb_first (&self->rb_root);while (node) { +u64 cumul; + child = rb_entry(node, struct callchain_node, rb_node); + cumul =cumul_hits (child); + remaining -= cumul;/* * The depth mask manages the output of pipes that show * the depth. We don't want to keep the pipes of the current -* level for the last child of this depth +* level for the last child of this depth. +* Except if we have remaining filtered hits. They will +* supersede the last child */ next =rb_next (node); -if (!next) +if (!next && (callchain_param.mode !=CHAIN_GRAPH_REL || !remaining)) new_depth_mask &= ~(1 << (depth -1 ));/*
@@ -933,7 +957,7 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self, ret +=ipchain__fprintf_graph (fp, chain, depth, new_depth_mask, i++, new_total, -cumul_hits (child)); + cumul); } ret += callchain__fprintf_graph(fp, child, new_total, depth + 1,
@@ -941,6 +965,19 @@ callchain__fprintf_graph(FILE *fp, struct callchain_node *self, node = next; } +if (callchain_param.mode ==CHAIN_GRAPH_REL && + remaining && remaining != new_total) { + +if (!rem_sq_bracket) +return ret; + + new_depth_mask &= ~(1 << (depth -1 )); + + ret +=ipchain__fprintf_graph (fp, &rem_hits, depth, + new_depth_mask,0 , new_total, + remaining); + } +return ret; }
@@ -1361,6 +1398,8 @@ static size_t output__fprintf(FILE *fp, u64 total_samples)unsigned int width;char *col_width = col_width_list_str; +init_rem_hits (); +fprintf (fp,"# Samples: %Ld\n" , (u64 )total_samples);fprintf (fp,"#\n" );
@@ -1432,6 +1471,8 @@ print_entries: }fprintf (fp,"\n" ); +free (rem_sq_bracket); +return ret; }
Em Sat, Aug 08, 2009 at 02:16:22AM +0200, Frederic Weisbecker escreveu:
Quoted 5 lines
When we fill a new node that is found to be empty (sign of a bug), we print a message each time that happens. Sometimes we reach thousands of lines printed.
Just warn only once in that case (and use stderr instead of stdout).
Wouldn't this be better done with a macro so that the every message is print once instead of any fprintf_once message?
I think it would even be nice to just use the same thing the kernel uses... WARN_ON_ONCE, and also using asm/bug.h, etc, just add it to tools/perf/util/include/asm, etc.
That way one can even forget that this is not kernel programming... :-)
- Arnaldo
Quoted 72 lines
Reported-by Ingo Molnar <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Mike Galbraith <[email protected]>--- tools/perf/util/callchain.c | 3 ++- tools/perf/util/util.h | 3 +++ tools/perf/util/wrapper.c | 16 ++++++++++++++++ 3 files changed, 21 insertions(+), 1 deletions(-)tools/perf/util/callchain.c +2
-1@@ -13,6 +13,7 @@#include <stdio.h> #include <stdbool.h> #include <errno.h> +#include "util.h" #include "callchain.h" @@ -203,7 +204,7 @@ fill_node(struct callchain_node *node, struct ip_callchain *chain, } node->val_nr = chain->nr - start;if (!node->val_nr) -printf ("Warning: empty node in callchain tree\n" ); +fprintf_once (stderr,"Warning: empty node in callchain tree\n" ); } static voidtools/perf/util/util.h +3
-0@@ -148,6 +148,9 @@ static inline const char *skip_prefix(const char *str, const char *prefix)return strncmp (str, prefix, len) ?NULL : str + len; } +extern int fprintf_once (FILE *fp,const char *fmt, ...) + __attribute__((format (printf,2 ,3 ))); +#if defined(NO_MMAP ) || defined(USE_WIN32_MMAP )#ifndef PROT_READ tools/perf/util/wrapper.c +16
-0@@ -205,3 +205,19 @@ int xmkstemp(char *template)die ("Unable to create temporary file: %s" ,strerror (errno));return fd; } + +int fprintf_once (FILE *fp,const char *fmt, ...) +{ +va_list args; +int ret; +static int once; + +if (once++) +return 1 ; + +va_start (args, fmt); + ret =vfprintf (fp, fmt, args); +va_end (args); + +return ret; +}Signature · 1 line
1.6.2.3
Quoted 9 lines · Arnaldo Carvalho de Melo
Em Sat, Aug 08, 2009 at 02:16:22AM +0200, Frederic Weisbecker escreveu:
Quoted 5 lines
When we fill a new node that is found to be empty (sign of a bug), we print a message each time that happens. Sometimes we reach thousands of lines printed.
Just warn only once in that case (and use stderr instead of stdout).
Wouldn't this be better done with a macro so that the every message is print once instead of any
fprintf_oncemessage?
Doh! You're right...
/me slaps his face.
Quoted 4 lines
I think it would even be nice to just use the same thing the kernel uses...
WARN_ON_ONCE, and also usingasm/bug.h, etc, just add it totools/perf/util/include/asm, etc.
Ok, I'll try that.
Quoted 1 line
That way one can even forget that this is not kernel programming... :-)
Hehe :)
Quoted 74 lines
- Arnaldo
Quoted 72 lines
Reported-by Ingo Molnar <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Cc Mike Galbraith <[email protected]>--- tools/perf/util/callchain.c | 3 ++- tools/perf/util/util.h | 3 +++ tools/perf/util/wrapper.c | 16 ++++++++++++++++ 3 files changed, 21 insertions(+), 1 deletions(-)tools/perf/util/callchain.c +2
-1@@ -13,6 +13,7 @@#include <stdio.h> #include <stdbool.h> #include <errno.h> +#include "util.h" #include "callchain.h" @@ -203,7 +204,7 @@ fill_node(struct callchain_node *node, struct ip_callchain *chain, } node->val_nr = chain->nr - start;if (!node->val_nr) -printf ("Warning: empty node in callchain tree\n" ); +fprintf_once (stderr,"Warning: empty node in callchain tree\n" ); } static voidtools/perf/util/util.h +3
-0@@ -148,6 +148,9 @@ static inline const char *skip_prefix(const char *str, const char *prefix)return strncmp (str, prefix, len) ?NULL : str + len; } +extern int fprintf_once (FILE *fp,const char *fmt, ...) + __attribute__((format (printf,2 ,3 ))); +#if defined(NO_MMAP ) || defined(USE_WIN32_MMAP )#ifndef PROT_READ tools/perf/util/wrapper.c +16
-0@@ -205,3 +205,19 @@ int xmkstemp(char *template)die ("Unable to create temporary file: %s" ,strerror (errno));return fd; } + +int fprintf_once (FILE *fp,const char *fmt, ...) +{ +va_list args; +int ret; +static int once; + +if (once++) +return 1 ; + +va_start (args, fmt); + ret =vfprintf (fp, fmt, args); +va_end (args); + +return ret; +}Signature · 1 line
1.6.2.3