Skip to contentSkip to conversation

perf top: Make -g refer to callchains

PATCH In review David Ahern 7 participants 13 messages Nov 15, 2013 – Nov 30, 2013 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf top: Make -g refer to callchains

In most commands -g is used for callchains. Make perf-top follow suit. Move group to just --group with no short cut making it similar to perf-record.

Cc Ingo Molnar <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
---
 tools/perf/Documentation/perf-top.txt | 5 ++---
 tools/perf/builtin-top.c              | 4 ++--
 2 files changed, 4 insertions(+), 5 deletions(-)
tools/perf/Documentation/perf-top.txt +2 -3
index 7de01dd79688..cdd8d4946dba 100644
@@ -50,7 +50,6 @@ Default is to monitor all CPUS.
 --count-filter=<count>::
 	Only display functions with more events than this.
 
--g::
 --group::
         Put the counters into a counter group.
 
@@ -143,12 +142,12 @@ Default is to monitor all CPUS.
 --asm-raw::
 	Show raw instruction encoding of assembly instructions.
 
--G::
+-g::
 	Enables call-graph (stack chain/backtrace) recording.
 
 --call-graph::
 	Setup and enable call-graph (stack chain/backtrace) recording,
-	implies -G.
+	implies -g.
 
 --max-stack::
 	Set the stack depth limit when parsing the callchain, anything
tools/perf/builtin-top.c +2 -2
index 71e6402729a8..531522d3d97b 100644
@@ -1084,7 +1084,7 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)
 			    "dump the symbol table used for profiling"),
 	OPT_INTEGER('f', "count-filter", &top.count_filter,
 		    "only display functions with more events than this"),
-	OPT_BOOLEAN('g', "group", &opts->group,
+	OPT_BOOLEAN(0, "group", &opts->group,
 			    "put the counters into a counter group"),
 	OPT_BOOLEAN('i', "no-inherit", &opts->no_inherit,
 		    "child tasks do not inherit counters"),
@@ -1105,7 +1105,7 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)
 		   " abort, in_tx, transaction"),
 	OPT_BOOLEAN('n', "show-nr-samples", &symbol_conf.show_nr_samples,
 		    "Show a column with the number of samples"),
-	OPT_CALLBACK_NOOPT('G', NULL, &top.record_opts,
+	OPT_CALLBACK_NOOPT('g', NULL, &top.record_opts,
 			   NULL, "enables call-graph recording",
 			   &callchain_opt),
 	OPT_CALLBACK(0, "call-graph", &top.record_opts,
Signature · 1 line
1.8.3.4 (Apple Git-47)
Acked-by on v1 1/1
#1 lore raw
Quoted 4 lines · David Ahern
tools/perf/Documentation/perf-top.txt +0 -0
@@ -143,12 +142,12 @@ Default is to monitor all CPUS.
 --asm-raw::

tools/perf/Documentation/perf-top.txt Default is to monitor all CPUS. · line 150

Quoted 10 lines
tools/perf/Documentation/perf-top.txt +2 -2
 	Show raw instruction encoding of assembly instructions.
 
--G::
+-g::
 	Enables call-graph (stack chain/backtrace) recording.
 
 --call-graph::
 	Setup and enable call-graph (stack chain/backtrace) recording,
-	implies -G.
+	implies -g.

Thanks,

Ingo

#2 lore raw
In reply to David Ahern

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                     
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf    

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf      

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

#
# Overhead Command Shared Object Symbol

# ........  .......  .........................  .................................................
#
     7.08%     perf  perf                       [.] access_mem
     7.03%     perf  perf                       [.] dso__data_read_offset
     5.83%     perf  perf                       [.] maps__find
     5.64%     perf  libunwind-x86_64.so.8.0.1  [.] 0x000000000000ba25
     4.75%     perf  perf                       [.] thread__find_addr_map
     3.81%     perf  [kernel.kallsyms]          [k] unmap_single_vma
     2.57%     perf  perf                       [.] map__map_ip
     2.48%     perf  libelf-0.156.so            [.] 0x0000000000003a84
     2.12%     perf  [kernel.kallsyms]          [k] memset
     2.12%     perf  perf                       [.] dso__data_read_addr
     2.10%     perf  libc-2.17.so               [.] __memcpy_sse2
     1.72%     perf  libc-2.17.so               [.] __memset_sse2
     1.58%     perf  [kernel.kallsyms]          [k] page_fault
     1.56%     perf  libc-2.17.so               [.] __memset_x86_64
     1.44%     perf  perf                       [.] find_proc_info
     1.25%     perf  libelf-0.156.so            [.] elf_end
     1.19%     perf  [kernel.kallsyms]          [k] flush_tlb_mm_range
     1.06%     perf  libc-2.17.so               [.] vfprintf
     1.04%     perf  libunwind-x86_64.so.8.0.1  [.] _Ux86_64_dwarf_search_unwind_table
     1.00%     perf  [kernel.kallsyms]          [k] __audit_syscall_exit
     0.94%     perf  libc-2.17.so               [.] _int_free
     0.92%     perf  libc-2.17.so               [.] _int_malloc
     0.84%     perf  libc-2.17.so               [.] __memcmp_sse2
     0.81%     perf  [kernel.kallsyms]          [k] unmapped_area_topdown
     0.71%     perf  [kernel.kallsyms]          [k] system_call
     0.71%     perf  [kernel.kallsyms]          [k] system_call_after_swapgs
     0.65%     perf  [kernel.kallsyms]          [k] sysret_check
     0.63%     perf  perf                       [.] dso__find_symbol
     0.58%     perf  [kernel.kallsyms]          [k] clear_page_c
     0.58%     perf  [kernel.kallsyms]          [k] handle_mm_fault
     0.56%     perf  libc-2.17.so               [.] __sigprocmask
     0.55%     perf  [kernel.kallsyms]          [k] copy_user_generic_string
     0.51%     perf  [kernel.kallsyms]          [k] __do_fault
     0.49%     perf  [kernel.kallsyms]          [k] find_vma
     0.47%     perf  libpthread-2.17.so         [.] __libc_close
     0.44%     perf  [kernel.kallsyms]          [k] __audit_syscall_entry
     0.44%     perf  [kernel.kallsyms]          [k] mmap_region
     0.42%     perf  [kernel.kallsyms]          [k] _raw_spin_lock
     0.41%     perf  [kernel.kallsyms]          [k] kmem_cache_free
     0.40%     perf  [kernel.kallsyms]          [k] kmem_cache_alloc
     0.40%     perf  libpthread-2.17.so         [.] pthread_mutex_unlock
     0.37%     perf  [kernel.kallsyms]          [k] perf_event_aux_ctx
     0.37%     perf  [kernel.kallsyms]          [k] do_munmap
     0.37%     perf  libc-2.17.so               [.] free
     [...]

Thanks,

Ingo

#3 lore raw

Em Fri, Nov 15, 2013 at 06:46:09AM +0100, Ingo Molnar escreveu:

Quoted 14 lines

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                     
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf    
Quoted 8 lines

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf      

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

Yeah, top dwarf callchain has been so far a proof of concept, it exacerbates problems that can be seen on 'report', but since its live, we can see it more clearly.

The work on improving callchain processing, (rb_tree'ing, new comm infrastructure) alleviated the problem a bit.

Tuning the stack size requested from the kernel and using --max-stack can help when it is really needed, but yes, work on it is *badly* needed.

- Arnaldo

#4 lore raw
Quoted 28 lines · Arnaldo Carvalho de Melo

Em Fri, Nov 15, 2013 at 06:46:09AM +0100, Ingo Molnar escreveu:

Quoted 14 lines

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                     
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf    
Quoted 8 lines

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf      

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

Yeah, top dwarf callchain has been so far a proof of concept, it exacerbates problems that can be seen on 'report', but since its live, we can see it more clearly.

Quoted 5 lines

The work on improving callchain processing, (rb_tree'ing, new comm infrastructure) alleviated the problem a bit.

Tuning the stack size requested from the kernel and using --max-stack can help when it is really needed, but yes, work on it is *badly* needed.

agreed ;-)

also there's new remote unwind interface recently added into libdw, which seems to be faster than libunwind. I plan on adding this soon.

jirka

#5 lore raw
Quoted 36 lines · Jiri Olsa
Quoted 28 lines · Arnaldo Carvalho de Melo

Em Fri, Nov 15, 2013 at 06:46:09AM +0100, Ingo Molnar escreveu:

Quoted 14 lines

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                     
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf    
Quoted 8 lines

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf      

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

Yeah, top dwarf callchain has been so far a proof of concept, it exacerbates problems that can be seen on 'report', but since its live, we can see it more clearly.

Quoted 6 lines

The work on improving callchain processing, (rb_tree'ing, new comm infrastructure) alleviated the problem a bit.

Tuning the stack size requested from the kernel and using --max-stack can help when it is really needed, but yes, work on it is *badly* needed.

Quoted 6 lines

agreed ;-)

also there's new remote unwind interface recently added into libdw, which seems to be faster than libunwind.

I plan on adding this soon.

If the main source of overhead is libunwind (which needs independent confirmation) then would it make sense to implement dwarf stack unwind support ourselves?

I think SysProf does that and it appears to be faster - its unwind.c is only 400 lines long as it only implements the small subset needed to walk the stack - AFAICS.

Thanks,

Ingo

#6 lore raw
Quoted 46 lines · Ingo Molnar
Quoted 36 lines · Jiri Olsa
Quoted 28 lines · Arnaldo Carvalho de Melo

Em Fri, Nov 15, 2013 at 06:46:09AM +0100, Ingo Molnar escreveu:

Quoted 14 lines

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                     
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf    
Quoted 8 lines

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf      

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

Yeah, top dwarf callchain has been so far a proof of concept, it exacerbates problems that can be seen on 'report', but since its live, we can see it more clearly.

Quoted 6 lines

The work on improving callchain processing, (rb_tree'ing, new comm infrastructure) alleviated the problem a bit.

Tuning the stack size requested from the kernel and using --max-stack can help when it is really needed, but yes, work on it is *badly* needed.

Quoted 6 lines

agreed ;-)

also there's new remote unwind interface recently added into libdw, which seems to be faster than libunwind.

I plan on adding this soon.

Quoted 7 lines

If the main source of overhead is libunwind (which needs independent confirmation) then would it make sense to implement dwarf stack unwind support ourselves?

I think SysProf does that and it appears to be faster - its unwind.c is only 400 lines long as it only implements the small subset needed to walk the stack - AFAICS.

I think it's an option.. but it'll simpler to try the libdw interface first and see if it's good/fast enough..

also I recall discussing the speed with libdw developer Jan Kratochvil (CC-ed) and AFAICS they're open for suggestions/optimizations

jirka

#7 lore raw
Quoted 55 lines · Jiri Olsa
Quoted 46 lines · Ingo Molnar
Quoted 36 lines · Jiri Olsa
Quoted 28 lines · Arnaldo Carvalho de Melo

Em Fri, Nov 15, 2013 at 06:46:09AM +0100, Ingo Molnar escreveu:

Quoted 14 lines

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                     
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf    
Quoted 8 lines

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf      

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

Yeah, top dwarf callchain has been so far a proof of concept, it exacerbates problems that can be seen on 'report', but since its live, we can see it more clearly.

Quoted 6 lines

The work on improving callchain processing, (rb_tree'ing, new comm infrastructure) alleviated the problem a bit.

Tuning the stack size requested from the kernel and using --max-stack can help when it is really needed, but yes, work on it is *badly* needed.

Quoted 6 lines

agreed ;-)

also there's new remote unwind interface recently added into libdw, which seems to be faster than libunwind.

I plan on adding this soon.

Quoted 7 lines

If the main source of overhead is libunwind (which needs independent confirmation) then would it make sense to implement dwarf stack unwind support ourselves?

I think SysProf does that and it appears to be faster - its unwind.c is only 400 lines long as it only implements the small subset needed to walk the stack - AFAICS.

Quoted 6 lines

I think it's an option.. but it'll simpler to try the libdw interface first and see if it's good/fast enough..

also I recall discussing the speed with libdw developer Jan Kratochvil (CC-ed) and AFAICS they're open for suggestions/optimizations

So it's terribly difficult to measure the performance problems, do something like this on an idle system:

  $ perf top --call-graph dwarf

and unless you have a very, very fast CPU this is going to use up 100% of CPU time. 20% on a very fast system. Both are anomalous and show this kind of dwarf processing overhead:

#
# Overhead Command Shared Object Symbol

# ........  .......  .........................  .................................................
#
     7.08%     perf  perf                       [.] access_mem
     7.03%     perf  perf                       [.] dso__data_read_offset
     5.83%     perf  perf                       [.] maps__find
     5.64%     perf  libunwind-x86_64.so.8.0.1  [.] 0x000000000000ba25
     4.75%     perf  perf                       [.] thread__find_addr_map
     3.81%     perf  [kernel.kallsyms]          [k] unmap_single_vma
     2.57%     perf  perf                       [.] map__map_ip
     2.48%     perf  libelf-0.156.so            [.] 0x0000000000003a84
     2.12%     perf  [kernel.kallsyms]          [k] memset
     2.12%     perf  perf                       [.] dso__data_read_addr
     2.10%     perf  libc-2.17.so               [.] __memcpy_sse2
     1.72%     perf  libc-2.17.so               [.] __memset_sse2
     1.58%     perf  [kernel.kallsyms]          [k] page_fault
     1.56%     perf  libc-2.17.so               [.] __memset_x86_64
     1.44%     perf  perf                       [.] find_proc_info
     1.25%     perf  libelf-0.156.so            [.] elf_end
     1.19%     perf  [kernel.kallsyms]          [k] flush_tlb_mm_range
     1.06%     perf  libc-2.17.so               [.] vfprintf
     1.04%     perf  libunwind-x86_64.so.8.0.1  [.] _Ux86_64_dwarf_search_unwind_table
     1.00%     perf  [kernel.kallsyms]          [k] __audit_syscall_exit
     0.94%     perf  libc-2.17.so               [.] _int_free
     0.92%     perf  libc-2.17.so               [.] _int_malloc
     0.84%     perf  libc-2.17.so               [.] __memcmp_sse2
     0.81%     perf  [kernel.kallsyms]          [k] unmapped_area_topdown
     0.71%     perf  [kernel.kallsyms]          [k] system_call
     0.71%     perf  [kernel.kallsyms]          [k] system_call_after_swapgs
     0.65%     perf  [kernel.kallsyms]          [k] sysret_check
     0.63%     perf  perf                       [.] dso__find_symbol
     0.58%     perf  [kernel.kallsyms]          [k] clear_page_c
     0.58%     perf  [kernel.kallsyms]          [k] handle_mm_fault
     0.56%     perf  libc-2.17.so               [.] __sigprocmask

the libunwind and libelf entries didn't get resolved because I didn't have a debug version of the libraries installed:

     5.64%     perf  libunwind-x86_64.so.8.0.1  [.] 0x000000000000ba25
     2.48%     perf  libelf-0.156.so            [.] 0x0000000000003a84

Btw., tools like GDB are able to resolve symbols in such cases even without debug packages installed:

(gdb) bt

#0  0x0000003e5908edf9 in __memcpy_sse2 () from /lib64/libc.so.6
#1  0x000000000046b61c in memcpy (__len=8, __src=<optimized out>, __dest=0x7fffc80b09b8) at /usr/include/bits/string3.h:51
#2  dso_cache__memcpy (size=8, data=0x7fffc80b09b8 "@\325\357\377\344\001", offset=1840096, cache=<optimized out>) at util/dso.c:259
#3  dso_cache_read (size=8, data=0x7fffc80b09b8 "@\325\357\377\344\001", offset=1840096, machine=0x9a2a48, dso=0x9b21a0) at util/dso.c:316
#4  dso__data_read_offset (dso=0x9b21a0, machine=0x9a2a48, offset=1840096, data=data@entry=0x7fffc80b09b8 "@\325\357\377\344\001", size=size@entry=8) at util/dso.c:330
#5  0x000000000046b7a5 in dso__data_read_addr (dso=<optimized out>, map=<optimized out>, machine=<optimized out>, addr=addr@entry=6034400, 
    data=data@entry=0x7fffc80b09b8 "@\325\357\377\344\001", size=size@entry=8) at util/dso.c:355
#6  0x00000000004bea3c in access_dso_mem (ui=0x7fffc80b18b0, ui=0x7fffc80b18b0, data=0x7fffc80b09b8, addr=6034400) at util/unwind.c:404
#7  access_mem (as=<optimized out>, addr=6034400, valp=0x7fffc80b09b8, __write=<optimized out>, arg=0x7fffc80b18b0) at util/unwind.c:455
#8  0x00007f885af02f2d in _Ux86_64_dwarf_read_encoded_pointer () from /lib64/libunwind-x86_64.so.8
#9  0x00007f885aeff992 in _Ux86_64_dwarf_extract_proc_info_from_fde () from /lib64/libunwind-x86_64.so.8

#10 0x00007f885af03e75 in _Ux86_64_dwarf_search_unwind_table () from /lib64/libunwind-x86_64.so.8
#11 0x00000000004bedbc in find_proc_info (as=0x1445560, ip=4975163, pi=0x7fffc80b15b0, need_unwind_info=1, arg=0x7fffc80b18b0) at util/unwind.c:335 #12 0x00007f885af00205 in fetch_proc_info () from /lib64/libunwind-x86_64.so.8
#13 0x00007f885af0246b in _Ux86_64_dwarf_find_save_locs () from /lib64/libunwind-x86_64.so.8
#14 0x00007f885af03769 in _Ux86_64_dwarf_step () from /lib64/libunwind-x86_64.so.8
#15 0x00007f885aefb3f1 in _Ux86_64_step () from /lib64/libunwind-x86_64.so.8

All those entries are within libunwind - and GDB was able to resolve them.

How do they do it and shouldn't perf be able to do such magick?

Thanks,

Ingo

#8 lore raw
In reply to Jiri Olsa
Quoted 2 lines · Jiri Olsa

I think it's an option.. but it'll simpler to try the libdw interface first and see if it's good/fast enough..

The elfutils libdw unwinder is being upstreamed these weeks, the x86* unwinder itself is already upstream now.

Quoted 1 line

also I recall discussing the speed with libdw developer

My tests with perf using elfutils unwinder were 10x faster than libunwind; this is by some simple caching of ELF files. Sure a similar cache could be implemented also for libunwind. But the cache is a wrong solution.

The problem is that currently perf loads the ELF files again and again for every process as the ELF file always gets automatically relocated for the address where it was loaded. The right way is to load the ELF file only once and access always the same copy with process-specific displacement only.

I did not investigate how much feasible it is with libunwind. For elfutils there is pmachata/sharing unfinished branch from 2008 to implement that. I have not checked it more before the unwinder gets fully upstreamed.

Jan

#11 lore raw
In reply to Arnaldo Carvalho de Melo

Hi,

Quoted 21 lines · Arnaldo Carvalho de Melo

Em Fri, Nov 15, 2013 at 06:46:09AM +0100, Ingo Molnar escreveu:

Quoted 20 lines

btw., here's some 'perf top' call graph performance and profiling quality feedback, with the latest perf code:

'perf top --call-graph fp' now works very well, using just 0.2% of CPU time on a fast system:

4676 mingo 20 0 612m 56m 9948 S 1 0.2 0:00.68 perf

'perf top --call-graph dwarf' on the other hand is horrendously slow, using 20% of CPU time on a 4 GHz CPU:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 4646 mingo     20   0  658m  81m  12m R    19  0.3   0:18.17 perf

On another system with a 2.4GHz CPU it's taking up 100% of CPU time (!):

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8018 mingo     20   0  290320  45220   8520 R  99.5  0.3   0:58.81 perf
Quoted 6 lines
Quoted 2 lines

Profiling 'perf top' shows all sorts of very high dwarf processing overhead:

Yeah, top dwarf callchain has been so far a proof of concept, it exacerbates problems that can be seen on 'report', but since its live, we can see it more clearly.

Indeed. Because of the poor performance of the dwarf unwinding code, the only practical use is to record the data (perf record) and then later parse it (perf report). perf top does all that at once.

Quoted 5 lines

The work on improving callchain processing, (rb_tree'ing, new comm infrastructure) alleviated the problem a bit.

Tuning the stack size requested from the kernel and using --max-stack can help when it is really needed, but yes, work on it is *badly* needed.

The problem is that the whole user stack is dumped for every sample while frame pointer unwinding only dumps the useful part of the callchain.

Also an important point is the robustness of libunwind wrt async
signal, cf. http://lists.nongnu.org/archive/html/libunwind-devel/2013-09/msg00005.html.

So yes some work is °badly* needed on:
- the data size being dumped,
- the data parsing optimization,
- the choice of an implementation of dwarf unwinding (libunwind, libdw etc.),
- the compatibility with 32 bit binaries on AARCH64, which I am now busy with in libunwind.

Jean

Quoted 7 lines

- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#9 lore raw
In reply to Jan Kratochvil

Hi,

Quoted 6 lines · Jan Kratochvil
Quoted 2 lines · Jiri Olsa

I think it's an option.. but it'll simpler to try the libdw interface first and see if it's good/fast enough..

The elfutils libdw unwinder is being upstreamed these weeks, the x86* unwinder itself is already upstream now.

Do you know about the support of AARCH64, both in 64-bit and 32-bit (compat) mode?
I would be glad to give it a try.

Quoted 11 lines
Quoted 1 line

also I recall discussing the speed with libdw developer

My tests with perf using elfutils unwinder were 10x faster than libunwind; this is by some simple caching of ELF files. Sure a similar cache could be implemented also for libunwind. But the cache is a wrong solution.

The problem is that currently perf loads the ELF files again and again for every process as the ELF file always gets automatically relocated for the address where it was loaded. The right way is to load the ELF file only once and access always the same copy with process-specific displacement only.

Quoted 6 lines

I did not investigate how much feasible it is with libunwind. For elfutils there is pmachata/sharing unfinished branch from 2008 to implement that. I have not checked it more before the unwinder gets fully upstreamed.

Jan

Jean

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#12 lore raw
In reply to David Ahern
perf top: Make -g refer to callchains
Commit-ID:  bf80669e4f689f181f23a54dfe2a0f264147ad67
Gitweb:     http://git.kernel.org/tip/bf80669e4f689f181f23a54dfe2a0f264147ad67
Author:     David Ahern <[email protected]>
AuthorDate: Thu, 14 Nov 2013 20:51:30 -0700
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Wed, 27 Nov 2013 14:58:35 -0300

perf top: Make -g refer to callchains

In most commands -g is used for callchains. Make perf-top follow suit. Move group to just --group with no short cut making it similar to perf-record.

---
 tools/perf/Documentation/perf-top.txt | 5 ++---
 tools/perf/builtin-top.c              | 4 ++--
 2 files changed, 4 insertions(+), 5 deletions(-)
tools/perf/Documentation/perf-top.txt +2 -3
index 7de01dd..cdd8d49 100644
@@ -50,7 +50,6 @@ Default is to monitor all CPUS.
 --count-filter=<count>::
 	Only display functions with more events than this.
 
--g::
 --group::
         Put the counters into a counter group.
 
@@ -143,12 +142,12 @@ Default is to monitor all CPUS.
 --asm-raw::
 	Show raw instruction encoding of assembly instructions.
 
--G::
+-g::
 	Enables call-graph (stack chain/backtrace) recording.
 
 --call-graph::
 	Setup and enable call-graph (stack chain/backtrace) recording,
-	implies -G.
+	implies -g.
 
 --max-stack::
 	Set the stack depth limit when parsing the callchain, anything
tools/perf/builtin-top.c +2 -2
index 71e6402..531522d 100644
@@ -1084,7 +1084,7 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)
 			    "dump the symbol table used for profiling"),
 	OPT_INTEGER('f', "count-filter", &top.count_filter,
 		    "only display functions with more events than this"),
-	OPT_BOOLEAN('g', "group", &opts->group,
+	OPT_BOOLEAN(0, "group", &opts->group,
 			    "put the counters into a counter group"),
 	OPT_BOOLEAN('i', "no-inherit", &opts->no_inherit,
 		    "child tasks do not inherit counters"),
@@ -1105,7 +1105,7 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)
 		   " abort, in_tx, transaction"),
 	OPT_BOOLEAN('n', "show-nr-samples", &symbol_conf.show_nr_samples,
 		    "Show a column with the number of samples"),
-	OPT_CALLBACK_NOOPT('G', NULL, &top.record_opts,
+	OPT_CALLBACK_NOOPT('g', NULL, &top.record_opts,
 			   NULL, "enables call-graph recording",
 			   &callchain_opt),
 	OPT_CALLBACK(0, "call-graph", &top.record_opts,