Skip to contentSkip to conversation

perf probe powerpc: Fix trace event post-processing

PATCH Applied Sandipan Das 4 participants 4 messages Aug 9, 2018 – Aug 18, 2018 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf probe powerpc: Fix trace event post-processing

In some cases, a symbol may have multiple aliases. Attempting to add an entry probe for such symbols results in a probe being added at an incorrect location while it fails altogether for return probes. This is only applicable for binaries with debug information.

During the arch-dependent post-processing, the offset from the start of the symbol at which the probe is to be attached is determined and added to the start address of the symbol to get the probe's location. In case there are multiple aliases, this offset gets added multiple times for each alias of the symbol and we end up with an incorrect probe location.

This can be verified on a powerpc64le system as shown below.

  $ nm /lib/modules/$(uname -r)/build/vmlinux | grep "sys_open$"
  ...
  c000000000414290 T __se_sys_open
  c000000000414290 T sys_open
  $ objdump -d /lib/modules/$(uname -r)/build/vmlinux | grep -A 10 "<__se_sys_open>:"

c000000000414290 <__se_sys_open>:

  c000000000414290:       19 01 4c 3c     addis   r2,r12,281
  c000000000414294:       70 c4 42 38     addi    r2,r2,-15248
  c000000000414298:       a6 02 08 7c     mflr    r0
  c00000000041429c:       e8 ff a1 fb     std     r29,-24(r1)
  c0000000004142a0:       f0 ff c1 fb     std     r30,-16(r1)
  c0000000004142a4:       f8 ff e1 fb     std     r31,-8(r1)
  c0000000004142a8:       10 00 01 f8     std     r0,16(r1)
  c0000000004142ac:       c1 ff 21 f8     stdu    r1,-64(r1)
  c0000000004142b0:       78 23 9f 7c     mr      r31,r4
  c0000000004142b4:       78 1b 7e 7c     mr      r30,r3

For both the entry probe and the return probe, the probe location should be _text+4276888 (0xc000000000414298). Since another alias exists for 'sys_open', the post-processing code will end up adding the offset (8 for powerpc64le) twice and perf will attempt to add the probe at _text+4276896 (0xc0000000004142a0) instead.

Before:

  # perf probe -v -a sys_open

probe-definition(0): sys_open
symbol:sys_open file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Writing event: p:probe/sys_open _text+4276896
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

probe-definition(0): sys_open%return
symbol:sys_open file:(null) line:0 offset:0 return:1 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290
Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/README write=0
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Parsing probe_events: p:probe/sys_open _text+4276896 Group:probe Event:sys_open probe:p
Writing event: r:probe/sys_open__return _text+4276896 Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

After:

  # perf probe -v -a sys_open

probe-definition(0): sys_open
symbol:sys_open file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Writing event: p:probe/sys_open _text+4276888
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

probe-definition(0): sys_open%return
symbol:sys_open file:(null) line:0 offset:0 return:1 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/README write=0 Opening /sys/kernel/debug/tracing/kprobe_events write=1 Parsing probe_events: p:probe/sys_open _text+4276888 Group:probe Event:sys_open probe:p
Writing event: r:probe/sys_open__return _text+4276888 Added new event:
probe:sys_open__return (on sys_open%return)
...

Fixes 99e608b5954c ("perf probe ppc64le: Fix probe location when using DWARF")
---
 tools/perf/arch/powerpc/util/sym-handling.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/perf/arch/powerpc/util/sym-handling.c +3 -1
index 53d83d7e6a09..20e7d74d86cd 100644
@@ -141,8 +141,10 @@ void arch__post_process_probe_trace_events(struct perf_probe_event *pev,
 	for (i = 0; i < ntevs; i++) {
 		tev = &pev->tevs[i];
 		map__for_each_symbol(map, sym, tmp) {
-			if (map->unmap_ip(map, sym->start) == tev->point.address)
+			if (map->unmap_ip(map, sym->start) == tev->point.address) {
 				arch__fix_tev_from_maps(pev, tev, map, sym);
+				break;
+			}
 		}
 	}
 }
Signature · 1 line
2.14.4
Acked-by on v1 1/1
#1 lore raw
Quoted 124 lines · Sandipan Das

In some cases, a symbol may have multiple aliases. Attempting to add an entry probe for such symbols results in a probe being added at an incorrect location while it fails altogether for return probes. This is only applicable for binaries with debug information.

During the arch-dependent post-processing, the offset from the start of the symbol at which the probe is to be attached is determined and added to the start address of the symbol to get the probe's location. In case there are multiple aliases, this offset gets added multiple times for each alias of the symbol and we end up with an incorrect probe location.

This can be verified on a powerpc64le system as shown below.

  $ nm /lib/modules/$(uname -r)/build/vmlinux | grep "sys_open$"
  ...
  c000000000414290 T __se_sys_open
  c000000000414290 T sys_open
  $ objdump -d /lib/modules/$(uname -r)/build/vmlinux | grep -A 10 "<__se_sys_open>:"

c000000000414290 <__se_sys_open>:

  c000000000414290:       19 01 4c 3c     addis   r2,r12,281
  c000000000414294:       70 c4 42 38     addi    r2,r2,-15248
  c000000000414298:       a6 02 08 7c     mflr    r0
  c00000000041429c:       e8 ff a1 fb     std     r29,-24(r1)
  c0000000004142a0:       f0 ff c1 fb     std     r30,-16(r1)
  c0000000004142a4:       f8 ff e1 fb     std     r31,-8(r1)
  c0000000004142a8:       10 00 01 f8     std     r0,16(r1)
  c0000000004142ac:       c1 ff 21 f8     stdu    r1,-64(r1)
  c0000000004142b0:       78 23 9f 7c     mr      r31,r4
  c0000000004142b4:       78 1b 7e 7c     mr      r30,r3

For both the entry probe and the return probe, the probe location should be _text+4276888 (0xc000000000414298). Since another alias exists for 'sys_open', the post-processing code will end up adding the offset (8 for powerpc64le) twice and perf will attempt to add the probe at _text+4276896 (0xc0000000004142a0) instead.

Before:

  # perf probe -v -a sys_open

probe-definition(0): sys_open
symbol:sys_open file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Writing event: p:probe/sys_open _text+4276896
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

probe-definition(0): sys_open%return
symbol:sys_open file:(null) line:0 offset:0 return:1 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290
Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/README write=0
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Parsing probe_events: p:probe/sys_open _text+4276896 Group:probe Event:sys_open probe:p
Writing event: r:probe/sys_open__return _text+4276896 Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

After:

  # perf probe -v -a sys_open

probe-definition(0): sys_open
symbol:sys_open file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Writing event: p:probe/sys_open _text+4276888
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

probe-definition(0): sys_open%return
symbol:sys_open file:(null) line:0 offset:0 return:1 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/README write=0 Opening /sys/kernel/debug/tracing/kprobe_events write=1 Parsing probe_events: p:probe/sys_open _text+4276888 Group:probe Event:sys_open probe:p
Writing event: r:probe/sys_open__return _text+4276888 Added new event:
probe:sys_open__return (on sys_open%return)
...

Quoted 3 lines
Fixes 99e608b5954c ("perf probe ppc64le: Fix probe location when using DWARF")

- Naveen

maintainer Applied
#2 lore raw

Em Thu, Aug 09, 2018 at 09:56:24PM +0530, Naveen N. Rao escreveu:

Quoted 1 line

Sandipan Das wrote:

<SNIP>

Quoted 1 line
Quoted 1 line

This can be verified on a powerpc64le system as shown below.

<SNIP>

Quoted 2 lines
Quoted 2 lines

Before:

  # perf probe -v -a sys_open

<SNIP>

Quoted 1 line
Quoted 1 line
  # perf probe -v -a sys_open%return $retval

<SNIP>

Quoted 4 lines
Quoted 4 lines

Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
After:

  # perf probe -v -a sys_open

<SNIP>

Quoted 6 lines
Quoted 6 lines

Writing event: p:probe/sys_open _text+4276888
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

<SNIP>

Quoted 8 lines
Quoted 6 lines

Writing event: r:probe/sys_open__return _text+4276888 Added new event:
probe:sys_open__return (on sys_open%return)

Fixes 99e608b5954c ("perf probe ppc64le: Fix probe location when using DWARF")

Thanks, applied!

Excellent commit log message, btw, keep it up.

- arnaldo

#3 lore raw
In reply to Sandipan Das
perf probe powerpc: Fix trace event post-processing
Commit-ID:  354b064b8ebc1e1ede58550ca9e08bfa81e6af43
Gitweb:     https://git.kernel.org/tip/354b064b8ebc1e1ede58550ca9e08bfa81e6af43
Author:     Sandipan Das <[email protected]>
AuthorDate: Thu, 9 Aug 2018 21:49:29 +0530
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Thu, 9 Aug 2018 14:40:11 -0300

perf probe powerpc: Fix trace event post-processing

In some cases, a symbol may have multiple aliases. Attempting to add an entry probe for such symbols results in a probe being added at an incorrect location while it fails altogether for return probes. This is only applicable for binaries with debug information.

During the arch-dependent post-processing, the offset from the start of the symbol at which the probe is to be attached is determined and added to the start address of the symbol to get the probe's location. In case there are multiple aliases, this offset gets added multiple times for each alias of the symbol and we end up with an incorrect probe location.

This can be verified on a powerpc64le system as shown below.

  $ nm /lib/modules/$(uname -r)/build/vmlinux | grep "sys_open$"
  ...
  c000000000414290 T __se_sys_open
  c000000000414290 T sys_open
  $ objdump -d /lib/modules/$(uname -r)/build/vmlinux | grep -A 10 "<__se_sys_open>:"

c000000000414290 <__se_sys_open>:

  c000000000414290:       19 01 4c 3c     addis   r2,r12,281
  c000000000414294:       70 c4 42 38     addi    r2,r2,-15248
  c000000000414298:       a6 02 08 7c     mflr    r0
  c00000000041429c:       e8 ff a1 fb     std     r29,-24(r1)
  c0000000004142a0:       f0 ff c1 fb     std     r30,-16(r1)
  c0000000004142a4:       f8 ff e1 fb     std     r31,-8(r1)
  c0000000004142a8:       10 00 01 f8     std     r0,16(r1)
  c0000000004142ac:       c1 ff 21 f8     stdu    r1,-64(r1)
  c0000000004142b0:       78 23 9f 7c     mr      r31,r4
  c0000000004142b4:       78 1b 7e 7c     mr      r30,r3

For both the entry probe and the return probe, the probe location should be _text+4276888 (0xc000000000414298). Since another alias exists for 'sys_open', the post-processing code will end up adding the offset (8 for powerpc64le) twice and perf will attempt to add the probe at _text+4276896 (0xc0000000004142a0) instead.

Before:

  # perf probe -v -a sys_open

probe-definition(0): sys_open
symbol:sys_open file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Writing event: p:probe/sys_open _text+4276896
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

probe-definition(0): sys_open%return
symbol:sys_open file:(null) line:0 offset:0 return:1 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290
Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/README write=0
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Parsing probe_events: p:probe/sys_open _text+4276896 Group:probe Event:sys_open probe:p
Writing event: r:probe/sys_open__return _text+4276896 Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

After:

  # perf probe -v -a sys_open

probe-definition(0): sys_open
symbol:sys_open file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1 Writing event: p:probe/sys_open _text+4276888
Added new event:
probe:sys_open (on sys_open)
...

  # perf probe -v -a sys_open%return $retval

probe-definition(0): sys_open%return
symbol:sys_open file:(null) line:0 offset:0 return:1 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.18.0-rc8+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.18.0-rc8+/build/vmlinux Try to find probe point from debuginfo.
Symbol sys_open address found : c000000000414290 Matched function: __se_sys_open [2ad03a0]
Probe point found: __se_sys_open+0
Found 1 probe_trace_events.
Opening /sys/kernel/debug/tracing/README write=0 Opening /sys/kernel/debug/tracing/kprobe_events write=1 Parsing probe_events: p:probe/sys_open _text+4276888 Group:probe Event:sys_open probe:p
Writing event: r:probe/sys_open__return _text+4276888 Added new event:
probe:sys_open__return (on sys_open%return)
...

Cc Aneesh Kumar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Ravi Bangoria <[email protected]>
Fixes 99e608b5954c ("perf probe ppc64le: Fix probe location when using DWARF")
---
 tools/perf/arch/powerpc/util/sym-handling.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/perf/arch/powerpc/util/sym-handling.c +3 -1
index 53d83d7e6a09..20e7d74d86cd 100644
@@ -141,8 +141,10 @@ void arch__post_process_probe_trace_events(struct perf_probe_event *pev,
 	for (i = 0; i < ntevs; i++) {
 		tev = &pev->tevs[i];
 		map__for_each_symbol(map, sym, tmp) {
-			if (map->unmap_ip(map, sym->start) == tev->point.address)
+			if (map->unmap_ip(map, sym->start) == tev->point.address) {
 				arch__fix_tev_from_maps(pev, tev, map, sym);
+				break;
+			}
 		}
 	}
 }