Skip to contentSkip to conversation

perf evsel: Improve callchain warning for s390

PATCH Applied Jens Remus 5 participants 5 messages Sep 22 – Sep 24

Subsystems (2)
lore
Share & tools
lore

Jens Remus

author
#0 lore raw

PATCH 1/1 perf evsel: Improve callchain warning for s390

On s390 the kernel uses s390 back chain instead of frame pointers for stack tracing of user space since v6.7 commit aa44433ac4ee ("s390: add USER_STACKTRACE support"). This is because frame pointers on s390 cannot be used for stack tracing. [1]

This requires user space to maintain a s390 back chain. For instance user space to be built with compiler option '-mbackchain' (instead of '-fno-omit-frame-pointer' used on other architectures, which should better not be used on s390 [1]). Only few distributions and users build user space with '-mbackchain'. Therefore '--call-graph fp' may in practice not produce the expected results.

Commit ca76fb67ebdd ("perf evlist: Improve default event for s390") changed the default '-g' option to 'dwarf' on s390 and added a warning for s390 that wrongly claimed that "Framepointer unwinding lacks kernel support". This warning resulted from a misinterpretation of comments in s390 cpumsf_pmu_event_init().

The restriction in cpumsf_pmu_event_init() applies to callchain sampling with the s390 CPU Measurement Sampling Facility (CPUMSF) hardware PMU. CPUMSF events, such as 'cycles', do not support callchain sampling. This is independent of whether perf uses the 'fp' or 'dwarf' call-graph mode. The CPUMSF PMU provides samples collected asynchronously, making it impossible to associate a callchain to the historic IPs. Therefore callchains can only be used with software events on s390.

Remove the incorrect warning. The kernel supports the 'fp' call-graph mode on s390 by walking the back chain. Do not replace it with a hint to use 'dwarf' when the resulting callchain is incomplete. Such a suggestion could imply that 'fp' is inherently inferior. The same general limitation exists on other architectures when user space is not built with frame pointers (i.e. '-fno-omit-frame-pointers').

Instead, warn on s390 when callchain sampling is requested with an event provided by the CPUMCF or CPUMSF hardware PMUs, because that combination is not supported.

Update the perf record documentation to state that 'dwarf' is the default call-graph mode on s390 and that 'fp' uses back chain instead of frame pointers on s390.

Note that '--call-graph fp' may also be useful for other applications, such as OpenJDK maintaining a s390 back chain (does not require JVM option '-XX:+PreserveFramePointer' on s390):

  $ perf record --call-graph fp ... -- \
    java -XX:+UnlockDiagnosticVMOptions -XX:+DumpPerfMapAtExit ...

[1]: s390: Stack tracing using Frame Pointer, Back Chain, and SFrame,

     https://conf.gnu-tools-cauldron.org/opo25/talk/Y3CVHY/
Fixes ca76fb67ebdd ("perf evlist: Improve default event for s390")

---

Notes (jremus):
Changes in v3:
- Updated warning message.
- Rebased on v7.3-rc4.

 tools/perf/Documentation/perf-record.txt | 7 ++++++-
 tools/perf/util/evsel.c                  | 6 +++---
 2 files changed, 9 insertions(+), 4 deletions(-)
tools/perf/Documentation/perf-record.txt +6 -1
index 178f483140ed..dc72606cb25d 100644
@@ -293,7 +293,7 @@ OPTIONS
 
 --call-graph::
 	Setup and enable call-graph (stack chain/backtrace) recording,
-	implies -g.  Default is "fp" (for user space).
+	implies -g.  Default is "fp" for user space (s390 uses "dwarf").
 
 	The unwinding method used for kernel space is dependent on the
 	unwinder used by the active kernel configuration, i.e
@@ -329,6 +329,11 @@ OPTIONS
 	enable deferred user callchain which will collect user-space callchains
 	when the thread returns to the user space.
 
+	On s390, the "fp" method uses back chain instead of frame pointers,
+	which requires user space to maintain a back chain (e.g. compiler
+	option -mbackchain).  Without a maintained back chain this produces
+	bogus call graphs.  In that case use "dwarf" instead.
+
 -q::
 --quiet::
 	Don't print any warnings or messages, useful for scripting.
tools/perf/util/evsel.c +3 -3
index d4cb455f4a7d..55297b8dab37 100644
@@ -1092,9 +1092,9 @@ static void __evsel__config_callchain(struct evsel *evsel, const struct record_o
 	bool function = evsel__is_function_event(evsel);
 	struct perf_event_attr *attr = &evsel->core.attr;
 
-	if (EM_HOST == EM_S390 && param->record_mode == CALLCHAIN_FP) {
-		pr_warning_once(
-			"Framepointer unwinding lacks kernel support. Use '--call-graph dwarf'\n");
+	if (EM_HOST == EM_S390 && evsel->pmu && !perf_pmu__is_software(evsel->pmu)) {
+		pr_warning_once("Cannot use hardware PMU 'cpum_cf' / 'cpum_sf' event with callchain. "
+				"Use a software 'cpu-clock' / 'task-clock' event.\n");
 	}
 
 	evsel__set_sample_bit(evsel, CALLCHAIN);

base-commit: 93f51579e7df248780214094418f205253383cc5

Signature · 1 line
2.53.0
#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 17 lines

On s390 the kernel uses s390 back chain instead of frame pointers for stack tracing of user space since v6.7 commit aa44433ac4ee ("s390: add USER_STACKTRACE support"). This is because frame pointers on s390 cannot be used for stack tracing. [1]

This requires user space to maintain a s390 back chain. For instance user space to be built with compiler option '-mbackchain' (instead of '-fno-omit-frame-pointer' used on other architectures, which should better not be used on s390 [1]). Only few distributions and users build user space with '-mbackchain'. Therefore '--call-graph fp' may in practice not produce the expected results.

Commit ca76fb67ebdd ("perf evlist: Improve default event for s390") changed the default '-g' option to 'dwarf' on s390 and added a warning for s390 that wrongly claimed that "Framepointer unwinding lacks kernel [ ... ]
[1]: s390: Stack tracing using Frame Pointer, Back Chain, and SFrame,

Quoted 4 lines
     https://conf.gnu-tools-cauldron.org/opo25/talk/Y3CVHY/
Fixes ca76fb67ebdd ("perf evlist: Improve default event for s390")

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
reviewer Acked-by on v1 1/1
#2 lore raw
In reply to Jens Remus
Quoted 46 lines · Jens Remus

On s390 the kernel uses s390 back chain instead of frame pointers for stack tracing of user space since v6.7 commit aa44433ac4ee ("s390: add USER_STACKTRACE support"). This is because frame pointers on s390 cannot be used for stack tracing. [1]

This requires user space to maintain a s390 back chain. For instance user space to be built with compiler option '-mbackchain' (instead of '-fno-omit-frame-pointer' used on other architectures, which should better not be used on s390 [1]). Only few distributions and users build user space with '-mbackchain'. Therefore '--call-graph fp' may in practice not produce the expected results.

Commit ca76fb67ebdd ("perf evlist: Improve default event for s390") changed the default '-g' option to 'dwarf' on s390 and added a warning for s390 that wrongly claimed that "Framepointer unwinding lacks kernel support". This warning resulted from a misinterpretation of comments in s390 cpumsf_pmu_event_init().

The restriction in cpumsf_pmu_event_init() applies to callchain sampling with the s390 CPU Measurement Sampling Facility (CPUMSF) hardware PMU. CPUMSF events, such as 'cycles', do not support callchain sampling. This is independent of whether perf uses the 'fp' or 'dwarf' call-graph mode. The CPUMSF PMU provides samples collected asynchronously, making it impossible to associate a callchain to the historic IPs. Therefore callchains can only be used with software events on s390.

Remove the incorrect warning. The kernel supports the 'fp' call-graph mode on s390 by walking the back chain. Do not replace it with a hint to use 'dwarf' when the resulting callchain is incomplete. Such a suggestion could imply that 'fp' is inherently inferior. The same general limitation exists on other architectures when user space is not built with frame pointers (i.e. '-fno-omit-frame-pointers').

Instead, warn on s390 when callchain sampling is requested with an event provided by the CPUMCF or CPUMSF hardware PMUs, because that combination is not supported.

Update the perf record documentation to state that 'dwarf' is the default call-graph mode on s390 and that 'fp' uses back chain instead of frame pointers on s390.

Note that '--call-graph fp' may also be useful for other applications, such as OpenJDK maintaining a s390 back chain (does not require JVM option '-XX:+PreserveFramePointer' on s390):

Quoted 8 lines
  $ perf record --call-graph fp ... -- \
    java -XX:+UnlockDiagnosticVMOptions -XX:+DumpPerfMapAtExit ...

[1]: s390: Stack tracing using Frame Pointer, Back Chain, and SFrame,

     https://conf.gnu-tools-cauldron.org/opo25/talk/Y3CVHY/
Fixes ca76fb67ebdd ("perf evlist: Improve default event for s390")

Thanks,
Ian

Quoted 58 lines

---

Notes (jremus):
Changes in v3:
- Updated warning message.
- Rebased on v7.3-rc4.

 tools/perf/Documentation/perf-record.txt | 7 ++++++-
 tools/perf/util/evsel.c                  | 6 +++---
 2 files changed, 9 insertions(+), 4 deletions(-)
tools/perf/Documentation/perf-record.txt +6 -1
index 178f483140ed..dc72606cb25d 100644
@@ -293,7 +293,7 @@ OPTIONS
 
 --call-graph::
        Setup and enable call-graph (stack chain/backtrace) recording,
-       implies -g.  Default is "fp" (for user space).
+       implies -g.  Default is "fp" for user space (s390 uses "dwarf").
 
        The unwinding method used for kernel space is dependent on the
        unwinder used by the active kernel configuration, i.e
@@ -329,6 +329,11 @@ OPTIONS
        enable deferred user callchain which will collect user-space callchains
        when the thread returns to the user space.
 
+       On s390, the "fp" method uses back chain instead of frame pointers,
+       which requires user space to maintain a back chain (e.g. compiler
+       option -mbackchain).  Without a maintained back chain this produces
+       bogus call graphs.  In that case use "dwarf" instead.
+
 -q::
 --quiet::
        Don't print any warnings or messages, useful for scripting.
tools/perf/util/evsel.c +3 -3
index d4cb455f4a7d..55297b8dab37 100644
@@ -1092,9 +1092,9 @@ static void __evsel__config_callchain(struct evsel *evsel, const struct record_o
        bool function = evsel__is_function_event(evsel);
        struct perf_event_attr *attr = &evsel->core.attr;
 
-       if (EM_HOST == EM_S390 && param->record_mode == CALLCHAIN_FP) {
-               pr_warning_once(
-                       "Framepointer unwinding lacks kernel support. Use '--call-graph dwarf'\n");
+       if (EM_HOST == EM_S390 && evsel->pmu && !perf_pmu__is_software(evsel->pmu)) {
+               pr_warning_once("Cannot use hardware PMU 'cpum_cf' / 'cpum_sf' event with callchain. "
+                               "Use a software 'cpu-clock' / 'task-clock' event.\n");
        }
 
        evsel__set_sample_bit(evsel, CALLCHAIN);

base-commit: 93f51579e7df248780214094418f205253383cc5 --
2.53.0

Reviewed-by
#3 lore raw
Quoted 113 lines · Ian Rogers
Quoted 46 lines · Jens Remus

On s390 the kernel uses s390 back chain instead of frame pointers for stack tracing of user space since v6.7 commit aa44433ac4ee ("s390: add USER_STACKTRACE support"). This is because frame pointers on s390 cannot be used for stack tracing. [1]

This requires user space to maintain a s390 back chain. For instance user space to be built with compiler option '-mbackchain' (instead of '-fno-omit-frame-pointer' used on other architectures, which should better not be used on s390 [1]). Only few distributions and users build user space with '-mbackchain'. Therefore '--call-graph fp' may in practice not produce the expected results.

Commit ca76fb67ebdd ("perf evlist: Improve default event for s390") changed the default '-g' option to 'dwarf' on s390 and added a warning for s390 that wrongly claimed that "Framepointer unwinding lacks kernel support". This warning resulted from a misinterpretation of comments in s390 cpumsf_pmu_event_init().

The restriction in cpumsf_pmu_event_init() applies to callchain sampling with the s390 CPU Measurement Sampling Facility (CPUMSF) hardware PMU. CPUMSF events, such as 'cycles', do not support callchain sampling. This is independent of whether perf uses the 'fp' or 'dwarf' call-graph mode. The CPUMSF PMU provides samples collected asynchronously, making it impossible to associate a callchain to the historic IPs. Therefore callchains can only be used with software events on s390.

Remove the incorrect warning. The kernel supports the 'fp' call-graph mode on s390 by walking the back chain. Do not replace it with a hint to use 'dwarf' when the resulting callchain is incomplete. Such a suggestion could imply that 'fp' is inherently inferior. The same general limitation exists on other architectures when user space is not built with frame pointers (i.e. '-fno-omit-frame-pointers').

Instead, warn on s390 when callchain sampling is requested with an event provided by the CPUMCF or CPUMSF hardware PMUs, because that combination is not supported.

Update the perf record documentation to state that 'dwarf' is the default call-graph mode on s390 and that 'fp' uses back chain instead of frame pointers on s390.

Note that '--call-graph fp' may also be useful for other applications, such as OpenJDK maintaining a s390 back chain (does not require JVM option '-XX:+PreserveFramePointer' on s390):

Quoted 8 lines
  $ perf record --call-graph fp ... -- \
    java -XX:+UnlockDiagnosticVMOptions -XX:+DumpPerfMapAtExit ...

[1]: s390: Stack tracing using Frame Pointer, Back Chain, and SFrame,

     https://conf.gnu-tools-cauldron.org/opo25/talk/Y3CVHY/
Fixes ca76fb67ebdd ("perf evlist: Improve default event for s390")

Thanks,
Ian

Quoted 52 lines

---

Notes (jremus):
Changes in v3:
- Updated warning message.
- Rebased on v7.3-rc4.

 tools/perf/Documentation/perf-record.txt | 7 ++++++-
 tools/perf/util/evsel.c                  | 6 +++---
 2 files changed, 9 insertions(+), 4 deletions(-)
tools/perf/Documentation/perf-record.txt +6 -1
index 178f483140ed..dc72606cb25d 100644
@@ -293,7 +293,7 @@ OPTIONS
 
 --call-graph::
        Setup and enable call-graph (stack chain/backtrace) recording,
-       implies -g.  Default is "fp" (for user space).
+       implies -g.  Default is "fp" for user space (s390 uses "dwarf").
 
        The unwinding method used for kernel space is dependent on the
        unwinder used by the active kernel configuration, i.e
@@ -329,6 +329,11 @@ OPTIONS
        enable deferred user callchain which will collect user-space callchains
        when the thread returns to the user space.
 
+       On s390, the "fp" method uses back chain instead of frame pointers,
+       which requires user space to maintain a back chain (e.g. compiler
+       option -mbackchain).  Without a maintained back chain this produces
+       bogus call graphs.  In that case use "dwarf" instead.
+
 -q::
 --quiet::
        Don't print any warnings or messages, useful for scripting.
tools/perf/util/evsel.c +3 -3
index d4cb455f4a7d..55297b8dab37 100644
@@ -1092,9 +1092,9 @@ static void __evsel__config_callchain(struct evsel *evsel, const struct record_o
        bool function = evsel__is_function_event(evsel);
        struct perf_event_attr *attr = &evsel->core.attr;
 
-       if (EM_HOST == EM_S390 && param->record_mode == CALLCHAIN_FP) {
-               pr_warning_once(
-                       "Framepointer unwinding lacks kernel support. Use '--call-graph dwarf'\n");
+       if (EM_HOST == EM_S390 && evsel->pmu && !perf_pmu__is_software(evsel->pmu)) {
+               pr_warning_once("Cannot use hardware PMU 'cpum_cf' / 'cpum_sf' event with callchain. "
+                               "Use a software 'cpu-clock' / 'task-clock' event.\n");
        }
 
Quoted 6 lines
Quoted 6 lines
tools/perf/util/evsel.c +0 -0
        evsel__set_sample_bit(evsel, CALLCHAIN);

base-commit: 93f51579e7df248780214094418f205253383cc5 --
2.53.0

Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany
--
IBM Deutschland Research & Development GmbH

Vorsitzender des Aufsichtsrats: Wolfgang Wendt

Geschäftsführung: David Faller

Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
maintainer Applied
#4 lore raw
Quoted 6 lines · Thomas Richter
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Jens Remus

On s390 the kernel uses s390 back chain instead of frame pointers for stack tracing of user space since v6.7 commit aa44433ac4ee ("s390: add USER_STACKTRACE support"). This is because frame pointers on s390 cannot be used for stack tracing. [1]

<SNIP>

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Fixes ca76fb67ebdd ("perf evlist: Improve default event for s390")

Thanks, applied to perf-tools-next, for v7.4.

- Arnaldo