perf record dumps core all the time on s390 (6.6.0rc1)
Using Linux kernel version 6.6.0rc1, the command
# perf record -e cycles
dumps core immediately on s390 running on a z/VM virtual machine. It works on an s390 LPAR (same kernel).
This issue showed up during my vacation on 25-Aug-2023 in linux-next and since Monday 11-Sep-2023 it shows up in linux 6.6.0rc1.
The event cycles is treated as a software event and it fails in:
(gdb) n
1014 pmu->events_table = perf_pmu__find_events_table(pmu);
(gdb) p *pmu
$4 = {name = 0x152a150 "software", alias_name = 0x0, id = 0x0, type = 1,
selectable = false, is_core = false,
is_uncore = false, auxtrace = false, formats_checked = false, config_masks_present = false,
config_masks_computed = false, max_precise = -1, default_config = 0x0, cpus = 0x0, format = {next = 0x152a0c8, prev = 0x152a0c8},
aliases = {next = 0x152a0d8, prev = 0x152a0d8}, events_table = 0x0, sysfs_aliases = 0, loaded_json_aliases = 0, sysfs_aliases_loaded = false,
cpu_aliases_added = false, caps_initialized = false,
nr_caps = 0, caps = {next = 0x152a100, prev = 0x152a100},
list = {next = 0x0, prev = 0x0}, config_masks = {0, 0, 0, 0}, missing_features = {exclude_guest = false}}
(gdb) n
Program received signal SIGSEGV, Segmentation fault.
0x0000000001344558 in perf_pmu__find_events_table (pmu=0x152a090) at pmu-events/pmu-events.c:1970
1970 for (i = 0; i < table->num_pmus; i++) {
(gdb) p table
$5 = (const struct pmu_events_table *) 0x0
(gdb)
(gdb) where
#0 perf_pmu__find_events_table (pmu=pmu@entry=0x2aa003d48d0)
at pmu-events/pmu-events.c:1970
#1 0x000002aa00180cee in perf_pmu__lookup (pmus=0x2aa0039baf8 <other_pmus>,
dirfd=dirfd@entry=3, lookup_name=lookup_name@entry=0x2aa003dca83 "software")
at util/pmu.c:1014
#2 0x000002aa0018151e in perf_pmu__find2 (name=0x2aa003dca83 "software",
dirfd=<optimized out>) at util/pmus.c:150
#3 pmu_read_sysfs (core_only=false) at util/pmus.c:198
#4 0x000002aa00181e04 in pmu_read_sysfs (core_only=false) at util/pmus.c:179
#5 perf_pmus__find_by_type (type=0) at util/pmus.c:238
#6 perf_pmus__find_by_type (type=type@entry=0) at util/pmus.c:231
#7 0x000002aa0012ee4a in parse_events_add_numeric (parse_state=0x3ffffff7718,
parse_state@entry=0x100000000,
list=0x2aa003d47b0, list@entry=<error reading variable:
value has been optimized out>, type=type@entry=0,
config=<optimized out>, head_config=<optimized out>, head_config@entry=0x0, wildcard=true)
at util/parse-events.c:1347
#8 0x000002aa0017b7c2 in parse_events_parse (_parse_state=0x100000000,
_parse_state@entry=0x3ffffff7718, scanner=<optimized out>) at util/parse-events.y:418
#9 0x000002aa0012ca6e in parse_events__scanner (parse_state=0x3ffffff7718, input=0x0,
str=0x3ffffffa58a "cycles") at util/parse-events.c:1822
#10 __parse_events (evlist=0x3ffffff7830, str=str@entry=0x3ffffffa58a "cycles",
pmu_filter=<optimized out>, err=err@entry=0x3ffffff7830, fake_pmu=fake_pmu@entry=0x0,
warn_if_reordered=true) at util/parse-events.c:2094
....rest omitted.....Variable table is not assigned any value and remains a NULL pointer. This happens because on a z/VM vertual machines (on s390) there is not CPU measurement facility and the directories
/sys/devices/cpum_cf
/sys/devices/cpum_sf
do not exist. Therefore the event cycles as treated as software event and this default does work with the latest pmu-events rework.
Note: This core-dump will also happen on an LPAR when the CPU Measurement facility is not configured. Then above PMU directory do not exist on an LPAR.
The other PMU for s390 are
/sys/devices/pai_crypto /sys/devices/pai_ext
do exist on z/VM and LPAR on s390 (when configured).
Do you have an idea how to fix this?
Since the pmu-event subtree was completely redesigned, I would like some guidance on how to proceed.
Thanks a lot for your help.
Signature · 5 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Re: perf record dumps core all the time on s390 (6.6.0rc1) part2
Quoted 81 lines · Thomas Richter
Using Linux kernel version 6.6.0rc1, the command
# perf record -e cyclesdumps core immediately on s390 running on a z/VM virtual machine. It works on an s390 LPAR (same kernel).
This issue showed up during my vacation on 25-Aug-2023 in linux-next and since Monday 11-Sep-2023 it shows up in linux 6.6.0rc1.
The event cycles is treated as a software event and it fails in:
(gdb) n
1014pmu->events_table=perf_pmu__find_events_table(pmu);
(gdb) p *pmu
$4 = {name =0x152a150"software",alias_name=0x0, id =0x0, type = 1,
selectable = false,is_core= false,
is_uncore= false, auxtrace = false,formats_checked= false,config_masks_present= false,
config_masks_computed= false,max_precise= -1,default_config=0x0, cpus =0x0, format = {next =0x152a0c8, prev =0x152a0c8},
aliases = {next =0x152a0d8, prev =0x152a0d8},events_table=0x0,sysfs_aliases= 0,loaded_json_aliases= 0,sysfs_aliases_loaded= false,
cpu_aliases_added= false,caps_initialized= false,
nr_caps= 0, caps = {next =0x152a100, prev =0x152a100},
list = {next =0x0, prev =0x0},config_masks= {0, 0, 0, 0},missing_features= {exclude_guest= false}}
(gdb) nProgram received signal SIGSEGV, Segmentation fault.
0x0000000001344558inperf_pmu__find_events_table(pmu=0x152a090) atpmu-events/pmu-events.c:1970
1970 for (i = 0; i <table->num_pmus; i++) {
(gdb) p table
$5 = (const structpmu_events_table*)0x0
(gdb)(gdb) where
#0perf_pmu__find_events_table(pmu=pmu@entry=0x2aa003d48d0)at pmu-events/pmu-events.c:1970 #1 0x000002aa00180cee in perf_pmu__lookup (pmus=0x2aa0039baf8 <other_pmus>, dirfd=dirfd@entry=3, lookup_name=lookup_name@entry=0x2aa003dca83 "software") at util/pmu.c:1014 #2 0x000002aa0018151e in perf_pmu__find2 (name=0x2aa003dca83 "software", dirfd=<optimized out>) at util/pmus.c:150 #3 pmu_read_sysfs (core_only=false) at util/pmus.c:198 #4 0x000002aa00181e04 in pmu_read_sysfs (core_only=false) at util/pmus.c:179 #5 perf_pmus__find_by_type (type=0) at util/pmus.c:238 #6 perf_pmus__find_by_type (type=type@entry=0) at util/pmus.c:231 #7 0x000002aa0012ee4a in parse_events_add_numeric (parse_state=0x3ffffff7718, parse_state@entry=0x100000000, list=0x2aa003d47b0, list@entry=<error reading variable: value has been optimized out>, type=type@entry=0, config=<optimized out>, head_config=<optimized out>, head_config@entry=0x0, wildcard=true) at util/parse-events.c:1347 #8 0x000002aa0017b7c2 in parse_events_parse (_parse_state=0x100000000, _parse_state@entry=0x3ffffff7718, scanner=<optimized out>) at util/parse-events.y:418 #9 0x000002aa0012ca6e in parse_events__scanner (parse_state=0x3ffffff7718, input=0x0, str=0x3ffffffa58a "cycles") at util/parse-events.c:1822 #10 __parse_events (evlist=0x3ffffff7830, str=str@entry=0x3ffffffa58a "cycles", pmu_filter=<optimized out>, err=err@entry=0x3ffffff7830, fake_pmu=fake_pmu@entry=0x0, warn_if_reordered=true) at util/parse-events.c:2094 ....rest omitted.....Variable table is not assigned any value and remains a NULL pointer. This happens because on a z/VM vertual machines (on s390) there is not CPU measurement facility and the directories
/sys/devices/cpum_cf
/sys/devices/cpum_sf
do not exist. Therefore the event cycles as treated as software event and this default does work with the latest pmu-events rework.Note: This core-dump will also happen on an LPAR when the CPU Measurement facility is not configured. Then above PMU directory do not exist on an LPAR.
The other PMU for s390 are
/sys/devices/pai_crypto/sys/devices/pai_ext
do exist on z/VM and LPAR on s390 (when configured).
Quoted 6 lines
Do you have an idea how to fix this?
Since the pmu-event subtree was completely redesigned, I would like some guidance on how to proceed.Thanks a lot for your help.
When I use software events like cpu-clock or cs, the commands
# ./perf stat -e cs -- true Segmentation fault (core dumped) # ./perf stat -e cpu-clock-- true Segmentation fault (core dumped) #
also dump core. This should not happen as these events are defined even when no hardware PMU is available.
Debugging this reveals this call chain:
perf_pmus__find_by_type(type=1)
+--> pmu_read_sysfs(core_only=false)
+--> perf_pmu__find2(dirfd=3, name=0x152a113 "software")
+--> perf_pmu__lookup(pmus=0x14f0568 <other_pmus>, dirfd=3,
lookup_name=0x152a113 "software")
+--> perf_pmu__find_events_table (pmu=0x1532130)Now the pmu is "software" and it tries to find a proper table generated by the pmu-event generation process for s390:
# cd pmu-events/
# ./jevents.py s390 all /root/linux/tools/perf/pmu-events/arch |\
grep -E '^const struct pmu_table_entry'
const struct pmu_table_entry pmu_events__cf_z10[] = {
const struct pmu_table_entry pmu_events__cf_z13[] = {
const struct pmu_table_entry pmu_metrics__cf_z13[] = {
const struct pmu_table_entry pmu_events__cf_z14[] = {
const struct pmu_table_entry pmu_metrics__cf_z14[] = {
const struct pmu_table_entry pmu_events__cf_z15[] = {
const struct pmu_table_entry pmu_metrics__cf_z15[] = {
const struct pmu_table_entry pmu_events__cf_z16[] = {
const struct pmu_table_entry pmu_metrics__cf_z16[] = {
const struct pmu_table_entry pmu_events__cf_z196[] = {
const struct pmu_table_entry pmu_events__cf_zec12[] = {
const struct pmu_table_entry pmu_metrics__cf_zec12[] = {
const struct pmu_table_entry pmu_events__test_soc_cpu[] = {
const struct pmu_table_entry pmu_metrics__test_soc_cpu[] = {
const struct pmu_table_entry pmu_events__test_soc_sys[] = {
#However event "software" is not listed, as can be seen in the generated const struct pmu_events_map pmu_events_map[].
So in function perf_pmu__find_events_table(), there variable table is initialized to NULL, but never set to a proper value. The function scans all generated &pmu_events_map[] tables, but no table matches, because the tables are s390 CPU Measurement unit specific:
i = 0;
for (;;) {
const struct pmu_events_map *map = &pmu_events_map[i++];
if (!map->arch)
break;
--> the maps are there because the build generated them
if (!strcmp_cpuid_str(map->cpuid, cpuid)) {
table = &map->event_table;
break;
}
--> Since no matching CPU string the table var remains 0x0
}
free(cpuid);
if (!pmu)
return table;
--> The pmu is "software" so it exists and no return
for (i = 0; i < table->num_pmus; i++) {
const struct pmu_table_entry *table_pmu = &table->pmus[i];
--> and here perf dies because table is 0x0To me something is missing, either generate a pmu_events_map[] for software events or return and handle these events somewhere else.
The events cpu-clock or cs are completely software driven and do not need any CPU measurement hardware at all.
Something is very strange and I am confused.
Does this work on other platforms?
Thanks
Signature · 5 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294