perf test hybrid failing on 14700K
Arnaldo Carvalho de Melo
maintainerHi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo
model name : Intel(R) Core(TM) i7-14700K
root@number:/home/acme/Downloads# uname -a
Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux
root@number:/home/acme/Downloads# perf -vv
perf version 6.7.rc6.g63daba4e2861
dwarf: [ on ] # HAVE_DWARF_SUPPORT
dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT
syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT
libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT
debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
libelf: [ on ] # HAVE_LIBELF_SUPPORT
libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
libperl: [ on ] # HAVE_LIBPERL_SUPPORT
libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
libslang: [ on ] # HAVE_SLANG_SUPPORT
libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT
libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT
zlib: [ on ] # HAVE_ZLIB_SUPPORT
lzma: [ on ] # HAVE_LZMA_SUPPORT
get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORT
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
aio: [ on ] # HAVE_AIO_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
bpf_skeletons: [ on ] # HAVE_BPF_SKEL
root@number:/home/acme/Downloads# perf test 75
75: x86 hybrid : FAILED!
root@number:/home/acme/Downloads# perf test -v 75
75: x86 hybrid :
--- start ---
test child forked, pid 4111587
Using CPUID GenuineIntel-6-B7-1
running test 0 'cpu_core/cpu-cycles/'
FAILED arch/x86/tests/hybrid.c:30 wrong type
Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'
FAILED arch/x86/tests/hybrid.c:42 wrong type
Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}'
FAILED arch/x86/tests/hybrid.c:65 wrong type
Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}'
FAILED arch/x86/tests/hybrid.c:78 wrong type
Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'
FAILED arch/x86/tests/hybrid.c:95 wrong type
Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a'
running test 6 'cpu_core/r1a/'
running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u'
WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)!
running test 8 'cpu_core/LLC-loads/'
test child finished with -1
---- end ----
x86 hybrid: FAILED!
root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75
75: x86 hybrid :
--- start ---
Using CPUID GenuineIntel-6-B7-1
running test 0 'cpu_core/cpu-cycles/'
FAILED arch/x86/tests/hybrid.c:30 wrong type
Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'
FAILED arch/x86/tests/hybrid.c:42 wrong type
Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}'
FAILED arch/x86/tests/hybrid.c:65 wrong type
Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}'
FAILED arch/x86/tests/hybrid.c:78 wrong type
Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'
FAILED arch/x86/tests/hybrid.c:95 wrong type
Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a'
running test 6 'cpu_core/r1a/'
running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u'
WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)!
running test 8 'cpu_core/LLC-loads/'
---- end ----
x86 hybrid: FAILED!
0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3
0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4
root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75
75: x86 hybrid :
--- start ---
Using CPUID GenuineIntel-6-B7-1
running test 0 'cpu_core/cpu-cycles/'
FAILED arch/x86/tests/hybrid.c:30 wrong type
Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'
FAILED arch/x86/tests/hybrid.c:42 wrong type
Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}'
FAILED arch/x86/tests/hybrid.c:65 wrong type
Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}'
FAILED arch/x86/tests/hybrid.c:78 wrong type
Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'
FAILED arch/x86/tests/hybrid.c:95 wrong type
Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a'
running test 6 'cpu_core/r1a/'
running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u'
WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)!
running test 8 'cpu_core/LLC-loads/'
perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3
perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4
---- end ----
x86 hybrid: FAILED!
+++ exited with 0 +++
root@number:/home/acme/Downloads#Quoted 5 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Quoted 26 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORT
Quoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
aio: [ on ] # HAVE_AIO_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
bpf_skeletons: [ on ] # HAVE_BPF_SKEL
root@number:/home/acme/Downloads# perf test 75
75: x86 hybrid : FAILED!The failure should be a regression caused by the a24d9d9dc096 ("perf
parse-events: Make legacy events lower priority than sysfs/JSON")
@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr,
err_str, /*help=*/NULL);
return -EINVAL;
}
- attr->type = PERF_TYPE_HARDWARE;
- attr->config = term->val.num;
- if (perf_pmus__supports_extended_type())
- attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT;
+ /*
+ * If the PMU has a sysfs or json event prefer it over
+ * legacy. ARM requires this.
+ */
+ if (perf_pmu__have_event(pmu, term->config))
For Intel hybrid, the perf_pmu__have_event() should be always true for all hw events and hw cache events. So the patch will mistakenly update the type to TYPE_USER. However, On Intel platforms, the type of the hw events should be TYPE_HARDWARE.
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER;
+ term->no_value = true;
+ } else {
+ attr->type = PERF_TYPE_HARDWARE;
+ attr->config = term->val.num;
+ if (perf_pmus__supports_extended_type())
+ attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT;
+ }
return 0;
}
if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||
Thanks,
Kan
Quoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Ian Rogers
reviewerQuoted 69 lines · Kan
Quoted 5 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Quoted 26 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORTQuoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT aio: [ on ] # HAVE_AIO_SUPPORT zstd: [ on ] # HAVE_ZSTD_SUPPORT libpfm4: [ on ] # HAVE_LIBPFM libtraceevent: [ on ] # HAVE_LIBTRACEEVENT bpf_skeletons: [ on ] # HAVE_BPF_SKEL root@number:/home/acme/Downloads# perf test 75 75: x86 hybrid : FAILED!The failure should be a regression caused by the
a24d9d9dc096("perf
parse-events: Make legacy events lower priority than sysfs/JSON")(quoted diff) +5
-4@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr, err_str, /*help=*/NULL); return -EINVAL; } - attr->type = PERF_TYPE_HARDWARE; - attr->config = term->val.num; - if (perf_pmus__supports_extended_type()) - attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + /* + * If the PMU has a sysfs or json event prefer it over + * legacy. ARM requires this. + */ + if (perf_pmu__have_event(pmu, term->config))For Intel hybrid, the
perf_pmu__have_event()should be always true for all hw events and hw cache events. So the patch will mistakenly update the type toTYPE_USER. However, On Intel platforms, the type of the hw events should beTYPE_HARDWARE.
Quoted 3 lines
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
Hi Kan/Mark,
Firstly, the perf_pmu__have_event is a test of whether /sys/devices/<pmu>/events or the equivalent json events have the specified event string like "inst_retired.any" - ie it isn't a test of whether the event is supported by the kernel. Mark was quite insistent that the behavior be changed so that if a legacy event is specified with a PMU, the PMU's sysfs/json event is the priority which is a big behavior change on x86.
To get the test passing again I've sent out a test update:
https://lore.kernel.org/lkml/[email protected]/
This switches the legacy events in the test to ones which don't have sysfs or json equivalents. The test is somewhat brittle as an x86 PMU change could decide to add /sys/devices/cpu/events/cycles alongside /sys/devices/cpu/events/cpu-cycles. Ideally we'd be testing all events on core PMUs, for legacy events they may have a sysfs/json override and the test expectation should expect this and assert that the type isn't PERF_TYPE_HARDWARE, etc. For now what I sent out is sufficient to get the "x86 hybrid" test passing. I should probably have done the whole Reported-by.. thing, sorry for missing that.
Thanks,
Ian
Quoted 90 lines
(quoted diff) +8
-0+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER; + term->no_value = true; + } else { + attr->type = PERF_TYPE_HARDWARE; + attr->config = term->val.num; + if (perf_pmus__supports_extended_type()) + attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + } return 0; } if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||Thanks,
KanQuoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Quoted 84 lines · Ian Rogers
Quoted 69 lines · Kan
Quoted 5 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Quoted 26 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORTQuoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT aio: [ on ] # HAVE_AIO_SUPPORT zstd: [ on ] # HAVE_ZSTD_SUPPORT libpfm4: [ on ] # HAVE_LIBPFM libtraceevent: [ on ] # HAVE_LIBTRACEEVENT bpf_skeletons: [ on ] # HAVE_BPF_SKEL root@number:/home/acme/Downloads# perf test 75 75: x86 hybrid : FAILED!The failure should be a regression caused by the
a24d9d9dc096("perf
parse-events: Make legacy events lower priority than sysfs/JSON")(quoted diff) +5
-4@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr, err_str, /*help=*/NULL); return -EINVAL; } - attr->type = PERF_TYPE_HARDWARE; - attr->config = term->val.num; - if (perf_pmus__supports_extended_type()) - attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + /* + * If the PMU has a sysfs or json event prefer it over + * legacy. ARM requires this. + */ + if (perf_pmu__have_event(pmu, term->config))For Intel hybrid, the
perf_pmu__have_event()should be always true for all hw events and hw cache events. So the patch will mistakenly update the type toTYPE_USER. However, On Intel platforms, the type of the hw events should beTYPE_HARDWARE.Quoted 3 lines
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
Hi Kan/Mark,
Firstly, the
perf_pmu__have_eventis a test of whether /sys/devices/<pmu>/events or the equivalent json events have the specified event string like "inst_retired.any" - ie it isn't a test of whether the event is supported by the kernel. Mark was quite insistent that the behavior be changed so that if a legacy event is specified with a PMU, the PMU's sysfs/json event is the priority which is a big behavior change on x86.
Quoted 6 lines
To get the test passing again I've sent out a test update:
https://lore.kernel.org/lkml/[email protected]/
This switches the legacy events in the test to ones which don't have sysfs or json equivalents. The test is somewhat brittle as an x86 PMU change could decide to add/sys/devices/cpu/events/cyclesalongside/sys/devices/cpu/events/cpu-cycles.
I think the cycles, cpu-cycles, and instructions are treated as architectural events in both Linux and X86. I don't think there is a plan to change the behavior of the architectural events.
Quoted 5 lines
Ideally we'd be testing all events
on core PMUs, for legacy events they may have a sysfs/json override and the test expectation should expect this and assert that the type isn'tPERF_TYPE_HARDWARE, etc. For now what I sent out is sufficient to get the "x86 hybrid" test passing.
Before the a24d9d9dc096, the "cpu-cycles" and the "cycles" are identical. It should be fine to verify either of them.
But after the a24d9d9dc096, the code path is different. So maybe it's better to verify both of the "cpu-cycles" and the "cycles" in the x86 hybrid test. Adding a new case something as below should be enough.
{
.name = "{cpu_core/cpu-cycles/,cpu_core/cycles/}",
.check = test__hybrid_hw_group_2_event,
},Thanks,
Kan
Quoted 97 lines
I should probably have done the
whole Reported-by.. thing, sorry for missing that.Thanks,
IanQuoted 90 lines
(quoted diff) +8
-0+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER; + term->no_value = true; + } else { + attr->type = PERF_TYPE_HARDWARE; + attr->config = term->val.num; + if (perf_pmus__supports_extended_type()) + attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + } return 0; } if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||Thanks,
KanQuoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Arnaldo Carvalho de Melo
maintainerEm Tue, Jan 02, 2024 at 02:41:07PM -0800, Ian Rogers escreveu:
Quoted 84 lines · Kan
Quoted 69 lines · Arnaldo Carvalho de Melo
Quoted 5 lines
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Quoted 26 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORTQuoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT aio: [ on ] # HAVE_AIO_SUPPORT zstd: [ on ] # HAVE_ZSTD_SUPPORT libpfm4: [ on ] # HAVE_LIBPFM libtraceevent: [ on ] # HAVE_LIBTRACEEVENT bpf_skeletons: [ on ] # HAVE_BPF_SKEL root@number:/home/acme/Downloads# perf test 75 75: x86 hybrid : FAILED!The failure should be a regression caused by the
a24d9d9dc096("perf
parse-events: Make legacy events lower priority than sysfs/JSON")(quoted diff) +5
-4@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr, err_str, /*help=*/NULL); return -EINVAL; } - attr->type = PERF_TYPE_HARDWARE; - attr->config = term->val.num; - if (perf_pmus__supports_extended_type()) - attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + /* + * If the PMU has a sysfs or json event prefer it over + * legacy. ARM requires this. + */ + if (perf_pmu__have_event(pmu, term->config))For Intel hybrid, the
perf_pmu__have_event()should be always true for all hw events and hw cache events. So the patch will mistakenly update the type toTYPE_USER. However, On Intel platforms, the type of the hw events should beTYPE_HARDWARE.Quoted 3 lines
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
Hi Kan/Mark,
Firstly, the
perf_pmu__have_eventis a test of whether /sys/devices/<pmu>/events or the equivalent json events have the specified event string like "inst_retired.any" - ie it isn't a test of whether the event is supported by the kernel. Mark was quite insistent that the behavior be changed so that if a legacy event is specified with a PMU, the PMU's sysfs/json event is the priority which is a big behavior change on x86.
Quoted 11 lines
To get the test passing again I've sent out a test update:
https://lore.kernel.org/lkml/[email protected]/
This switches the legacy events in the test to ones which don't have sysfs or json equivalents. The test is somewhat brittle as an x86 PMU change could decide to add/sys/devices/cpu/events/cyclesalongside/sys/devices/cpu/events/cpu-cycles. Ideally we'd be testing all events on core PMUs, for legacy events they may have a sysfs/json override and the test expectation should expect this and assert that the type isn'tPERF_TYPE_HARDWARE, etc. For now what I sent out is sufficient to get the "x86 hybrid" test passing. I should probably have done the whole Reported-by.. thing, sorry for missing that.
I can fix that once we're fine with that fix.
- Arnaldo
Quoted 93 lines
Thanks,
IanQuoted 90 lines
(quoted diff) +8
-0+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER; + term->no_value = true; + } else { + attr->type = PERF_TYPE_HARDWARE; + attr->config = term->val.num; + if (perf_pmus__supports_extended_type()) + attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + } return 0; } if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||Thanks,
KanQuoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Signature · 1 line
- Arnaldo
Ian Rogers
reviewerQuoted 105 lines · Kan
Quoted 84 lines · Ian Rogers
Quoted 69 lines · Kan
Quoted 5 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Quoted 26 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORTQuoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT aio: [ on ] # HAVE_AIO_SUPPORT zstd: [ on ] # HAVE_ZSTD_SUPPORT libpfm4: [ on ] # HAVE_LIBPFM libtraceevent: [ on ] # HAVE_LIBTRACEEVENT bpf_skeletons: [ on ] # HAVE_BPF_SKEL root@number:/home/acme/Downloads# perf test 75 75: x86 hybrid : FAILED!The failure should be a regression caused by the
a24d9d9dc096("perf
parse-events: Make legacy events lower priority than sysfs/JSON")(quoted diff) +5
-4@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr, err_str, /*help=*/NULL); return -EINVAL; } - attr->type = PERF_TYPE_HARDWARE; - attr->config = term->val.num; - if (perf_pmus__supports_extended_type()) - attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + /* + * If the PMU has a sysfs or json event prefer it over + * legacy. ARM requires this. + */ + if (perf_pmu__have_event(pmu, term->config))For Intel hybrid, the
perf_pmu__have_event()should be always true for all hw events and hw cache events. So the patch will mistakenly update the type toTYPE_USER. However, On Intel platforms, the type of the hw events should beTYPE_HARDWARE.Quoted 3 lines
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
Hi Kan/Mark,
Firstly, the
perf_pmu__have_eventis a test of whether /sys/devices/<pmu>/events or the equivalent json events have the specified event string like "inst_retired.any" - ie it isn't a test of whether the event is supported by the kernel. Mark was quite insistent that the behavior be changed so that if a legacy event is specified with a PMU, the PMU's sysfs/json event is the priority which is a big behavior change on x86.Quoted 6 lines
To get the test passing again I've sent out a test update:
https://lore.kernel.org/lkml/[email protected]/
This switches the legacy events in the test to ones which don't have sysfs or json equivalents. The test is somewhat brittle as an x86 PMU change could decide to add/sys/devices/cpu/events/cyclesalongside/sys/devices/cpu/events/cpu-cycles.I think the cycles, cpu-cycles, and instructions are treated as architectural events in both Linux and X86. I don't think there is a plan to change the behavior of the architectural events.
Quoted 5 lines
Ideally we'd be testing all events
on core PMUs, for legacy events they may have a sysfs/json override and the test expectation should expect this and assert that the type isn'tPERF_TYPE_HARDWARE, etc. For now what I sent out is sufficient to get the "x86 hybrid" test passing.
Quoted 4 lines
Before the
a24d9d9dc096, the "cpu-cycles" and the "cycles" are identical. It should be fine to verify either of them.But after the
a24d9d9dc096, the code path is different.
It isn't so much that the code path is different, after a24d9d9dc096 we prefer events defined in sysfs/json over legacy events, as that exists for "cpu-cycles" then we get the test failures. Using "cycles" is a fix purely because /sys/devices/cpu_core/events/cycles doesn't exist, but this is somewhat arbitrary.
Quoted 11 lines
So maybe it's
better to verify both of the "cpu-cycles" and the "cycles" in the x86 hybrid test. Adding a new case something as below should be enough.{ .name = "{cpu_core/cpu-cycles/,cpu_core/cycles/}", .check = test__hybrid_hw_group_2_event, },Thanks,
Kan
More tests is a good thing, I'll add that in v2 along with the Reported-by.
Thanks,
Ian
Quoted 97 lines
Quoted 97 lines
I should probably have done the
whole Reported-by.. thing, sorry for missing that.Thanks,
IanQuoted 90 lines
(quoted diff) +8
-0+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER; + term->no_value = true; + } else { + attr->type = PERF_TYPE_HARDWARE; + attr->config = term->val.num; + if (perf_pmus__supports_extended_type()) + attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + } return 0; } if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||Thanks,
KanQuoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Mark Rutland
reviewerQuoted 9 lines · Ian Rogers
Quoted 8 lines · Kan
Quoted 5 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Godt nytår!
Quoted 62 lines
Quoted 56 lines
Quoted 25 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORTQuoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT aio: [ on ] # HAVE_AIO_SUPPORT zstd: [ on ] # HAVE_ZSTD_SUPPORT libpfm4: [ on ] # HAVE_LIBPFM libtraceevent: [ on ] # HAVE_LIBTRACEEVENT bpf_skeletons: [ on ] # HAVE_BPF_SKEL root@number:/home/acme/Downloads# perf test 75 75: x86 hybrid : FAILED!The failure should be a regression caused by the
a24d9d9dc096("perf
parse-events: Make legacy events lower priority than sysfs/JSON")(quoted diff) +5
-4@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr, err_str, /*help=*/NULL); return -EINVAL; } - attr->type = PERF_TYPE_HARDWARE; - attr->config = term->val.num; - if (perf_pmus__supports_extended_type()) - attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + /* + * If the PMU has a sysfs or json event prefer it over + * legacy. ARM requires this. + */ + if (perf_pmu__have_event(pmu, term->config))For Intel hybrid, the
perf_pmu__have_event()should be always true for all hw events and hw cache events. So the patch will mistakenly update the type toTYPE_USER. However, On Intel platforms, the type of the hw events should beTYPE_HARDWARE.Quoted 3 lines
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
Hi Kan/Mark,
Quoted 7 lines
Firstly, the
perf_pmu__have_eventis a test of whether /sys/devices/<pmu>/events or the equivalent json events have the specified event string like "inst_retired.any" - ie it isn't a test of whether the event is supported by the kernel. Mark was quite insistent that the behavior be changed so that if a legacy event is specified with a PMU, the PMU's sysfs/json event is the priority which is a big behavior change on x86.
For the record, I was insistent that the behaviour was *restored*; that was the existing behaivour prior to commit:
5ea8f2ccffb23983 ("perf parse-events: Support hardware events as terms")
... which was itself a big behaviour change for all architectures, in part led to the issue that Hector and Martin were hitting on arm/arm64, and provided no recourse to get the prior behaviour when desired.
I appreciate that the change I requested was a big behaviour change on x86 relative to v6.5, but the change above in v6.5 was a big behaviour change on arm/arm64 relative to the behaviour established many years prior. I'm sorry that I did not catch this earlier at the review stage.
Quoted 2 lines
To get the test passing again I've sent out a test update:
https://lore.kernel.org/lkml/[email protected]/
FWIW that looks good to me; I've given my Ack there.
Thanks,
Mark.
Quoted 103 lines
This switches the legacy events in the test to ones which don't have sysfs or json equivalents. The test is somewhat brittle as an x86 PMU change could decide to add
/sys/devices/cpu/events/cyclesalongside/sys/devices/cpu/events/cpu-cycles. Ideally we'd be testing all events on core PMUs, for legacy events they may have a sysfs/json override and the test expectation should expect this and assert that the type isn'tPERF_TYPE_HARDWARE, etc. For now what I sent out is sufficient to get the "x86 hybrid" test passing. I should probably have done the whole Reported-by.. thing, sorry for missing that.Thanks,
IanQuoted 90 lines
(quoted diff) +8
-0+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER; + term->no_value = true; + } else { + attr->type = PERF_TYPE_HARDWARE; + attr->config = term->val.num; + if (perf_pmus__supports_extended_type()) + attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + } return 0; } if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||Thanks,
KanQuoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Ian Rogers
reviewerQuoted 84 lines · Mark Rutland
Quoted 9 lines · Ian Rogers
Quoted 8 lines · Kan
Quoted 5 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
Happy new year!
Godt nytår!
Quoted 62 lines
Quoted 56 lines
Quoted 25 lines
- Arnaldo
root@number:/home/acme/Downloads# grep "model name" -m1 /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:/home/acme/Downloads# uname -a Linux number 6.6.4-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 3 18:13:11 UTC 2023 x86_64 GNU/Linux root@number:/home/acme/Downloads# perf -vv perf version 6.7.rc6.g63daba4e2861 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT zlib: [ on ] # HAVE_ZLIB_SUPPORT lzma: [ on ] # HAVE_LZMA_SUPPORT get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORTQuoted 8 lines
bpf: [ on ] # HAVE_LIBBPF_SUPPORT aio: [ on ] # HAVE_AIO_SUPPORT zstd: [ on ] # HAVE_ZSTD_SUPPORT libpfm4: [ on ] # HAVE_LIBPFM libtraceevent: [ on ] # HAVE_LIBTRACEEVENT bpf_skeletons: [ on ] # HAVE_BPF_SKEL root@number:/home/acme/Downloads# perf test 75 75: x86 hybrid : FAILED!The failure should be a regression caused by the
a24d9d9dc096("perf
parse-events: Make legacy events lower priority than sysfs/JSON")(quoted diff) +5
-4@@ -1004,10 +1012,19 @@ static int config_term_pmu(struct perf_event_attr *attr, err_str, /*help=*/NULL); return -EINVAL; } - attr->type = PERF_TYPE_HARDWARE; - attr->config = term->val.num; - if (perf_pmus__supports_extended_type()) - attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + /* + * If the PMU has a sysfs or json event prefer it over + * legacy. ARM requires this. + */ + if (perf_pmu__have_event(pmu, term->config))For Intel hybrid, the
perf_pmu__have_event()should be always true for all hw events and hw cache events. So the patch will mistakenly update the type toTYPE_USER. However, On Intel platforms, the type of the hw events should beTYPE_HARDWARE.Quoted 3 lines
Seems ARM needs to find another way to distinguish the case.
Ian, any suggestions?
Hi Kan/Mark,
Quoted 7 lines
Firstly, the
perf_pmu__have_eventis a test of whether /sys/devices/<pmu>/events or the equivalent json events have the specified event string like "inst_retired.any" - ie it isn't a test of whether the event is supported by the kernel. Mark was quite insistent that the behavior be changed so that if a legacy event is specified with a PMU, the PMU's sysfs/json event is the priority which is a big behavior change on x86.
Quoted 8 lines
For the record, I was insistent that the behaviour was *restored*; that was the existing behaivour prior to commit:
5ea8f2ccffb23983("perf parse-events: Support hardware events as terms")... which was itself a big behaviour change for all architectures, in part led to the issue that Hector and Martin were hitting on arm/arm64, and provided no recourse to get the prior behaviour when desired.
Fwiw, I don't agree with the description here. The original issue was that on ARM perf's behavior had become reliant on the core PMUs being treated as non-core PMUs in that non-core PMUs don't support things like legacy events. This is/was unsustainable and once fixed exposed a second bug where ARM's PMU driver didn't support legacy events correctly, breaking Hector and Martin. I'm not aware of this issue being fixed in the driver although the failure has been made clear by this episode. Fixing the PMU driver wouldn't have fixed older kernels so there is some sense in fixing the tool too.
In the perf tool we've worked around all the issues brought to us by the ARM PMU by inverting the priority of legacy and sysfs events. Once again an issue here and Kan was suggesting inverting the sense for Intel. I'm working hard to make all PMUs on all architectures work the same and the priority of legacy and sysfs/JSON events during parsing is some what arbitrary.
It was argued about PMU namespaces in event parsing, no such concept exists in perf's event parsing code so for the record I've never bought this. It seemed an excuse to try push ARM's PMU driver failures onto the perf tool. I can elaborate but don't see the point in furthering the argument.
It was requested that ARM provide patches to improve testing coverage for their PMU in perf test, specifically tools/perf/tests/parse-events.c where relevant tests are skipped as the PMU name is hard coded to "cpu" which is conventional for a core PMU, but not on ARM. Were this testing covering big.LITTLE/hybrid these issues should have come to light earlier.
While much has been done in the perf tool to work around latent issues with ARM's PMU drivers and the behavior changes they have instigated, except for 5c816728651a ("arm_pmu: Add PERF_PMU_CAP_EXTENDED_HW_TYPE capability") where I wrote the original version, there doesn't seem to have been progress made on the ARM PMU driver nor on ARM testing - which was why such an issue could exist for so long, across numerous Linux releases and ultimately break Hector and Martin.
Fwiw, I have tickets in the UK next week for FurbyFest:
https://www.linkedin.com/posts/csmcr_furbyfest-a-festschrift-for-professor-steve-activity-7129811310799642624-YOOe/
if it is useful for me to talk to ARM people face-to-face or with a convenient timezone.
Thanks,
Ian
Quoted 116 lines
I appreciate that the change I requested was a big behaviour change on x86 relative to v6.5, but the change above in v6.5 was a big behaviour change on arm/arm64 relative to the behaviour established many years prior. I'm sorry that I did not catch this earlier at the review stage.
Quoted 2 lines
To get the test passing again I've sent out a test update:
https://lore.kernel.org/lkml/[email protected]/FWIW that looks good to me; I've given my Ack there.
Thanks,
Mark.Quoted 103 lines
This switches the legacy events in the test to ones which don't have sysfs or json equivalents. The test is somewhat brittle as an x86 PMU change could decide to add
/sys/devices/cpu/events/cyclesalongside/sys/devices/cpu/events/cpu-cycles. Ideally we'd be testing all events on core PMUs, for legacy events they may have a sysfs/json override and the test expectation should expect this and assert that the type isn'tPERF_TYPE_HARDWARE, etc. For now what I sent out is sufficient to get the "x86 hybrid" test passing. I should probably have done the whole Reported-by.. thing, sorry for missing that.Thanks,
IanQuoted 90 lines
(quoted diff) +8
-0+ term->type_term = PARSE_EVENTS__TERM_TYPE_USER; + term->no_value = true; + } else { + attr->type = PERF_TYPE_HARDWARE; + attr->config = term->val.num; + if (perf_pmus__supports_extended_type()) + attr->config |= (__u64)pmu->type << PERF_PMU_TYPE_SHIFT; + } return 0; } if (term->type_term == PARSE_EVENTS__TERM_TYPE_USER ||Thanks,
KanQuoted 74 lines
root@number:/home/acme/Downloads# perf test -v 75 75: x86 hybrid : --- start --- test child forked, pid 4111587 Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' test child finished with -1 ---- end ---- x86 hybrid: FAILED! root@number:/home/acme/Downloads#root@number:/home/acme/Downloads# perf trace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' ---- end ---- x86 hybrid: FAILED! 0.000 ( 0.008 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0xa00000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925535355, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 3 0.010 ( 0.002 ms): :4115165/4115165 perf_event_open(attr_uptr: { type: 0 (PERF_TYPE_HARDWARE), config: 0x400000000, disabled: 1, { bp_len, config2 }: 0x900000000, branch_sample_type: USER|COUNTERS, sample_regs_user: 0x3ecaddffffffff, sample_stack_user: 4115165, clockid: 925538919, sample_regs_intr: 0x8140c90000a8f7, sample_max_stack: 8, sig_data: 120259084288 }, cpu: -1, group_fd: -1, flags: FD_CLOEXEC) = 4 root@number:/home/acme/Downloads# strace -e perf_event_open perf test -F -v 75 75: x86 hybrid : --- start --- Using CPUID GenuineIntel-6-B7-1 running test 0 'cpu_core/cpu-cycles/' FAILED arch/x86/tests/hybrid.c:30 wrong type Event test failure: test 0 'cpu_core/cpu-cycles/'running test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}' FAILED arch/x86/tests/hybrid.c:42 wrong type Event test failure: test 1 '{cpu_core/cpu-cycles/,cpu_core/instructions/}'running test 2 '{cpu-clock,cpu_core/cpu-cycles/}' FAILED arch/x86/tests/hybrid.c:65 wrong type Event test failure: test 2 '{cpu-clock,cpu_core/cpu-cycles/}'running test 3 '{cpu_core/cpu-cycles/,cpu-clock}' FAILED arch/x86/tests/hybrid.c:78 wrong type Event test failure: test 3 '{cpu_core/cpu-cycles/,cpu-clock}'running test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}' FAILED arch/x86/tests/hybrid.c:95 wrong type Event test failure: test 4 '{cpu_core/cpu-cycles/k,cpu_core/instructions/u}'running test 5 'r1a' running test 6 'cpu_core/r1a/' running test 7 'cpu_core/config=10,config1,config2=3,period=1000/u' WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)! running test 8 'cpu_core/LLC-loads/' perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0xa<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 3 perf_event_open({type=PERF_TYPE_HARDWARE, size=0 /* PERF_ATTR_SIZE_??? */, config=0x4<<32|PERF_COUNT_HW_CPU_CYCLES, sample_period=0, sample_type=0, read_format=0, disabled=1, precise_ip=0 /* arbitrary skid */, ...}, 0, -1, -1, PERF_FLAG_FD_CLOEXEC) = 4 ---- end ---- x86 hybrid: FAILED! +++ exited with 0 +++ root@number:/home/acme/Downloads#
Mark Rutland
reviewerQuoted 7 lines · Ian Rogers
While much has been done in the perf tool to work around latent issues with ARM's PMU drivers and the behavior changes they have instigated, except for
5c816728651a("arm_pmu: AddPERF_PMU_CAP_EXTENDED_HW_TYPEcapability") where I wrote the original version, there doesn't seem to have been progress made on the ARM PMU driver nor on ARM testing - which was why such an issue could exist for so long, across numerous Linux releases and ultimately break Hector and Martin.
Ian, which latent issues in the PMU drivers are you referring to?
I already pointed out that one of the problems you have claimed to be a driver bug is actually due to userspace incorrectly detecting support for the extended HW type, and I suggested some options which you have so far ignored:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
I agree that testing is a problem, and we need to do better from the arm side.
Thanks,
Mark.
Ian Rogers
reviewerQuoted 11 lines · Mark Rutland
Quoted 7 lines · Ian Rogers
While much has been done in the perf tool to work around latent issues with ARM's PMU drivers and the behavior changes they have instigated, except for
5c816728651a("arm_pmu: AddPERF_PMU_CAP_EXTENDED_HW_TYPEcapability") where I wrote the original version, there doesn't seem to have been progress made on the ARM PMU driver nor on ARM testing - which was why such an issue could exist for so long, across numerous Linux releases and ultimately break Hector and Martin.Ian, which latent issues in the PMU drivers are you referring to?
Core PMUs generally have a lookup from legacy to non-legacy event, from the past discussion this was missing on the problematic PMUs hence the sysfs versions.
Quoted 6 lines
I already pointed out that one of the problems you have claimed to be a driver bug is actually due to userspace incorrectly detecting support for the extended HW type, and I suggested some options which you have so far ignored:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
Those suggestions look fine but you seemed to be suggesting testing. Lacking hardware with the broken PMU(s) I'm not able to do this. Could you turn the suggestion into a patch and test on BIG.little? I can test on Intel Alderlake.
Thanks,
Ian
Quoted 4 lines
I agree that testing is a problem, and we need to do better from the arm side.
Thanks,
Mark.
Mark Rutland
reviewerQuoted 11 lines · Ian Rogers
Quoted 10 lines · Mark Rutland
Quoted 7 lines · Ian Rogers
While much has been done in the perf tool to work around latent issues with ARM's PMU drivers and the behavior changes they have instigated, except for
5c816728651a("arm_pmu: AddPERF_PMU_CAP_EXTENDED_HW_TYPEcapability") where I wrote the original version, there doesn't seem to have been progress made on the ARM PMU driver nor on ARM testing - which was why such an issue could exist for so long, across numerous Linux releases and ultimately break Hector and Martin.
Quoted 5 lines
Quoted 1 line
Ian, which latent issues in the PMU drivers are you referring to?
Core PMUs generally have a lookup from legacy to non-legacy event, from the past discussion this was missing on the problematic PMUs hence the sysfs versions.
I don't know what you mean there, so I think you may have the wrong end of the stick.
All the ARM PMU drivers handle revelant PERF_EVENT_TYPE_HARDWARE events directly; they check perf_event_attr::type, and interpret perf_event_attr::config dependent upon this.
For example, the Apple M1 PMU driver has a table mapping PERF_COUNT_HW_CPU_CYCLES and PERF_COUNT_HW_INSTRUCTIONS to its native events:
... which is used by the m1_pmu_map_event() and m2_pmu_map_event() callbacks:
... which is used by __hw_perf_event_init() when called by armpmu_event_init, which is used as the pmu::event_init() callback:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmu.c?h=v6.7#n443
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/perf/arm_pmu.c?h=v6.7#n500
The sysfs events were added so that PMU-specific events (in the namespace of the PMU's dynamic type id) could be advertised to userspace. Those weren't added in lieu of the PERF_EVENT_TYPE_HARDWARE events.
When support for big.LITTLE was added (years prior to the introduction of the extended HW type support), it was necessary to use those sysfs events to target events to specific PMUs (since those used the PMU's dynamic type id, and would only be handled by that specific PMU). Hence why Marc and Hector were using those events on their systems.
AFAICT there's nothing missing from the drivers w.r.t. event mapping.
The problem Hector and Marc encountered was the result of a number of interacting parts:
1) The logic in the perf tool to detect support for extended type support was not robust to PMUs requiring event filters and failed to detect support for the extended type for the Apple PMUs which require the exclude_guest filter.
2) The interpretation of 'pmu/event/' was changed to try to interpret 'event' as a PERF_EVENT_TYPE_HARDWARE event. For the Apple PMU, this changed the interpretation of 'apple_pmu_name/cycles/' and 'apple_pmu_name/instructions/'.
3) Even in the absence of (detected) support for extended type support, the perf tool interpreted 'apple_pmu_name/cycles/' and 'apple_pmu_name/instructions/' as PERF_EVENT_TYPE_HARDWARE events, but since it thought that the kernel didn't support the extended HW type, it didn't set the extended bits to target specific PMUs. Hence the kernel didn't know which PMUs to route the events to specifically, and those were handled by an arbitrary PMU which could handle PERF_EVENT_TYPE_HARDWARE events.
... hence the issue that Hector and Marc reported with events going to a different PMU from what they explicitly requested.
As I've said before, I think that (2) is a problem in its own right, since even with (1) fixed, clashing event names could have different semantics, and it would be better to have a distinct syntax for "PERF_EVENT_TYPE_HARDWARE event on this specific PMU" vs "native event on this PMU".
Quoted 8 lines
Quoted 6 lines
I already pointed out that one of the problems you have claimed to be a driver bug is actually due to userspace incorrectly detecting support for the extended HW type, and I suggested some options which you have so far ignored:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/Those suggestions look fine but you seemed to be suggesting testing.
I was only asking for an opinion on the analysis/suggestion, no testing necessary.
Quoted 3 lines
Lacking hardware with the broken PMU(s) I'm not able to do this. Could you turn the suggestion into a patch and test on BIG.little? I can test on Intel Alderlake.
I've spun up the diff below, which works in local testing. I'll go write up a commit message and send it out as a proper patch shortly.
Mark.
---->8----
tools/perf/util/print-events.c +19 -8
@@ -232,7 +232,6 @@ void print_sdt_events(const struct print_callbacks *print_cb, void *print_state)bool is_event_supported (u8 type,u64 config) {bool ret = true; -int open_return;struct evsel *evsel;struct perf_event_attr attr = { .type = type,
@@ -246,20 +245,32 @@ bool is_event_supported(u8 type, u64 config) evsel =evsel__new (&attr);if (evsel) { - open_return =evsel__open (evsel,NULL , tmap); - ret = open_return >=0 ; + ret =evsel__open (evsel,NULL , tmap) >=0 ; -if (open_return == -EACCES ) { +if (!ret) {/* -* This happens if the paranoid value +* The event may fail to open if the paranoid value * /proc/sys/kernel/perf_event_paranoid is set to 2 -* Re-run with exclude_kernel set; we don't do that -* by default as some ARM machines do not support it. -* +* Re-run with exclude_kernel set; we don't do that by +* default as some ARM machines do not support it. */ evsel->core.attr.exclude_kernel =1 ; ret =evsel__open (evsel,NULL , tmap) >=0 ; } + +if (!ret) { +/* +* The event may fail to open if the PMU requires +* exclude_guest to be set (e.g. as the Apple M1 PMU +* requires). +* Re-run with exclude_guest set; we don't do that by +* default as it's equally legitimate for another PMU +* driver to require that exclude_guest is clear. +*/ + evsel->core.attr.exclude_guest =1 ; + ret =evsel__open (evsel,NULL , tmap) >=0 ; + } +evsel__delete (evsel); }
Mark Rutland
reviewerQuoted 5 lines · Mark Rutland
Quoted 3 lines · Ian Rogers
Lacking hardware with the broken PMU(s) I'm not able to do this. Could you turn the suggestion into a patch and test on BIG.little? I can test on Intel Alderlake.
For the sake of the archive; I sent that out as:
https://lore.kernel.org/lkml/[email protected]/
I've given that some testing on big.LITTLE: I've asked Hector and Marc to give it a go with the Apple M1/M2 PMU specifically.
Mark.
Ian Rogers
reviewerQuoted 8 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
- Arnaldo
Hi Arnaldo,
I'm seeing a test failure on Alderlake in perf-tools-next and wondered if it was on your radar:
``` 32: Session topology : --- start --- test child forked, pid 539222 templ file: /tmp/perf-test-HMet21 Using CPUID GenuineIntel-6-97-2 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) config 0x800000000 disabled 1 ------------------------------------------------------------ sys_perf_event_open: pid 0 cpu -1 group_fd -1 flags 0x8 = 4 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) config 0x400000000 disabled 1 ------------------------------------------------------------ sys_perf_event_open: pid 0 cpu -1 group_fd -1 flags 0x8 = 5 non matching sample_type FAILED tests/topology.c:73 can't get session test child finished with -1 ---- end ---- Session topology: FAILED! ```
The code is hitting this line in evlist__valid_sample_type as core.nr_entries is 2 on hybrid not 1:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/evlist.c?h=perf-tools-next#n1215
this causes perf_session__open to fail:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/session.c?h=perf-tools-next#n129
Thanks,
Ian
Arnaldo Carvalho de Melo
maintainerEm Fri, Jan 19, 2024 at 10:09:10PM -0800, Ian Rogers escreveu:
Quoted 42 lines
Quoted 8 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
- Arnaldo
Hi Arnaldo,
I'm seeing a test failure on Alderlake in perf-tools-next and wondered if it was on your radar:
``` 32: Session topology : --- start --- test child forked, pid 539222 templ file: /tmp/perf-test-HMet21 Using CPUID GenuineIntel-6-97-2 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) config 0x800000000 disabled 1 ------------------------------------------------------------ sys_perf_event_open: pid 0 cpu -1 group_fd -1 flags 0x8 = 4 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) config 0x400000000 disabled 1 ------------------------------------------------------------ sys_perf_event_open: pid 0 cpu -1 group_fd -1 flags 0x8 = 5 non matching sample_type FAILED tests/topology.c:73 can't get session test child finished with -1 ---- end ---- Session topology: FAILED! ```
Quoted 5 lines
The code is hitting this line in
evlist__valid_sample_typeas core.nr_entries is 2 on hybrid not 1:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/evlist.c?h=perf-tools-next#n1215
this causesperf_session__opento fail:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/session.c?h=perf-tools-next#n129
Yes, I see this as well, haven't yet analysed it
root@number:~# perf test -v topology 38: Session topology : --- start --- test child forked, pid 260120 templ file: /tmp/perf-test-WMa4M5 Using CPUID GenuineIntel-6-B7-1 non matching sample_type FAILED tests/topology.c:73 can't get session test child finished with -1 ---- end ---- Session topology: FAILED! root@number:~#
Quoted 43 lines · Arnaldo Carvalho de Melo
Em Fri, Jan 19, 2024 at 10:09:10PM -0800, Ian Rogers escreveu:
Quoted 42 lines
Quoted 8 lines · Arnaldo Carvalho de Melo
Hi Kan,
I noticed the following problem, are you able to reproduce it?
Happy solstice!
- Arnaldo
Hi Arnaldo,
I'm seeing a test failure on Alderlake in perf-tools-next and wondered if it was on your radar:
``` 32: Session topology : --- start --- test child forked, pid 539222 templ file: /tmp/perf-test-HMet21 Using CPUID GenuineIntel-6-97-2 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) config 0x800000000 disabled 1 ------------------------------------------------------------ sys_perf_event_open: pid 0 cpu -1 group_fd -1 flags 0x8 = 4 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) config 0x400000000 disabled 1 ------------------------------------------------------------ sys_perf_event_open: pid 0 cpu -1 group_fd -1 flags 0x8 = 5 non matching sample_type FAILED tests/topology.c:73 can't get session test child finished with -1 ---- end ---- Session topology: FAILED! ```
Quoted 7 lines
Quoted 5 lines
The code is hitting this line in
evlist__valid_sample_typeas core.nr_entries is 2 on hybrid not 1:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/evlist.c?h=perf-tools-next#n1215
this causesperf_session__opento fail:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/session.c?h=perf-tools-next#n129Yes, I see this as well, haven't yet analysed it
This should has been fixed by James.
https://lore.kernel.org/lkml/[email protected]/
Thanks,
Kan
Quoted 14 lines
root@number:~# perf test -v topology 38: Session topology : --- start --- test child forked, pid 260120 templ file: /tmp/perf-test-WMa4M5 Using CPUID GenuineIntel-6-B7-1 non matching sample_type FAILED tests/topology.c:73 can't get session test child finished with -1 ---- end ---- Session topology: FAILED! root@number:~#