NMI received for unknown reason when running perf with IBS on AMD
Hi Arnaldo,
I've been having a strange problem with perf when using IBS on an AMD 3950X processor. Whenever I use perf top or perf record with the default event, which corresponds to cycles:P, I get these messages in dmesg output:
[443324.266243] Uhhuh. NMI received for unknown reason 3c on CPU 1. [443324.266246] Dazed and confused, but trying to continue [443324.290039] Uhhuh. NMI received for unknown reason 2c on CPU 9. [443324.290042] Dazed and confused, but trying to continue [443324.307334] Uhhuh. NMI received for unknown reason 3c on CPU 9. [443324.307336] Dazed and confused, but trying to continue [443324.404938] Uhhuh. NMI received for unknown reason 2c on CPU 9. [443324.404940] Dazed and confused, but trying to continue
If I decrease the frequency I use for sampling, the messages also decrease in frequency, but even with a low sampling frequency, eventually they start to appear. Interestingly, if I use simply cycles as the event, the problem does not happen. However, if I use cycles:pp and a single CPU, it is less frequent, but does happen. Only CPUs used for measurement show up in the error messages as well (i.e. if I restrict to CPU 0, only CPU 0 shows in the messages above), and the more CPUs I use, the more frequent the messages.
Please let me know how I could help to debug this problem further. The output of perf record then perf report --header-only follows below, along with perf version --build-options.
Best regards,
-Guilherme
$ perf report --header-only
# ========
# captured on : Mon Oct 16 13:57:34 2023
# header version : 1
# data offset : 664
# data size : 884904
# feat offset : 885568
# hostname : gentoo.cern.ch
# os release : 6.5.5-gentoo
# perf version : 6.5
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : AMD Ryzen 9 3950X 16-Core Processor
# cpuid : AuthenticAMD,23,113,0
# total memory : 32748648 kB
# cmdline : /usr/bin/perf record -a -e cycles:pp -- sleep 1
# event : name = cycles:pp, , id = { 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824 }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0 (PERF_COUNT_HW_CPU_CYCLES), { sample_period, sample_freq } = 4000, sample_type = IP|TID|TIME|ID|CPU|PERIOD, read_format = ID, disabled = 1, inherit = 1, freq = 1, precise_ip = 2, sample_id_all = 1
# event : name = dummy:HG, , id = { 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840 }, type = 1 (PERF_TYPE_SOFTWARE), size = 136, config = 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq } = 4000, sample_type = IP|TID|TIME|ID|CPU|PERIOD, read_format = ID, inherit = 1, mmap = 1, comm = 1, freq = 1, task = 1, sample_id_all = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1
# CPU_TOPOLOGY info available, use -I to display
# NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, amd_df = 13, software = 1, ibs_op = 12, power = 10, ibs_fetch = 11, uprobe = 9, amd_iommu_0 = 15, breakpoint = 5, amd_l3 = 14, tracepoint = 2, kprobe = 8, msr = 16
# CACHE info available, use -I to display
# time of first sample : 444374.952454
# time of last sample : 444375.960893
# sample duration : 1008.439 ms
# cpu pmu capabilities: max_precise=0
# missing features: TRACING_DATA BRANCH_STACK GROUP_DESC AUXTRACE STAT MEM_TOPOLOGY CLOCKID DIR_FORMAT COMPRESSED CLOCK_DATA HYBRID_TOPOLOGY
# ========
#$ perf version --build-options
perf version 6.5
dwarf: [ on ] # HAVE_DWARF_SUPPORT
dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT
syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT
libbfd: [ on ] # HAVE_LIBBFD_SUPPORT
debuginfod: [ OFF ] # HAVE_DEBUGINFOD_SUPPORT
libelf: [ on ] # HAVE_LIBELF_SUPPORT
libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
libperl: [ on ] # HAVE_LIBPERL_SUPPORT
libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
libslang: [ on ] # HAVE_SLANG_SUPPORT
libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT
libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT
zlib: [ on ] # HAVE_ZLIB_SUPPORT
lzma: [ on ] # HAVE_LZMA_SUPPORT
get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORT
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
aio: [ on ] # HAVE_AIO_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libtraceevent: [ on ] # HAVE_LIBTRACEEVENTArnaldo Carvalho de Melo
maintainerpointers:
Ravi: the max_precise for core/ibs, can you please see if I got it right?
Em Mon, Oct 16, 2023 at 02:00:58PM +0200, Guilherme Amadio escreveu:
Quoted 41 lines
Hi Arnaldo,
I've been having a strange problem with perf when using IBS on an AMD 3950X processor. Whenever I use perf top or perf record with the default event, which corresponds to cycles:P, I get these messages in dmesg output:
[443324.266243] Uhhuh. NMI received for unknown reason 3c on CPU 1. [443324.266246] Dazed and confused, but trying to continue [443324.290039] Uhhuh. NMI received for unknown reason 2c on CPU 9. [443324.290042] Dazed and confused, but trying to continue [443324.307334] Uhhuh. NMI received for unknown reason 3c on CPU 9. [443324.307336] Dazed and confused, but trying to continue [443324.404938] Uhhuh. NMI received for unknown reason 2c on CPU 9. [443324.404940] Dazed and confused, but trying to continueIf I decrease the frequency I use for sampling, the messages also decrease in frequency, but even with a low sampling frequency, eventually they start to appear. Interestingly, if I use simply cycles as the event, the problem does not happen. However, if I use cycles:pp and a single CPU, it is less frequent, but does happen. Only CPUs used for measurement show up in the error messages as well (i.e. if I restrict to CPU 0, only CPU 0 shows in the messages above), and the more CPUs I use, the more frequent the messages.
Please let me know how I could help to debug this problem further. The output of perf record then perf report --header-only follows below, along with perf version --build-options.
Best regards,
-Guilherme$ perf report --header-only # ======== # captured on : Mon Oct 16 13:57:34 2023 # header version : 1 # data offset : 664 # data size : 884904 # feat offset : 885568 # hostname : gentoo.cern.ch # os release : 6.5.5-gentoo
Quoted 6 lines
# perf version : 6.5 # arch : x86_64 # nrcpus online : 16 # nrcpus avail : 16 # cpudesc : AMD Ryzen 9 3950X 16-Core Processor # cpuid : AuthenticAMD,23,113,0
So, testing on a Ryzen 9 5950X:
[root@five ~]# uname -a
Linux five 6.5.5-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Sep 24 15:52:44 UTC 2023 x86_64 GNU/Linux
[root@five ~]# dmesg | grep -i nmi
[ 0.059840] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.592555] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[root@five ~]#
When I ran with "perf top -e cycles:P' I got:
[19184.078663] perf: interrupt took too long (2539 > 2500), lowering kernel.perf_event_max_sample_rate to 78000 [19198.077425] perf: interrupt took too long (3206 > 3173), lowering kernel.perf_event_max_sample_rate to 62000
[root@five ~]# pidof perf
57383
[root@five ~]# gdb -p 57383
GNU gdb (GDB) Fedora Linux 13.2-5.fc38
Attaching to process 57383
[New LWP 57384]
[New LWP 57417]
[New LWP 57418]
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f7144d2734d in poll () from /lib64/libc.so.6 (gdb) bt
#0 0x00007f7144d2734d in poll () from /lib64/libc.so.6 #1 0x000000000069f174 in fdarray__poll (fda=0x19c0aa0, timeout=100) at fd/array.c:138 #2 0x00000000006a4d55 in perf_evlist__poll (evlist=0x19c0a60, timeout=100) at evlist.c:370 #3 0x0000000000523171 in evlist__poll (evlist=0x19c0a60, timeout=100) at util/evlist.c:636 #4 0x000000000043d6f4 in __cmd_top (top=0x7ffc83ef4070) at builtin-top.c:1353 #5 0x000000000043f348 in cmd_top (argc=0, argv=0x7ffc83ef82b0) at builtin-top.c:1835 #6 0x000000000050462e in run_builtin (p=0xdf0dc0 <commands+384>, argc=3, argv=0x7ffc83ef82b0) at perf.c:322 #7 0x000000000050489d in handle_internal_command (argc=3, argv=0x7ffc83ef82b0) at perf.c:375 #8 0x00000000005049ec in run_argv (argcp=0x7ffc83ef80cc, argv=0x7ffc83ef80c0) at perf.c:419 #9 0x0000000000504cd9 in main (argc=3, argv=0x7ffc83ef82b0) at perf.c:535
(gdb) fr 2
#2 0x00000000006a4d55 in perf_evlist__poll (evlist=0x19c0a60, timeout=100) at evlist.c:370
370 return fdarray__poll(&evlist->pollfd, timeout);
(gdb) print perf_evlist__first(evlist)
$1 = (struct perf_evsel *) 0x19e86b0
(gdb) print evsel__name(perf_evlist__first(evlist))
$2 = 0x19e2a40 "cycles:P"
(gdb) print perf_evlist__first(evlist)->attr
$3 = {type = 0, size = 136, config = 0, {sample_period = 4000, sample_freq = 4000}, sample_type = 391, read_format = 20, disabled = 1, inherit = 1, pinned = 0, exclusive = 0, exclude_user = 0,
exclude_kernel = 0, exclude_hv = 0, exclude_idle = 0, mmap = 1, comm = 1, freq = 1, inherit_stat = 0, enable_on_exec = 0, task = 1, watermark = 0, precise_ip = 2, mmap_data = 0, sample_id_all = 1, exclude_host = 0, exclude_guest = 0, exclude_callchain_kernel = 0, exclude_callchain_user = 0, mmap2 = 1, comm_exec = 1, use_clockid = 0, context_switch = 0, write_backward = 0, namespaces = 0, ksymbol = 1,
bpf_event = 1, aux_output = 0, cgroup = 0, text_poke = 0, build_id = 0, inherit_thread = 0, remove_on_exec = 0, sigtrap = 0, __reserved_1 = 0, {wakeup_events = 0, wakeup_watermark = 0}, bp_type = 0, {
bp_addr = 0, kprobe_func = 0, uprobe_path = 0, config1 = 0}, {bp_len = 0, kprobe_addr = 0, probe_offset = 0, config2 = 0}, branch_sample_type = 0, sample_regs_user = 0, sample_stack_user = 0, clockid = 0, sample_regs_intr = 0, aux_watermark = 0, sample_max_stack = 0, __reserved_2 = 0, aux_sample_size = 0, __reserved_3 = 0, sig_data = 0, config3 = 0}
(gdb) print perf_evlist__first(evlist)->attr.precise_ip
$4 = 2
(gdb)
So it seems to be using IBS:
[root@five ~]# dmesg | grep -w perf
[ 0.740947] perf: AMD IBS detected (0x000003ff) [ 0.741476] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank). [19184.078663] perf: interrupt took too long (2539 > 2500), lowering kernel.perf_event_max_sample_rate to 78000 [19198.077425] perf: interrupt took too long (3206 > 3173), lowering kernel.perf_event_max_sample_rate to 62000
[root@five ~]#
[root@five ~]# for handle in 0 1 2 ; do dmidecode --handle $handle ; done
# dmidecode 3.4 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Table at 0x000E8D60.
Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: American Megatrends International, LLC.
Version: F35d
Release Date: 10/13/2021
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16 MB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
Japanese floppy for NEC 9800 1.2 MB is supported (int 13h) Japanese floppy for Toshiba 1.2 MB is supported (int 13h) 5.25"/360 kB floppy services are supported (int 13h) 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) Serial services are supported (int 14h) Printer services are supported (int 17h) CGA/mono video services are supported (int 10h) ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported UEFI is supported
BIOS Revision: 5.17
# dmidecode 3.4 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Table at 0x000E8D60.
Handle 0x0001, DMI type 1, 27 bytes
System Information
Manufacturer: Gigabyte Technology Co., Ltd. Product Name: X570 AORUS PRO
Version: -CF
Serial Number: Default string
UUID: 032e02b4-0499-05e3-6906-f30700080009 Wake-up Type: PCI PME#
SKU Number: Default string
Family: X570 MB
# dmidecode 3.4 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Table at 0x000E8D60.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Gigabyte Technology Co., Ltd. Product Name: X570 AORUS PRO
Version: Default string
Serial Number: Default string
Asset Tag: Default string
Features:
Board is a hosting board Board is replaceable
Location In Chassis: Default string Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
[root@five ~]#
Quoted 11 lines
# total memory : 32748648 kB
# cmdline : /usr/bin/perf record -a -e cycles:pp -- sleep 1
# event : name = cycles:pp, , id = { 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824 }, type = 0 (PERF_TYPE_HARDWARE), size = 136, config = 0 (PERF_COUNT_HW_CPU_CYCLES), { sample_period, sample_freq } = 4000, sample_type = IP|TID|TIME|ID|CPU|PERIOD, read_format = ID, disabled = 1, inherit = 1, freq = 1, precise_ip = 2, sample_id_all = 1
# event : name = dummy:HG, , id = { 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840 }, type = 1 (PERF_TYPE_SOFTWARE), size = 136, config = 0x9 (PERF_COUNT_SW_DUMMY), { sample_period, sample_freq } = 4000, sample_type = IP|TID|TIME|ID|CPU|PERIOD, read_format = ID, inherit = 1, mmap = 1, comm = 1, freq = 1, task = 1, sample_id_all = 1, mmap2 = 1, comm_exec = 1, ksymbol = 1, bpf_event = 1
# CPU_TOPOLOGY info available, use -I to display
# NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, amd_df = 13, software = 1, ibs_op = 12, power = 10, ibs_fetch = 11, uprobe = 9, amd_iommu_0 = 15, breakpoint = 5, amd_l3 = 14, tracepoint = 2, kprobe = 8, msr = 16
# CACHE info available, use -I to display
# time of first sample : 444374.952454
# time of last sample : 444375.960893
# sample duration : 1008.439 msHumm:
Quoted 1 line
# cpu pmu capabilities: max_precise=0
For me:
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0 [root@five ~]#
this is weird for both machines, as:
static ssize_t max_precise_show(struct device *cdev,
struct device_attribute *attr, char *buf)
{
return snprintf(buf, PAGE_SIZE, "%d\n", x86_pmu_max_precise());
}static DEVICE_ATTR_RO(max_precise);
static struct attribute *x86_pmu_caps_attrs[] = {
&dev_attr_max_precise.attr,
NULL
};
int x86_pmu_hw_config(struct perf_event *event)
{
if (event->attr.precise_ip) {
int precise = x86_pmu_max_precise();
if (event->attr.precise_ip > precise)
return -EOPNOTSUPP;
/* There's no sense in having PEBS for non sampling events: */
if (!is_sampling_event(event))
return -EINVAL;
}[root@five ~]# cat /sys/devices/cpu/caps/max_precise 0
[root@five ~]#
Ravi, this probably is the max_precise for the "core" PMU, but the "ibs" one, that gets used when :pp and :p (but not :ppp) is used, i.e. that:
+++ b/arch/x86/events/amd/core.c
@@ -374,7 +374,7 @@ static int amd_pmu_hw_config(struct perf_event *event)
/* pass precise event sampling to ibs: */
if (event->attr.precise_ip && get_ibs_caps())- return -ENOENT;
+ return forward_event_to_ibs(event);
Thing..
Shouldn't we have some sort of max_precise cap for IBS so that we could use it and not have this confusing "max_precise=0" for the core PMU but accept attr.precise_ip = 1 and 2?
Quoted 20 lines
# missing features: TRACING_DATA BRANCH_STACK GROUP_DESC AUXTRACE STAT MEM_TOPOLOGY CLOCKID DIR_FORMAT COMPRESSED CLOCK_DATA HYBRID_TOPOLOGY # ======== #$ perf version --build-options perf version 6.5 dwarf: [ on ] # HAVE_DWARF_SUPPORT dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT libbfd: [ on ] # HAVE_LIBBFD_SUPPORT debuginfod: [ OFF ] # HAVE_DEBUGINFOD_SUPPORT libelf: [ on ] # HAVE_LIBELF_SUPPORT libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT libperl: [ on ] # HAVE_LIBPERL_SUPPORT libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT libslang: [ on ] # HAVE_SLANG_SUPPORT libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT
Quoted 8 lines
zlib: [ on ] # HAVE_ZLIB_SUPPORT
lzma: [ on ] # HAVE_LZMA_SUPPORT
get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORT
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
aio: [ on ] # HAVE_AIO_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libtraceevent: [ on ] # HAVE_LIBTRACEEVENT⬢[acme@toolbox perf-tools-next]$ perf -vv
perf version 6.6.rc1.gcdfd9c216f54
dwarf: [ on ] # HAVE_DWARF_SUPPORT
dwarf_getlocations: [ on ] # HAVE_DWARF_GETLOCATIONS_SUPPORT
syscall_table: [ on ] # HAVE_SYSCALL_TABLE_SUPPORT
libbfd: [ OFF ] # HAVE_LIBBFD_SUPPORT
debuginfod: [ on ] # HAVE_DEBUGINFOD_SUPPORT
libelf: [ on ] # HAVE_LIBELF_SUPPORT
libnuma: [ on ] # HAVE_LIBNUMA_SUPPORT
numa_num_possible_cpus: [ on ] # HAVE_LIBNUMA_SUPPORT
libperl: [ on ] # HAVE_LIBPERL_SUPPORT
libpython: [ on ] # HAVE_LIBPYTHON_SUPPORT
libslang: [ on ] # HAVE_SLANG_SUPPORT
libcrypto: [ on ] # HAVE_LIBCRYPTO_SUPPORT
libunwind: [ on ] # HAVE_LIBUNWIND_SUPPORT
libdw-dwarf-unwind: [ on ] # HAVE_DWARF_SUPPORT
zlib: [ on ] # HAVE_ZLIB_SUPPORT
lzma: [ on ] # HAVE_LZMA_SUPPORT
get_cpuid: [ on ] # HAVE_AUXTRACE_SUPPORT
bpf: [ on ] # HAVE_LIBBPF_SUPPORT
aio: [ on ] # HAVE_AIO_SUPPORT
zstd: [ on ] # HAVE_ZSTD_SUPPORT
libpfm4: [ on ] # HAVE_LIBPFM
libtraceevent: [ on ] # HAVE_LIBTRACEEVENT
bpf_skeletons: [ on ] # HAVE_BPF_SKEL⬢[acme@toolbox perf-tools-next]$
Hi Arnaldo, Guilherme,
Quoted 2 lines
Quoted 2 lines
[443324.404938] Uhhuh. NMI received for unknown reason 2c on CPU 9. [443324.404940] Dazed and confused, but trying to continue
This is a known hw bug in Zen2:
https://lore.kernel.org/lkml/[email protected]
You won't see it on later Zen processors, or at least the possibility of hitting it is very less.
Quoted 5 lines
(gdb) print perf_evlist__first(evlist)->attr.precise_ip $4 = 2
(gdb)So it seems to be using IBS:
Correct. More details below.
Quoted 8 lines
Humm:
Quoted 1 line
# cpu pmu capabilities: max_precise=0For me:
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0
Although confusing, this is technically correct because AMD core pmu does not support precise mode.
However, as a special case, 3 core pmu events: cpu-cycles, r076 (same as cpu-cycles) and r0C1 (micro-ops) are supported with attr.precise_ip > 0, and they are forwarded to IBS OP pmu. All other core pmu events with attr.precise_ip > 0 fails with -EINVAL.
Quoted 17 lines
[root@five ~]# cat
/sys/devices/cpu/caps/max_precise0
[root@five ~]#Ravi, this probably is the
max_precisefor the "core" PMU, but the "ibs" one, that gets used when :pp and :p (but not :ppp) is used, i.e. that:+++
b/arch/x86/events/amd/core.c(quoted diff) +1
-1@@ -374,7 +374,7 @@ static int amd_pmu_hw_config(struct perf_event *event) /* pass precise event sampling to ibs: */ if (event->attr.precise_ip && get_ibs_caps()) - return -ENOENT; + return forward_event_to_ibs(event);Thing..
Quoted 3 lines
Shouldn't we have some sort of
max_precisecap for IBS so that we could use it and not have this confusing "max_precise=0" for the core PMU but accept attr.precise_ip = 1 and 2?
Unlike Intel PEBS which provides levels of precision, AMD core pmu is inherently non-precise and IBS is inherently precise. So max_precise value is irrelevant for IBS. I mean ibs_op//, ibs_op//p, ibs_op//pp and ibs_op//ppp are all same.
Thanks,
Ravi
PS: I don't know the history behind not supporting attr.precise_ip = 3 when event is forwarded from core pmu to IBS. This check was added long back in 2012. Robert might know.
Arnaldo Carvalho de Melo
maintainerEm Tue, Oct 17, 2023 at 01:31:30PM +0530, Ravi Bangoria escreveu:
Quoted 10 lines
Hi Arnaldo, Guilherme,
Quoted 2 lines
Quoted 2 lines
[443324.404938] Uhhuh. NMI received for unknown reason 2c on CPU 9. [443324.404940] Dazed and confused, but trying to continueThis is a known hw bug in Zen2:
https://lore.kernel.org/lkml/[email protected]You won't see it on later Zen processors, or at least the possibility of hitting it is very less.
Do you think we can detect that it is a zen2 processor, that it has this problem, and then the tools can just avoid using precise_ip != 0? I.e. "cycles:P" becomes plain "cycles" and asking for, say, "cycles:p" states that this isn't possible on Zen2?
Quoted 17 lines
Quoted 5 lines
(gdb) print perf_evlist__first(evlist)->attr.precise_ip $4 = 2
(gdb)So it seems to be using IBS:
Correct. More details below.
Quoted 8 lines
Humm:
Quoted 1 line
# cpu pmu capabilities: max_precise=0For me:
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0
Quoted 7 lines
Although confusing, this is technically correct because AMD core pmu does not support precise mode.
However, as a special case, 3 core pmu events: cpu-cycles, r076 (same as cpu-cycles) and r0C1 (micro-ops) are supported with attr.precise_ip > 0, and they are forwarded to IBS OP pmu. All other core pmu events with attr.precise_ip > 0 fails with
-EINVAL.
If we can programatically detect that IBS is present we can add a note right after that 'perf report --header-only' line with this information.
Quoted 17 lines
Quoted 17 lines
[root@five ~]# cat
/sys/devices/cpu/caps/max_precise0
[root@five ~]#Ravi, this probably is the
max_precisefor the "core" PMU, but the "ibs" one, that gets used when :pp and :p (but not :ppp) is used, i.e. that:+++
b/arch/x86/events/amd/core.c(quoted diff) +1
-1@@ -374,7 +374,7 @@ static int amd_pmu_hw_config(struct perf_event *event) /* pass precise event sampling to ibs: */ if (event->attr.precise_ip && get_ibs_caps()) - return -ENOENT; + return forward_event_to_ibs(event);Thing..
Quoted 8 lines
Quoted 3 lines
Shouldn't we have some sort of
max_precisecap for IBS so that we could use it and not have this confusing "max_precise=0" for the core PMU but accept attr.precise_ip = 1 and 2?Unlike Intel PEBS which provides levels of precision, AMD core pmu is inherently non-precise and IBS is inherently precise. So
max_precisevalue is irrelevant for IBS. I mean ibs_op//, ibs_op//p, ibs_op//pp and ibs_op//ppp are all same.
This part can go to the man page, where :p, :P is documented.
Quoted 6 lines
Thanks,
RaviPS: I don't know the history behind not supporting attr.precise_ip = 3 when event is forwarded from core pmu to IBS. This check was added long back in 2012. Robert might know.
Robert?
Quoted 4 lines
Do you think we can detect that it is a zen2 processor, that it has this problem, and then the tools can just avoid using
precise_ip!= 0? I.e. "cycles:P" becomes plain "cycles" and asking for, say, "cycles:p" states that this isn't possible on Zen2?
It's possible, but there is no architected way to detect Zen revision 🙁. I will need to key off based on Family and Model checks, which is ugly. Also, the same problem can happen if user invokes IBS pmus directly.
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0...
If we can programatically detect that IBS is present we can add a note right after that 'perf report --header-only' line with this information.
This should be easy to implement.
Quoted 6 lines
Quoted 4 lines
Unlike Intel PEBS which provides levels of precision, AMD core pmu is inherently non-precise and IBS is inherently precise. So
max_precisevalue is irrelevant for IBS. I mean ibs_op//, ibs_op//p, ibs_op//pp and ibs_op//ppp are all same.This part can go to the man page, where :p, :P is documented.
Sure. Let me prepare a patch.
Thanks,
Ravi
Arnaldo Carvalho de Melo
maintainerEm Wed, Oct 18, 2023 at 09:29:01AM +0530, Ravi Bangoria escreveu:
Quoted 9 lines
Quoted 4 lines
Do you think we can detect that it is a zen2 processor, that it has this problem, and then the tools can just avoid using
precise_ip!= 0? I.e. "cycles:P" becomes plain "cycles" and asking for, say, "cycles:p" states that this isn't possible on Zen2?It's possible, but there is no architected way to detect Zen revision 🙁. I will need to key off based on Family and Model checks, which is ugly. Also, the same problem can happen if user invokes IBS pmus directly.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0...
If we can programatically detect that IBS is present we can add a note right after that 'perf report --header-only' line with this information.This should be easy to implement.
Yeah, I took at stab at it, see the patch below:
[root@five ~]# perf report --header-only | grep "cpu pmu" -A1
# cpu pmu capabilities: max_precise=0 # AMD systems will use ibs_op PMU for some precise events, e.g.: cycles:p, see the 'perf report' man page for further details. [root@five ~]#
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Unlike Intel PEBS which provides levels of precision, AMD core pmu is inherently non-precise and IBS is inherently precise. So
max_precisevalue is irrelevant for IBS. I mean ibs_op//, ibs_op//p, ibs_op//pp and ibs_op//ppp are all same.This part can go to the man page, where :p, :P is documented.
Sure. Let me prepare a patch.
Great!
- Arnaldo
tools/perf/util/env.c +18 -0
@@ -531,6 +531,24 @@ int perf_env__numa_node(struct perf_env *env, struct perf_cpu cpu)return cpu.cpu >=0 && cpu.cpu < env->nr_numa_map ? env->numa_map[cpu.cpu] :-1 ; } +bool perf_env__has_pmu_mapping (struct perf_env *env,const char *pmu_name) +{ +char *pmu_mapping = env->pmu_mappings, *colon; + +for (int i =0 ; i < env->nr_pmu_mappings; ++i) { +if (strtoul (pmu_mapping, &colon,0 ) ==ULONG_MAX || *colon !=':' ) + goto out_error; + + pmu_mapping = colon +1 ; +if (strcmp (pmu_mapping, pmu_name) ==0 ) +return true; + + pmu_mapping +=strlen (pmu_mapping) +1 ; + } +out_error: +return false; +} +char *perf_env__find_pmu_cap (struct perf_env *env,const char *pmu_name,const char *cap) {
tools/perf/util/env.h +2 -0
@@ -174,4 +174,6 @@ struct btf_node *perf_env__find_btf(struct perf_env *env, __u32 btf_id);int perf_env__numa_node (struct perf_env *env,struct perf_cpu cpu);char *perf_env__find_pmu_cap (struct perf_env *env,const char *pmu_name,const char *cap); + +bool perf_env__has_pmu_mapping (struct perf_env *env,const char *pmu_name);#endif /* __PERF_ENV_H */
tools/perf/util/header.c +8 -0
@@ -2145,6 +2145,14 @@ static void print_pmu_caps(struct feat_fd *ff, FILE *fp)__print_pmu_caps (fp, pmu_caps->nr_caps, pmu_caps->caps, pmu_caps->pmu_name); } + +if (strcmp (perf_env__arch (&ff->ph->env),"x86" ) ==0 && +perf_env__has_pmu_mapping (&ff->ph->env,"ibs_op" )) { +char *max_precise =perf_env__find_pmu_cap (&ff->ph->env,"cpu" ,"max_precise" ); + +if (max_precise !=NULL &&atoi (max_precise) ==0 ) +fprintf (fp,"# AMD systems will use ibs_op PMU for some precise events, e.g.: cycles:p, see the 'perf report' man page for further details.\n" ); + } }static void print_pmu_mappings (struct feat_fd *ff,FILE *fp)
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0...
If we can programatically detect that IBS is present we can add a note right after that 'perf report --header-only' line with this information.This should be easy to implement.
Yeah, I took at stab at it, see the patch below:
Looks good. Thanks!
Ravi
Arnaldo Carvalho de Melo
maintainerEm Wed, Oct 18, 2023 at 09:22:26PM +0530, Ravi Bangoria escreveu:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
[root@five ~]# perf report --header-only | grep "cpu pmu capabilities"
# cpu pmu capabilities: max_precise=0...
If we can programatically detect that IBS is present we can add a note right after that 'perf report --header-only' line with this information.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
This should be easy to implement.
Yeah, I took at stab at it, see the patch below:
Looks good. Thanks!
Ravi
Thanks for checking, I'll submit it properly and will have your Acked-by there then,
- Arnaldo