perf/amd/ibs: [BUG?] ibs_op sampling silently stops on Zen3: MSR_AMD64_IBSOPCTL latched, no NMI delivered
Hi Perf Community,
I found an issue with ibs_op sampling that I can reproduce on my AMD Zen3 laptop. Under full CPU load, sampling PMIs stop arriving on each CPU, one after another. Reading MSR_AMD64_IBSOPCTL (0xc0011033) while perf is still running shows the hardware has latched a completed sample (IBS_OP_ENABLE=1, IBS_OP_VAL=1, IBS_OP_CUR_CNT frozen) and is waiting for an NMI that never arrives. The CPU never recovers on its own.
I am really not very familiar with CPU, so I would appreciate any guidance on whether this is known and what to test next.
Environment
===========
- CPU: AMD Ryzen 5 5600H with Radeon Graphics (family 25 model 80,
Zen3/Cezanne), 12 cores, microcode 0xa500012
- Kernel: 7.3.0-rc5-mm-new-damon-1 (self-built from the mm tree)
- perf: perf version 7.2.11-1
- Also running: stress-ng 0.21.01
How to reproduce
================
Reproducer is here:
https://github.com/aethernet65535/amd-zen3-nmi-lost-reproducer.git
The essence is a system-wide ibs_op record on top of a full CPU load:
perf record -a -e ibs_op// -c 100000 -- stress-ng --cpu 0 &
and then watch the per-CPU PMI counters in /proc/interrupts once a second. To rule perf's dynamic throttling out as a factor, I set kernel.perf_event_max_sample_rate=100000 and kernel.perf_cpu_time_max_percent=0 before starting.
Reproduction is fast and consistent: the first CPU goes silent ~20 s after perf starts, and all 12 CPUs are silent within ~5 minutes.
Observed behaviour
==================
Found that per-CPU PMI counters (/proc/interrupts) stop incrementing. From the most recent run, time from perf start until each CPU was detected silent (zero PMI delta for 3 consecutive 1 s windows):
CPU 1,5: +23.7 s
CPU 2,3: +31.6 s
CPU 8: +62.2 s
CPU 9: +68.3 s
CPU 10: +96.9 s
CPU 6: +166.9 s
CPU 4: +192.5 s
CPU 0: +198.5 s
CPU 11: +220.0 s
CPU 7: +271.3 sThe run then continued for another 120 s to confirm if CPU comes back; none did.
For each CPU at the moment it went silent, I read MSR_AMD64_IBSOPCTL three times, via rdmsr while perf was still running. Decoding IBS_OP_ENABLE (bit 17), IBS_OP_VAL (bit 18) and IBS_OP_CUR_CNT (bits 32-58):
Live control CPU 0 -- IBS_OP_VAL=0, IBS_OP_CUR_CNT keeps moving between reads:
read 1: raw=0x231f0002186a IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=8991 read 2: raw=0x10ca50002186a IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=68773 read 3: raw=0x15fa10002186a IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=90017
Silent CPU 0 -- three reads are bit-for-bit identical:
read 1: raw=0xc80006186a IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200 read 2: raw=0xc80006186a IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200 read 3: raw=0xc80006186a IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200
perf report --header-only excerpt (trimmed):
# os release : 7.3.0-rc5-mm-new-damon-1
# perf version : 7.2.11-1
# arch : x86_64
# nrcpus online : 12
# cpudesc : AMD Ryzen 5 5600H with Radeon Graphics
# cpuid : AuthenticAMD,25,80,0
# event : name = ibs_op//, , type = 11 (ibs_op), config = 0,
{ sample_period, sample_freq } = 100000,
sample_type = IP|TID|TIME|ID|CPU, read_format = ID|LOST
# pmu mappings: ... ibs_fetch = 10, ibs_op = 11 ...
# cpu pmu capabilities: max_precise=0What I think I have ruled out
=============================
- Not perf's dynamic throttling: it is disabled (percent=0) for every run.
- Not the kernel stopping the event: IBS_OP_ENABLE stays 1; the event is still armed.
- Sampling does work normally for a while before it stops, so the setup itself is fine.
One more observation
====================
In earlier testing I noticed there is a workaround -- enable NMI watchdog. PMI counters start incrementing again after it is enabled.
Questions
=========
1. Is this known on Zen3/Cezanne?
2. Any recommended next step?
Thank you for your time, and for all your work on perf.
Best regards,
Rui Yan
Quoted 9 lines
I found an issue with
ibs_opsampling that I can reproduce on my AMD Zen3 laptop. Under full CPU load, sampling PMIs stop arriving on each CPU, one after another. ReadingMSR_AMD64_IBSOPCTL(0xc0011033) while perf is still running shows the hardware has latched a completed sample (IBS_OP_ENABLE=1, IBS_OP_VAL=1,IBS_OP_CUR_CNTfrozen) and is waiting for an NMI that never arrives. The CPU never recovers on its own.I am really not very familiar with CPU, so I would appreciate any guidance on whether this is known and what to test next.
This is a known Zen3 hardware bug. The workaround involves opening a PMC event alongside the IBS event so that the IBS sample is acknowledged during the PMC NMI, thereby resuming the stalled IBS instance.
More details in The Revision guide:
https://bugzilla.kernel.org/attachment.cgi?id=311307
Erratum 1285 Processor May Fail to Report IBS (Instruction Based
Sampling) Performance Sample
Thanks,
Ravi