Skip to contentSkip to conversation

perf/amd/ibs: [BUG?] ibs_op sampling silently stops on Zen3: MSR_AMD64_IBSOPCTL latched, no NMI delivered

DISCUSSION Liew Rui Yan 2 participants 2 messages Oct 19 – Oct 27

lore
Share & tools
lore
#0 lore raw

Hi Perf Community,

I found an issue with ibs_op sampling that I can reproduce on my AMD Zen3 laptop. Under full CPU load, sampling PMIs stop arriving on each CPU, one after another. Reading MSR_AMD64_IBSOPCTL (0xc0011033) while perf is still running shows the hardware has latched a completed sample (IBS_OP_ENABLE=1, IBS_OP_VAL=1, IBS_OP_CUR_CNT frozen) and is waiting for an NMI that never arrives. The CPU never recovers on its own.

I am really not very familiar with CPU, so I would appreciate any guidance on whether this is known and what to test next.

Environment
===========

- CPU: AMD Ryzen 5 5600H with Radeon Graphics (family 25 model 80,

          Zen3/Cezanne), 12 cores, microcode 0xa500012

- Kernel: 7.3.0-rc5-mm-new-damon-1 (self-built from the mm tree)
- perf: perf version 7.2.11-1
- Also running: stress-ng 0.21.01

How to reproduce
================

Reproducer is here:

    https://github.com/aethernet65535/amd-zen3-nmi-lost-reproducer.git

The essence is a system-wide ibs_op record on top of a full CPU load:

    perf record -a -e ibs_op// -c 100000 -- stress-ng --cpu 0 &

and then watch the per-CPU PMI counters in /proc/interrupts once a second. To rule perf's dynamic throttling out as a factor, I set kernel.perf_event_max_sample_rate=100000 and kernel.perf_cpu_time_max_percent=0 before starting.

Reproduction is fast and consistent: the first CPU goes silent ~20 s after perf starts, and all 12 CPUs are silent within ~5 minutes.

Observed behaviour
==================

Found that per-CPU PMI counters (/proc/interrupts) stop incrementing. From the most recent run, time from perf start until each CPU was detected silent (zero PMI delta for 3 consecutive 1 s windows):

    CPU 1,5:        +23.7 s
    CPU 2,3:        +31.6 s
    CPU 8:          +62.2 s
    CPU 9:          +68.3 s
    CPU 10:         +96.9 s
    CPU 6:          +166.9 s
    CPU 4:          +192.5 s
    CPU 0:          +198.5 s
    CPU 11:         +220.0 s
    CPU 7:          +271.3 s

The run then continued for another 120 s to confirm if CPU comes back; none did.

For each CPU at the moment it went silent, I read MSR_AMD64_IBSOPCTL three times, via rdmsr while perf was still running. Decoding IBS_OP_ENABLE (bit 17), IBS_OP_VAL (bit 18) and IBS_OP_CUR_CNT (bits 32-58):

Live control CPU 0 -- IBS_OP_VAL=0, IBS_OP_CUR_CNT keeps moving between reads:

read 1: raw=0x231f0002186a IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=8991 read 2: raw=0x10ca50002186a IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=68773 read 3: raw=0x15fa10002186a IBS_OP_ENABLE=1 IBS_OP_VAL=0 IBS_OP_CUR_CNT=90017

Silent CPU 0 -- three reads are bit-for-bit identical:

  read 1: raw=0xc80006186a  IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200
  read 2: raw=0xc80006186a  IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200
  read 3: raw=0xc80006186a  IBS_OP_ENABLE=1 IBS_OP_VAL=1 IBS_OP_CUR_CNT=200

perf report --header-only excerpt (trimmed):

    # os release : 7.3.0-rc5-mm-new-damon-1
    # perf version : 7.2.11-1
    # arch : x86_64
    # nrcpus online : 12
    # cpudesc : AMD Ryzen 5 5600H with Radeon Graphics
    # cpuid : AuthenticAMD,25,80,0
    # event : name = ibs_op//, , type = 11 (ibs_op), config = 0,
      { sample_period, sample_freq } = 100000,
      sample_type = IP|TID|TIME|ID|CPU, read_format = ID|LOST
    # pmu mappings: ... ibs_fetch = 10, ibs_op = 11 ...
    # cpu pmu capabilities: max_precise=0

What I think I have ruled out
=============================

- Not perf's dynamic throttling: it is disabled (percent=0) for every run.
- Not the kernel stopping the event: IBS_OP_ENABLE stays 1; the event is still armed.
- Sampling does work normally for a while before it stops, so the setup itself is fine.

One more observation
====================

In earlier testing I noticed there is a workaround -- enable NMI watchdog. PMI counters start incrementing again after it is enabled.

Questions
=========

1. Is this known on Zen3/Cezanne?
2. Any recommended next step?

Thank you for your time, and for all your work on perf.

Best regards,
Rui Yan

#1 lore raw
Quoted 9 lines

I found an issue with ibs_op sampling that I can reproduce on my AMD Zen3 laptop. Under full CPU load, sampling PMIs stop arriving on each CPU, one after another. Reading MSR_AMD64_IBSOPCTL (0xc0011033) while perf is still running shows the hardware has latched a completed sample (IBS_OP_ENABLE=1, IBS_OP_VAL=1, IBS_OP_CUR_CNT frozen) and is waiting for an NMI that never arrives. The CPU never recovers on its own.

I am really not very familiar with CPU, so I would appreciate any guidance on whether this is known and what to test next.

This is a known Zen3 hardware bug. The workaround involves opening a PMC event alongside the IBS event so that the IBS sample is acknowledged during the PMC NMI, thereby resuming the stalled IBS instance.

More details in The Revision guide:

https://bugzilla.kernel.org/attachment.cgi?id=311307
Erratum 1285 Processor May Fail to Report IBS (Instruction Based
Sampling) Performance Sample

Thanks,
Ravi