arm64: cross-CPU NMI via SDEI

From: "Kiryl Shutsemau (Meta)" <[email protected]>

A class of debug/observability features needs to interrupt a CPU that has its interrupts locally masked: the all-CPU backtrace behind sysrq-l / RCU-stall / hung-task / hard-lockup dumps, and crash_smp_send_stop() capturing a stuck CPU's state into the vmcore. On arm64 these need a mechanism that reaches a CPU spinning with DAIF masked, which a normal IPI cannot.

arm64 has two such mechanisms today:

- GICv3 pseudo-NMI (interrupt priority masking). The cost lands on the interrupt mask/unmask hot path: local_irq_enable() becomes an ICC_PMR_EL1 write, and exception entry/exit save and restore the PMR, paid on every CPU whether or not an NMI is ever delivered.

Measured on Grace (Neoverse V2; ICC_CTLR_EL1.PMHE=0, so the PMR-sync DSB is already patched to a NOP), pseudo_nmi=0 vs pseudo_nmi=1:

gettid() loop: 178 -> 253 ns/call (+42%, ~74 ns)
will-it-scale sched_yield: 0.705x throughput, flat from 1 to 72 cores will-it-scale page_fault1: within ~5%

The ~74 ns is a fixed per-syscall entry/exit tax -- it reproduces at +73.5 ns on Neoverse N2 -- so the hit tracks syscall/exception density and is unacceptable on syscall-bound fleet workloads, which therefore run with pseudo-NMI disabled.

- FEAT_NMI (Armv8.8) -- the architectural fix, but absent from deployed silicon and from most of the fleet for years to come.

For deployments that do not run pseudo-NMI, the backtrace and crash paths are degraded: a plain IPI can't reach the masked CPU, so the backtrace of the CPU you care about comes back empty and the kdump is missing the culprit's registers. The hard-lockup detector on these systems is the software buddy detector (HARDLOCKUP_DETECTOR_BUDDY): it detects a stall from a neighbour CPU, but it cannot itself interrupt the wedged CPU, so its report has no stack for the culprit and (with hardlockup_panic) the panic runs on the bystander.

This series adds a third delivery backend that costs nothing on the hot
path: SDEI. Firmware delivers an SDEI event into a CPU regardless of its DAIF state, so interrupt masking stays the cheap PSTATE.DAIF operation and the firmware round-trip is paid only at the rare moment a CPU must be interrupted.

It does not add a hard-lockup detector. Detection stays with the buddy detector (CONFIG_HARDLOCKUP_DETECTOR_PREFER_BUDDY); this series gives the backtrace and crash-stop paths -- including the buddy detector's backtrace of the stalled CPU -- a way to actually reach a masked CPU.

Mechanism
=========

It uses the standard SDEI software-signalled event (event 0) and the SDEI_EVENT_SIGNAL call (DEN0054) -- a spec-defined cross-PE signal, not a vendor extension. The driver registers a handler for event 0 and pokes a target CPU with sdei_event_signal(0, target_mpidr); firmware makes event 0 pending on that PE and dispatches the handler NMI-like.

No firmware change is required beyond SDEI being enabled, which firmware-first RAS (APEI/GHES) deployments already have; the only SDEI-core addition is a thin sdei_event_signal() wrapper over the standard call.

Prior SDEI watchdog work
========================

Out-of-tree SDEI hard-lockup watchdogs exist (e.g. in the openEuler and Anolis kernels). They bind the secure physical timer as an SDEI event, so firmware delivers a periodic self-CPU tick that drives a detector. That requires a new SDEI interrupt-binding API, pushes the watchdog period into firmware, and adds secure-timer EOI handling on the kexec path. This series instead uses only the standard software-signalled event 0, keeps all timing in the kernel (the buddy detector), and the same delivery primitive serves the backtrace and crash-stop users, not just lockup reporting.

Not included / follow-ups
=========================

- No SDEI hard-lockup-detector backend. v1 had one; it is dropped here. The buddy detector plus this series' backtrace already cover the no-pseudo-NMI case, and a dedicated SDEI backend duplicated the perf-NMI detector it had to compile-exclude. Run PREFER_BUDDY.

- A CPU stopped by the SDEI rung is parked, not powered off via PSCI CPU_OFF. Reaching and dumping the wedged CPU -- the point of the series -- works, and it matches the shared stop path's own park fallback when CPU_OFF is unavailable. The consequence is that an SMP crash-capture kernel cannot re-online such a CPU (it stays "already on"); the capture kernel boots and runs on the remaining CPUs. Powering the stopped CPU off so a capture kernel can reclaim it would need CPU_OFF from the SDEI stop context, which does not work in practice (see the v4 discussion); left as a follow-up and does not affect the dump's contents.

Testing
=======

Developed on QEMU 'virt' (Trusted Firmware-A with SDEI enabled) and validated on NVIDIA Grace (Neoverse V2) hardware, under irqchip.gicv3_pseudo_nmi=0 with HARDLOCKUP_DETECTOR_PREFER_BUDDY=y:

- sysrq-l backtrace of an interrupt-masked CPU returns its real stack,

    pstate showing DAIF set -- proof SDEI delivered into the masked CPU;
  - buddy detector catches a hard lockup (LKDTM) and the wedged CPU's
    stack is fetched via the SDEI backtrace;

- reboot/halt and the panic/kdump crash stop reach a wedged CPU via the SDEI rung ("SMP: retry stop with SDEI NMI for CPUs N"), and the kdump captures the wedged CPU's registers in the vmcore;
- with SDEI absent (plain QEMU 'virt', no firmware support) the driver stays inert: no event registration and no boot-time warning.

Changes since v4
================

- Strengthen the publish barrier from smp_wmb() to dsb(ishst) on the stop path, and add the missing one on the backtrace path; the SMC is not a memory store, so ordering alone is not enough (Catalin Marinas).
- Drop the redundant ARM64 Kconfig dependency, already implied by ARM_SDE_INTERFACE (Julian Braha).
- Reviewed-by from Douglas Anderson now on all four patches.
- Replaced the stale perf numbers in this cover with fresh, reproducible measurements (Catalin Marinas, Marc Zyngier).

Changes since v3
================

- New sdei_is_present() patch; the NMI initcall now skips registration (and its boot warning) on non-SDEI systems (Puranjay Mohan).
- Fixed a NULL deref on a parallel-panic crash stop and the CONFIG_KEXEC_CORE=n build (Puranjay Mohan).
- kernel-doc + barrier comments on the stop path; reordered the two arm_sdei core patches (Doug Anderson).

Changes since v2
================

- Unified the CPU-stop paths into one arm64_nmi_cpu_stop(regs, die_on_crash), dropping local_cpu_stop()/ipi_cpu_crash_stop().
- SDEI rung tests sdei_nmi_active() first; sdei_nmi_stop_cpus() is void.
- Replaced the per-CPU stop cpumask with a write-once flag.
- Commented the SDEI-park / no-CPU_OFF rationale.

Changes since v1
================

- Dropped the SDEI hard-lockup-detector patch; use the buddy detector.
- Reworked crash-stop into a third rung of smp_send_stop().
- Renamed the driver to arm_sdei_nmi.c; widened the MAINTAINERS glob.

v4: https://lore.kernel.org/all/[email protected]
v3: https://lore.kernel.org/all/[email protected]
v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]

Also available at:

git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git sdei-nmi/v5

Kiryl Shutsemau (Meta) (4):
firmware: arm_sdei: add sdei_is_present()
firmware: arm_sdei: add SDEI_EVENT_SIGNAL support drivers/firmware: add SDEI cross-CPU NMI service for arm64
arm64: escalate smp_send_stop() to an SDEI NMI as a last resort

 MAINTAINERS                     |   2 +-
 arch/arm64/include/asm/nmi.h    |  48 +++++++
 arch/arm64/kernel/smp.c         | 124 +++++++++++-----
 drivers/firmware/Kconfig        |  21 +++
 drivers/firmware/Makefile       |   1 +
 drivers/firmware/arm_sdei.c     |  22 +++
 drivers/firmware/arm_sdei_nmi.c | 246 ++++++++++++++++++++++++++++++++
 include/linux/arm_sdei.h        |   9 ++
 include/uapi/linux/arm_sdei.h   |   1 +
 9 files changed, 435 insertions(+), 39 deletions(-)
 create mode 100644 arch/arm64/include/asm/nmi.h
 create mode 100644 drivers/firmware/arm_sdei_nmi.c

base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6 --
2.54.0