arm64: cross-CPU NMI via SDEI

From: "Kiryl Shutsemau (Meta)" <[email protected]>

A class of debug/observability features needs to interrupt a CPU that has its interrupts locally masked: the all-CPU backtrace behind sysrq-l / RCU-stall / hung-task / hard-lockup dumps, and crash_smp_send_stop() capturing a stuck CPU's state into the vmcore. On arm64 these need a mechanism that reaches a CPU spinning with DAIF masked, which a normal IPI cannot.

arm64 has two such mechanisms today:

- GICv3 pseudo-NMI (interrupt priority masking). The cost lands on the interrupt mask/unmask hot path: local_irq_enable() becomes an ICC_PMR_EL1 write, and exception entry/exit save and restore the PMR, paid on every CPU whether or not an NMI is ever delivered.

Measured on Grace (Neoverse V2; ICC_CTLR_EL1.PMHE=0, so the PMR-sync DSB is already patched to a NOP), pseudo_nmi=0 vs pseudo_nmi=1:

gettid() loop: 178 -> 253 ns/call (+42%, ~74 ns)
will-it-scale sched_yield: 0.705x throughput, flat from 1 to 72 cores will-it-scale page_fault1: within ~5%

The ~74 ns is a fixed per-syscall entry/exit tax -- it reproduces at +73.5 ns on Neoverse N2 -- so the hit tracks syscall/exception density and is unacceptable on syscall-bound fleet workloads, which therefore run with pseudo-NMI disabled.

- FEAT_NMI (Armv8.8) -- the architectural fix, but absent from deployed silicon and from most of the fleet for years to come.

For deployments that do not run pseudo-NMI, the backtrace and crash paths are degraded: a plain IPI can't reach the masked CPU, so the backtrace of the CPU you care about comes back empty and the kdump is missing the culprit's registers. The hard-lockup detector on these systems is the software buddy detector (HARDLOCKUP_DETECTOR_BUDDY): it detects a stall from a neighbour CPU, but it cannot itself interrupt the wedged CPU, so its report has no stack for the culprit and (with hardlockup_panic) the panic runs on the bystander.

This series adds a third delivery backend that costs nothing on the hot
path: SDEI. Firmware delivers an SDEI event into a CPU regardless of its DAIF state, so interrupt masking stays the cheap PSTATE.DAIF operation and the firmware round-trip is paid only at the rare moment a CPU must be interrupted.

It does not add a hard-lockup detector. Detection stays with the buddy detector (CONFIG_HARDLOCKUP_DETECTOR_PREFER_BUDDY); this series gives the backtrace and crash-stop paths -- including the buddy detector's backtrace of the stalled CPU -- a way to actually reach a masked CPU.

Mechanism
=========

It uses the standard SDEI software-signalled event (event 0) and the SDEI_EVENT_SIGNAL call (DEN0054) -- a spec-defined cross-PE signal, not a vendor extension. The driver registers a handler for event 0 and pokes a target CPU with sdei_event_signal(0, target_mpidr); firmware makes event 0 pending on that PE and dispatches the handler NMI-like.

No firmware change is required beyond SDEI being enabled, which firmware-first RAS (APEI/GHES) deployments already have; the only SDEI-core addition is a thin sdei_event_signal() wrapper over the standard call.

Prior SDEI watchdog work
========================

Out-of-tree SDEI hard-lockup watchdogs exist (e.g. in the openEuler and Anolis kernels). They bind the secure physical timer as an SDEI event, so firmware delivers a periodic self-CPU tick that drives a detector. That requires a new SDEI interrupt-binding API, pushes the watchdog period into firmware, and adds secure-timer EOI handling on the kexec path. This series instead uses only the standard software-signalled event 0, keeps all timing in the kernel (the buddy detector), and the same delivery primitive serves the backtrace and crash-stop users, not just lockup reporting.

Not included / follow-ups
=========================

- No SDEI hard-lockup-detector backend. v1 had one; it is dropped here. The buddy detector plus this series' backtrace already cover the no-pseudo-NMI case, and a dedicated SDEI backend duplicated the perf-NMI detector it had to compile-exclude. Run PREFER_BUDDY.

- A CPU stopped by the SDEI rung is parked, not powered off via PSCI CPU_OFF. Reaching and dumping the wedged CPU -- the point of the series -- works, and it matches the shared stop path's own park fallback when CPU_OFF is unavailable. The consequence is that an SMP crash-capture kernel cannot re-online such a CPU (it stays "already on"); the capture kernel boots and runs on the remaining CPUs. Powering the stopped CPU off so a capture kernel can reclaim it would need CPU_OFF from the SDEI stop context, which does not work in practice (see the v4 discussion); left as a follow-up and does not affect the dump's contents.

Testing
=======

Developed on QEMU 'virt' (Trusted Firmware-A with SDEI enabled) and validated on NVIDIA Grace (Neoverse V2) hardware, under irqchip.gicv3_pseudo_nmi=0 with HARDLOCKUP_DETECTOR_PREFER_BUDDY=y:

- sysrq-l backtrace of an interrupt-masked CPU returns its real stack,

    pstate showing DAIF set -- proof SDEI delivered into the masked CPU;
  - buddy detector catches a hard lockup (LKDTM) and the wedged CPU's
    stack is fetched via the SDEI backtrace;

- reboot/halt and the panic/kdump crash stop reach a wedged CPU via the SDEI rung ("SMP: retry stop with SDEI NMI for CPUs N"), and the kdump captures the wedged CPU's registers in the vmcore;
- with SDEI absent (plain QEMU 'virt', no firmware support) the driver stays inert: no event registration and no boot-time warning.

Changes since v4
================

- Strengthen the publish barrier from smp_wmb() to dsb(ishst) on the stop path, and add the missing one on the backtrace path; the SMC is not a memory store, so ordering alone is not enough (Catalin Marinas).
- Drop the redundant ARM64 Kconfig dependency, already implied by ARM_SDE_INTERFACE (Julian Braha).
- Reviewed-by from Douglas Anderson now on all four patches.
- Replaced the stale perf numbers in this cover with fresh, reproducible measurements (Catalin Marinas, Marc Zyngier).

Changes since v3
================

- New sdei_is_present() patch; the NMI initcall now skips registration (and its boot warning) on non-SDEI systems (Puranjay Mohan).
- Fixed a NULL deref on a parallel-panic crash stop and the CONFIG_KEXEC_CORE=n build (Puranjay Mohan).
- kernel-doc + barrier comments on the stop path; reordered the two arm_sdei core patches (Doug Anderson).

Changes since v2
================

- Unified the CPU-stop paths into one arm64_nmi_cpu_stop(regs, die_on_crash), dropping local_cpu_stop()/ipi_cpu_crash_stop().
- SDEI rung tests sdei_nmi_active() first; sdei_nmi_stop_cpus() is void.
- Replaced the per-CPU stop cpumask with a write-once flag.
- Commented the SDEI-park / no-CPU_OFF rationale.

Changes since v1
================

- Dropped the SDEI hard-lockup-detector patch; use the buddy detector.
- Reworked crash-stop into a third rung of smp_send_stop().
- Renamed the driver to arm_sdei_nmi.c; widened the MAINTAINERS glob.

v4: https://lore.kernel.org/all/[email protected]
v3: https://lore.kernel.org/all/[email protected]
v2: https://lore.kernel.org/all/[email protected]
v1: https://lore.kernel.org/all/[email protected]

Also available at:

git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git sdei-nmi/v5

Kiryl Shutsemau (Meta) (4):
firmware: arm_sdei: add sdei_is_present()
firmware: arm_sdei: add SDEI_EVENT_SIGNAL support drivers/firmware: add SDEI cross-CPU NMI service for arm64
arm64: escalate smp_send_stop() to an SDEI NMI as a last resort

 MAINTAINERS                     |   2 +-
 arch/arm64/include/asm/nmi.h    |  48 +++++++
 arch/arm64/kernel/smp.c         | 124 +++++++++++-----
 drivers/firmware/Kconfig        |  21 +++
 drivers/firmware/Makefile       |   1 +
 drivers/firmware/arm_sdei.c     |  22 +++
 drivers/firmware/arm_sdei_nmi.c | 246 ++++++++++++++++++++++++++++++++
 include/linux/arm_sdei.h        |   9 ++
 include/uapi/linux/arm_sdei.h   |   1 +
 9 files changed, 435 insertions(+), 39 deletions(-)
 create mode 100644 arch/arm64/include/asm/nmi.h
 create mode 100644 drivers/firmware/arm_sdei_nmi.c

base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6 --
2.54.0

From: "Kiryl Shutsemau (Meta)" <[email protected]>

invoke_sdei_fn() returns -EIO when no SDEI conduit was probed, and the core warns ("Failed to create event ...") on any registration that hits that. An optional consumer that registers an event from an unconditional initcall would therefore make every boot on a non-SDEI system emit that warning for what is simply absent firmware.

Expose whether SDEI firmware is present so such a consumer can skip registration -- and the warning -- when there is nothing to talk to.

---
 drivers/firmware/arm_sdei.c | 10 ++++++++++
 include/linux/arm_sdei.h    |  3 +++
 2 files changed, 13 insertions(+)

index f39ed7ba3a38..c161cf263547 100644
@@ -339,6 +339,16 @@ static void _ipi_unmask_cpu(void *ignored)
 	sdei_unmask_local_cpu();
 }
 
+/*
+ * Was SDEI firmware probed and is it usable?  Lets optional consumers skip
+ * registering an event -- and the warning a failed registration emits -- on
+ * systems with no SDEI.
+ */
+bool sdei_is_present(void)
+{
+	return sdei_firmware_call;
+}
+
 static void _ipi_private_reset(void *ignored)
 {
 	int err;

index f652a5028b59..b07113eeeff7 100644
@@ -37,6 +37,9 @@ int sdei_event_unregister(u32 event_num);
 int sdei_event_enable(u32 event_num);
 int sdei_event_disable(u32 event_num);
 
+/* Was SDEI firmware probed and usable? */
+bool sdei_is_present(void);
+
 /* GHES register/unregister helpers */
 int sdei_register_ghes(struct ghes *ghes, sdei_event_callback *normal_cb,
 		       sdei_event_callback *critical_cb);

2.54.0

Can this be merged in patch 3 where this function is actually used?

I can do it either way. Standalone commit looks good to me.

But if maintainer wants, I can fold it.

Again, I don't care much, but checkpatch doesn't like comparison to NULL.

  Kiryl Shutsemau / Kirill A. Shutemov

From: "Kiryl Shutsemau (Meta)" <[email protected]>

Add sdei_event_signal(), a thin wrapper over the SDEI_EVENT_SIGNAL call (DEN0054) that makes the software-signalled event (event 0) pending on a target PE -- delivered NMI-like even when that PE has interrupts masked. It takes no locks, so it is safe to call from NMI / crash context.

---
 drivers/firmware/arm_sdei.c   | 12 ++++++++++++
 include/linux/arm_sdei.h      |  6 ++++++
 include/uapi/linux/arm_sdei.h |  1 +
 3 files changed, 19 insertions(+)

index c161cf263547..e8dd2f0f3919 100644
@@ -339,6 +339,18 @@ static void _ipi_unmask_cpu(void *ignored)
 	sdei_unmask_local_cpu();
 }
 
+/*
+ * Signal the software-signalled event (event 0) to @mpidr. Does nothing
+ * but the SMC -- no locks, no event lookup -- so it is safe from NMI /
+ * crash context (e.g. the cross-CPU NMI service).
+ */
+int sdei_event_signal(u32 event_num, u64 mpidr)
+{
+	return invoke_sdei_fn(SDEI_1_0_FN_SDEI_EVENT_SIGNAL, event_num,
+			      mpidr, 0, 0, 0, NULL);
+}
+NOKPROBE_SYMBOL(sdei_event_signal);
+
 /*
  * Was SDEI firmware probed and is it usable?  Lets optional consumers skip
  * registering an event -- and the warning a failed registration emits -- on

index b07113eeeff7..b9dc21c241be 100644
@@ -37,6 +37,12 @@ int sdei_event_unregister(u32 event_num);
 int sdei_event_enable(u32 event_num);
 int sdei_event_disable(u32 event_num);
 
+/*
+ * Signal the software-signalled event (event 0) to another PE, NMI-like.
+ * @mpidr is the target's MPIDR affinity.
+ */
+int sdei_event_signal(u32 event_num, u64 mpidr);
+
 /* Was SDEI firmware probed and usable? */
 bool sdei_is_present(void);
 

index af0630ba5437..22eb61612673 100644
@@ -22,6 +22,7 @@
 #define SDEI_1_0_FN_SDEI_PE_UNMASK			SDEI_1_0_FN(0x0C)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_BIND			SDEI_1_0_FN(0x0D)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE		SDEI_1_0_FN(0x0E)
+#define SDEI_1_0_FN_SDEI_EVENT_SIGNAL			SDEI_1_0_FN(0x0F)
 #define SDEI_1_0_FN_SDEI_PRIVATE_RESET			SDEI_1_0_FN(0x11)
 #define SDEI_1_0_FN_SDEI_SHARED_RESET			SDEI_1_0_FN(0x12)
 

2.54.0

Patch 3 is big as it is. I don't think folding code that can be a standalone patch helps the situation.

  Kiryl Shutsemau / Kirill A. Shutemov

From: "Kiryl Shutsemau (Meta)" <[email protected]>

Deliver an NMI-like event to an interrupt-masked arm64 CPU via the standard SDEI software-signalled event (event 0), without the pseudo-NMI hot-path cost: register a handler for event 0 and poke a target with sdei_event_signal(0, mpidr).

First user is arch_trigger_cpumask_backtrace() (sysrq-l, RCU stalls, hung-task/soft-lockup dumps), which otherwise rides an IPI that can't reach a masked CPU. Falls back to the IPI path when SDEI is absent; no watchdog backend yet, so the stock detector is untouched.

---
 MAINTAINERS                     |   2 +-
 arch/arm64/include/asm/nmi.h    |  24 +++++
 arch/arm64/kernel/smp.c         |  11 +++
 drivers/firmware/Kconfig        |  19 ++++
 drivers/firmware/Makefile       |   1 +
 drivers/firmware/arm_sdei_nmi.c | 159 ++++++++++++++++++++++++++++++++
 6 files changed, 215 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/nmi.h
 create mode 100644 drivers/firmware/arm_sdei_nmi.c

index c8d4b913f26c..b5ddfb85dce9 100644
@@ -24797,7 +24797,7 @@ M:	James Morse <[email protected]>
 L:	[email protected] (moderated for non-subscribers)
 S:	Maintained
 F:	Documentation/devicetree/bindings/arm/firmware/sdei.txt
-F:	drivers/firmware/arm_sdei.c
+F:	drivers/firmware/arm_sdei*
 F:	include/linux/arm_sdei.h
 F:	include/uapi/linux/arm_sdei.h
 

new file mode 100644
index 000000000000..9366be419d18
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_NMI_H
+#define __ASM_NMI_H
+
+#include <linux/cpumask.h>
+
+/*
+ * Cross-CPU NMI provider hooks, consulted by the arm64 arch code before
+ * its regular-IRQ / pseudo-NMI IPI paths. The SDEI provider in
+ * drivers/firmware/arm_sdei_nmi.c implements them when active; a future
+ * FEAT_NMI provider could slot in here too. The stubs let callers stay
+ * unconditional when ARM_SDEI_NMI is off.
+ */
+#ifdef CONFIG_ARM_SDEI_NMI
+bool sdei_nmi_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu);
+#else
+static inline bool sdei_nmi_trigger_cpumask_backtrace(const cpumask_t *mask,
+						      int exclude_cpu)
+{
+	return false;
+}
+#endif
+
+#endif /* __ASM_NMI_H */

index 1aa324104afb..a670434a8cae 100644
@@ -45,6 +45,7 @@
 #include <asm/daifflags.h>
 #include <asm/kvm_mmu.h>
 #include <asm/mmu_context.h>
+#include <asm/nmi.h>
 #include <asm/numa.h>
 #include <asm/processor.h>
 #include <asm/smp_plat.h>

@@ -927,6 +928,16 @@ static void arm64_backtrace_ipi(cpumask_t *mask)
 
 void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
 {
+	/*
+	 * Prefer the SDEI cross-CPU NMI provider when active: firmware
+	 * dispatches the event out of EL3 and reaches CPUs that have
+	 * interrupts locally masked, without the per-IRQ-mask cost that
+	 * pseudo-NMI pays for the same reach. The plain IPI path below
+	 * can't reach such a CPU unless pseudo-NMI is enabled.
+	 */
+	if (sdei_nmi_trigger_cpumask_backtrace(mask, exclude_cpu))
+		return;
+
 	/*
 	 * NOTE: though nmi_trigger_cpumask_backtrace() has "nmi_" in the name,
 	 * nothing about it truly needs to be implemented using an NMI, it's

index bbd2155d8483..dd47584d3b57 100644
@@ -36,6 +36,25 @@ config ARM_SDE_INTERFACE
 	  standard for registering callbacks from the platform firmware
 	  into the OS. This is typically used to implement RAS notifications.
 
+config ARM_SDEI_NMI
+	bool "SDEI-based cross-CPU NMI service (arm64)"
+	depends on ARM_SDE_INTERFACE
+	help
+	  Provides SDEI-based cross-CPU NMI delivery for hooks that need
+	  to reach interrupt-masked CPUs on silicon that lacks FEAT_NMI:
+
+	    - arch_trigger_cpumask_backtrace()  (sysrq-l, RCU stalls,
+	      hardlockup_all_cpu_backtrace, soft-lockup secondary dumps,
+	      hung-task auxiliary dumps)
+
+	  The driver registers a handler for the SDEI software-signalled
+	  event (event 0) and reaches a target CPU by signalling it with
+	  SDEI_EVENT_SIGNAL. Firmware delivers the event out of EL3
+	  regardless of the target's PSTATE.DAIF -- forced delivery into a
+	  CPU wedged with interrupts locally masked.
+
+	  If unsure, say N.
+
 config EDD
 	tristate "BIOS Enhanced Disk Drive calls determine boot disk"
 	depends on X86

index 4ddec2820c96..be46f1e1dc77 100644
@@ -4,6 +4,7 @@
 #
 obj-$(CONFIG_ARM_SCPI_PROTOCOL)	+= arm_scpi.o
 obj-$(CONFIG_ARM_SDE_INTERFACE)	+= arm_sdei.o
+obj-$(CONFIG_ARM_SDEI_NMI)	+= arm_sdei_nmi.o
 obj-$(CONFIG_DMI)		+= dmi_scan.o
 obj-$(CONFIG_DMI_SYSFS)		+= dmi-sysfs.o
 obj-$(CONFIG_EDD)		+= edd.o

2.54.0

From: "Kiryl Shutsemau (Meta)" <[email protected]>

A CPU wedged with interrupts masked ignores the stop IPI, and without pseudo-NMI there is no NMI IPI to escalate to: a reboot proceeds with the CPU still running, and a kdump misses its registers.

Add a third rung to smp_send_stop(): once the IPI (and pseudo-NMI IPI, if enabled) rungs have run, signal SDEI event 0 at whatever stayed online. Firmware delivers it regardless of the target's DAIF, so it reaches a CPU a plain IPI cannot; the target acks by going offline, which the caller already polls for.

Fold the stop bookkeeping into one arm64_nmi_cpu_stop(regs, die_on_crash), shared by the stop IPI handlers, panic_smp_self_stop() and the SDEI handler, replacing the near-duplicate local_cpu_stop() and ipi_cpu_crash_stop(). @die_on_crash is the only difference: the IPI handlers pass true and PSCI CPU_OFF the CPU on a crash stop so a capture kernel can reclaim it; the SDEI handler and self-stop pass false and park. The SDEI park is required, not conservative -- its handler runs inside an SDEI event that is never completed (completing it resumes the wedged context), and a CPU_OFF from that unfinished-event context wedges EL3 on some firmware (left as a follow-up). The dump is unaffected; only re-onlining the CPU in an SMP capture kernel is lost.

---
 arch/arm64/include/asm/nmi.h    |  24 +++++++
 arch/arm64/kernel/smp.c         | 113 +++++++++++++++++++++-----------
 drivers/firmware/Kconfig        |   2 +
 drivers/firmware/arm_sdei_nmi.c |  87 ++++++++++++++++++++++++
 4 files changed, 188 insertions(+), 38 deletions(-)

index 9366be419d18..2e8974ff8d63 100644
@@ -4,21 +4,45 @@
 
 #include <linux/cpumask.h>
 
+struct pt_regs;
+
 /*
  * Cross-CPU NMI provider hooks, consulted by the arm64 arch code before
  * its regular-IRQ / pseudo-NMI IPI paths. The SDEI provider in
  * drivers/firmware/arm_sdei_nmi.c implements them when active; a future
  * FEAT_NMI provider could slot in here too. The stubs let callers stay
  * unconditional when ARM_SDEI_NMI is off.
+ *
+ * sdei_nmi_active() lets a caller test for the service before committing
+ * to (and waiting on) the SDEI stop rung; sdei_nmi_stop_cpus() then signals
+ * the targets, which ack by going offline.
  */
 #ifdef CONFIG_ARM_SDEI_NMI
 bool sdei_nmi_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu);
+bool sdei_nmi_active(void);
+void sdei_nmi_stop_cpus(const cpumask_t *mask);
 #else
 static inline bool sdei_nmi_trigger_cpumask_backtrace(const cpumask_t *mask,
 						      int exclude_cpu)
 {
 	return false;
 }
+
+static inline bool sdei_nmi_active(void)
+{
+	return false;
+}
+
+static inline void sdei_nmi_stop_cpus(const cpumask_t *mask) { }
 #endif
 
+/*
+ * The common "stop this CPU" entry every arm64 stop path funnels through:
+ * the regular/pseudo-NMI stop IPI handlers, panic_smp_self_stop(), and the
+ * SDEI cross-CPU NMI handler. @die_on_crash powers the CPU off on the kdump
+ * crash path (IPI handlers) instead of parking it (SDEI / self-stop).
+ * Defined in arch/arm64/kernel/smp.c.
+ */
+void __noreturn arm64_nmi_cpu_stop(struct pt_regs *regs, bool die_on_crash);
+
 #endif /* __ASM_NMI_H */

index dd47584d3b57..b5f502bd2da3 100644
@@ -46,6 +46,8 @@ config ARM_SDEI_NMI
 	    - arch_trigger_cpumask_backtrace()  (sysrq-l, RCU stalls,
 	      hardlockup_all_cpu_backtrace, soft-lockup secondary dumps,
 	      hung-task auxiliary dumps)
+	    - smp_send_stop() escalation         (reboot/halt and the
+	      panic / kdump crash stop)
 
 	  The driver registers a handler for the SDEI software-signalled
 	  event (event 0) and reaches a target CPU by signalling it with

2.54.0

Hi Kirill,

V> arm64 has two such mechanisms today:

This patchset works perfectly on our Neoverse N2 ARM64 platform. So

Regards
Yin, Fengwei

Gentle ping. Any feedback?

I'm looking forward to finding an upstreamable solution to the problem.

  Kiryl Shutsemau / Kirill A. Shutemov

This is all pretty small, self-contained and it's useful to you, so I'm inclined to merge it.

However, the one vague concern I have is about the direction of SDEI in the future. AFAIK, the TF-A implementation is known to have issues, it's not supported at all by R-FA and I worry that the spec is going to fall behind the architecture, particularly as FEAT_NMI becomes available.

So it would be good to understand what we're signing up to maintain here. Kiryl, do you see an (eventual) migration over to FEAT_NMI, giving us a path to deprecating SDEI altogether, or do you think the two will live alongside each other for the forseeable future?

Cheers,

Will

Thanks!

Realistically, the foreseeable future.

FEAT_NMI is the endgame and a machine that has it needs none of this, but non-FEAT_NMI parts stay in fleets for many years, so I can't honestly give you a date for dropping SDEI.

While this change is pretty self-contained, I understand that SDEI infra maintenance can be a burden.

Is there anything on the wider SDEI infrastructure that would lower the maintenance burden or keep it more out of your way -- consolidation, tighter isolation, docs, help carrying it? If there's work there that makes SDEI more palatable to keep while non-FEAT_NMI parts are in service, I'm glad to take it on.

  Kiryl Shutsemau / Kirill A. Shutemov

That's fair, I was really just trying to understand the direction. It sounds like you will *eventually* move to FEAT_NMI, even if it's many years away.

I'm somewhat trying to predict the future here, but I wouldn't be shocked if some whizz-bang new architecture feature interacted badly with SDEI, either due to outdated specs or because of a fundamental incompatability. In this case, I would like the option of being able to make new hardware support mutually exclusive with SDEI rather than try to hack around the mess. It sounds like that would be ok for you?

Will

Right, that's the plan.

Yes, that's fine by me.

I'd prefer that exclusivity at boot rather than compile time, so we can keep a single kernel binary across the fleet. The delivery side already works this way -- the arch selects one cross-CPU-NMI provider, so it's a boot-time decision already.

The piece that isn't boot-time yet is the hardlockup detector that drives all this. perf and buddy are mutually exclusive at compile time. We want to use perf watchdog for FEAT_NMI while SDEI needs buddy.

But it is solvable.

  Kiryl Shutsemau / Kirill A. Shutemov

Hi,

FWIW: can I ask why? You're worried about the "all CPUs lock up at the same time" problem? I am aware of some work towards solving that (even with buddy) by letting the hardware watchdog driver's "bark" interrupt run at NMI-level (or pseudo-NMI-level). With that change, I'm not aware of any benefits of the "perf" detector over the buddy one. The buddy one should generally be more efficient (fewer interrupts to process) and less resource intensive (doesn't reserve a perf counter for the whole boot).

Yeah, someone posted proof-of-concept patches that made both selectable at boot time. Nobody was really interested in shepherding those patches all the way through.

-Doug

Honestly, no strong reason. I flagged perf mostly as "the detector you'd reach for once NMIs are back", not from a concrete need.

The all-CPUs-wedge-at-once case is the only gap I was vaguely worried about, since there's no live buddy left to notice it. If the NMI-level bark work you mention closes that for buddy, then I don't see a reason to want perf at all -- and as you say, buddy is cheaper.

Buddy everywhere would be strictly better for us.

  Kiryl Shutsemau / Kirill A. Shutemov

Applied to arm64 (for-next/sdei), thanks!

[1/4] firmware: arm_sdei: add sdei_is_present()

      https://git.kernel.org/arm64/c/5686f2bed02d
[2/4] firmware: arm_sdei: add SDEI_EVENT_SIGNAL support
      https://git.kernel.org/arm64/c/9141eda9eec7
[3/4] drivers/firmware: add SDEI cross-CPU NMI service for arm64
      https://git.kernel.org/arm64/c/0c2c56598cc1
[4/4] arm64: escalate smp_send_stop() to an SDEI NMI as a last resort
      https://git.kernel.org/arm64/c/d639f7626292

Cheers,

Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev