Skip to contentSkip to conversation

panic: a pre kdump notifier list for hypervisor upcalls

RFC PATCH In review Bradley Morgan 2 participants 7 messages Jul 11 – Jul 17

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover panic: a pre kdump notifier list for hypervisor upcalls

When a crash kernel is loaded, panic() jumps to it before the panic notifiers run, unless crash_kexec_post_notifiers is set. So the hypervisor or firmware never finds out the guest panicked. Hyper-V doesn't get the crash registers, gsmi drops its firmware log entry, pvpanic stays silent, SEV-SNP skips its firmware and IOMMU shutdown. Whatever's watching the machine, the host, the BMC, fleet, sees a clean reboot instead of a crash.

The only way around it today is crash_kexec_post_notifiers, but that runs the whole legacy notifier list before the kdump. That list has slow callbacks in it, IPMI being the obvious one since it talks to a BMC, so turning it on slows down every crash dump on the box. Hyper-V turns it on anyway and eats the cost for one callback. SNP did the same.

This adds a separate list that runs before the crash kexec no matter what. Callbacks on it have to follow a contract: no locks, no allocation, no sleeping, other CPUs may still be running, and it has to tolerate being entered again if the panic path itself panics. The list head is private and you register through a function, because that's how you make people follow the contract. Priorities order it like any notifier chain. It lives in its own file because it's not panic code, just a notifier chain vpanic() calls once, and the custom walk it needs doesn't belong in the generic notifier API either. A separate file is also the only thing a MAINTAINERS entry can scope a reviewer to.

Running anything before the crash kexec is a reliability trade. If an upcall hangs or faults and costs the dump, you want to know which one did it, so the notify path walks the chain itself and prints each callback's name on the console before calling it. Return values are ignored. A nested panic picks up the walk after the callback that was running, instead of re-faulting on it.

This is the minimum: the list, the hook in the panic path, one driver converted so there's a real user, and the MAINTAINERS entry. pvpanic is the first one. Its callback is a self contained upcall that already takes its lock as a trylock, so it fits the contract as is. The rest (Hyper-V, Xen, gsmi, SNP) come in a follow on series. gsmi needs a rework to a trylock because its old deadlock guard assumed the other CPUs were stopped, which isn't true on this list. Once SNP's notifier is on the new list, the crash_kexec_post_notifiers forcing in snp_rmptable_init() goes away and SNP gets the early crash kexec back. Hyper-V keeps its forcing for now because its kmsg dump pass also needs to run before kdump. The register report just doesn't depend on it anymore.

IPMI stays out. Its panic handling assumes the other CPUs have been stopped, and poking a BMC before the crash kexec would slow down every kdump on any box with a BMC. It stays on the legacy list, where kdump skips it like before.

The legacy list, its position in the panic path and crash_kexec_post_notifiers itself are untouched, so the remaining registrants see zero change and nothing is renamed.

This is on purpose. Piccoli's 2022 series tried to classify every panic notifier and the list split didn't go in. This does the one piece that fixes the kdump versus hypervisor conflict and stops.

One alternative is splitting the legacy list by priority and calling the high priority half before the crash kexec. That can't do what this list does. The walk prints each callback's name on the console before it runs, and it ignores return values. atomic_notifier_call_chain can't do either. It honors NOTIFY_STOP_MASK, so one callback can halt the pass and lose every upcall after it. The walk also resumes past the faulting callback on a nested panic and runs under RCU instead of the atomic spinlock. Getting this behavior under a priority split means pushing a custom walk into the generic notifier API, which is more invasive than a file.

Build tested on arm64.

Bradley Morgan (4):
panic: add a pre kdump notifier list
panic: run the pre kdump list before crash kexec misc/pvpanic: notify the host on the pre kdump list
MAINTAINERS: add an entry for the pre kdump notifier list

 Documentation/admin-guide/kernel-parameters.txt |  3 ++
 MAINTAINERS                                     |  6 +++
 drivers/misc/pvpanic/pvpanic.c                  |  5 +-
 include/linux/panic_notifier.h                  | 18 +++++++
 kernel/Makefile                                 |  2 +-
 kernel/panic.c                                  |  7 +++
 kernel/panic_notifiers.c                        | 65 +++++++++++++++++++++++++
 7 files changed, 102 insertions(+), 4 deletions(-)
 create mode 100644 kernel/panic_notifiers.c

--
2.53.0

#1 lore raw

RFC PATCH 1/4 panic: add a pre kdump notifier list

When a crash kernel is loaded, panic() jumps to it before the panic notifiers run. A hypervisor or firmware therefore never learns that the guest panicked: the Hyper-V host never receives the crash registers, a Google gsmi firmware log entry is dropped, a pvpanic device never signals the host, and an AMD SEV-SNP guest skips the firmware and IOMMU shutdown. Fleet alerting, availability tracking, and crash logging all miss the event.

The only escape hatch today is crash_kexec_post_notifiers, but it runs the entire legacy notifier list before the kdump. That list carries slow callbacks, IPMI being the obvious one because it talks to a BMC, so enabling it delays every crash dump on the machine. Hyper-V forces it on anyway and penalizes every kdump for the sake of one upcall; this series also removes the forcing that SEV-SNP added for the same reason, giving SNP hosts the normal early crash kexec back.

Add a small dedicated list that always runs before crash kexec. The list head stays private and registration goes through a function, because the callbacks on this list have a strict contract and an API is how you enforce one. The list lives in its own file, not in panic.c, so the contract can carry a MAINTAINERS entry and the existing panic code does not have to move.

Priorities order the list like any atomic notifier chain. The header documents that they are reserved for callbacks that must run last, for example because they may not return or because they tear down a transport that other callbacks still need.

The notify path walks the chain by hand and names each callback on the console before invoking it. Running anything before the crash kexec is a reliability trade, if an upcall hangs or faults and costs us the crash dump, the console identifies the culprit instead of leaving a machine that silently never rebooted into the kdump kernel. Callback return values are deliberately ignored, and a nested panic resumes the walk after the callback that was running instead of refaulting on it.

---
 include/linux/panic_notifier.h | 18 ++++++++++
 kernel/Makefile                |  2 +-
 kernel/panic_notifiers.c       | 65 ++++++++++++++++++++++++++++++++++
 3 files changed, 84 insertions(+), 1 deletion(-)
 create mode 100644 kernel/panic_notifiers.c
include/linux/panic_notifier.h +18 -0
index 41e32483d7a7..1e99de3a8a6a 100644
@@ -7,6 +7,24 @@
 
 extern struct atomic_notifier_head panic_notifier_list;
 
+/*
+ * The pre kdump list runs on every panic, before a potential crash
+ * kexec, so a hypervisor or firmware learns of the panic even when a
+ * crash kernel is loaded. A callback:
+ *
+ *  - must not take locks, allocate memory or sleep
+ *  - must not assume other CPUs have been stopped
+ *  - must tolerate being entered again if the panic path panics
+ *
+ * One that cannot belongs on the legacy panic_notifier_list instead.
+ * Reserve low priorities for callbacks that must run last, for
+ * example because they do not return. Return values are ignored: the
+ * whole list always runs.
+ */
+int panic_notifier_register_pre_kdump(struct notifier_block *nb);
+int panic_notifier_unregister_pre_kdump(struct notifier_block *nb);
+void panic_pre_kdump_notify(const char *buf);
+
 extern bool crash_kexec_post_notifiers;
 
 #endif	/* _LINUX_PANIC_NOTIFIERS_H */
kernel/Makefile +1 -1
index 1e1a31673577..7add98a5acc2 100644
@@ -3,7 +3,7 @@
 # Makefile for the linux kernel.
 #
 
-obj-y     = fork.o exec_domain.o exec_state.o panic.o \
+obj-y     = fork.o exec_domain.o exec_state.o panic.o panic_notifiers.o \
 	    cpu.o exit.o softirq.o resource.o \
 	    sysctl.o capability.o ptrace.o user.o \
 	    signal.o sys.o umh.o workqueue.o pid.o task_work.o \
kernel/panic_notifiers.c new file +65 -0

65 lines not on this page · raw message

Signature · 1 line
2.53.0
#2 lore raw

Re: [RFC PATCH 1/4] panic: add a pre kdump notifier list

Quoted 28 lines · Bradley Morgan

When a crash kernel is loaded, panic() jumps to it before the panic notifiers run. A hypervisor or firmware therefore never learns that the guest panicked: the Hyper-V host never receives the crash registers, a Google gsmi firmware log entry is dropped, a pvpanic device never signals the host, and an AMD SEV-SNP guest skips the firmware and IOMMU shutdown. Fleet alerting, availability tracking, and crash logging all miss the event.

The only escape hatch today is crash_kexec_post_notifiers, but it runs the entire legacy notifier list before the kdump. That list carries slow callbacks, IPMI being the obvious one because it talks to a BMC, so enabling it delays every crash dump on the machine. Hyper-V forces it on anyway and penalizes every kdump for the sake of one upcall; this series also removes the forcing that SEV-SNP added for the same reason, giving SNP hosts the normal early crash kexec back.

Add a small dedicated list that always runs before crash kexec. The list head stays private and registration goes through a function, because the callbacks on this list have a strict contract and an API is how you enforce one. The list lives in its own file, not in panic.c, so the contract can carry a MAINTAINERS entry and the existing panic code does not have to move.

Priorities order the list like any atomic notifier chain. The header documents that they are reserved for callbacks that must run last, for example because they may not return or because they tear down a transport that other callbacks still need.

Quoted 7 lines

The notify path walks the chain by hand and names each callback on the console before invoking it. Running anything before the crash kexec is a reliability trade, if an upcall hangs or faults and costs us the crash dump, the console identifies the culprit instead of leaving a machine that silently never rebooted into the kdump kernel. Callback return values are deliberately ignored, and a nested panic resumes the walk after the callback that was running instead of refaulting on it.

This is a long blob of text. I refuse to find the details there. I miss a clear summary how this list is different from atomic_notifier_chain and why we need it.

My understanting is that there two "main" differences. The new list:

+ ignores errors. But this might be achieved by requesting that panic() notifiers should always return 0.

+ skips already proceed notifiers during a nested panic. But this is a rather theoretical corner case.

IMHO, it not worth the effort.

Did I miss anything, please?

Best Regards,
Petr

#3 lore raw

RFC PATCH 2/4 panic: run the pre kdump list before crash kexec

In reply to Bradley Morgan

Call the new list unconditionally in vpanic, right before the early crash kexec. The legacy list, crash_kexec_post_notifiers and the cpu shutdown sequence are untouched. Also note in the kernel parameters entry that the option has no effect on the pre kdump list.

---
 Documentation/admin-guide/kernel-parameters.txt | 3 +++
 kernel/panic.c                                  | 7 +++++++
 2 files changed, 10 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +3 -0
index b5493a7f8f22..a77ecf3bd564 100644
@@ -1042,6 +1042,9 @@ Kernel parameters
 			or extra data dumped by panic_print. Note that some
 			configurations enable this option unconditionally,
 			like Hyper-V, PowerPC (fadump) and AMD SEV-SNP.
+			This option does not affect the pre kdump notifier
+			list (see include/linux/panic_notifier.h), which
+			always runs before the jump to the kdump kernel.
 
 	crashkernel=size[KMG][@offset[KMG]]
 			[KNL,EARLY] Using kexec, Linux can switch to a 'crash kernel'
kernel/panic.c +7 -0
index 94ce7a94f118..f9ad0250da67 100644
@@ -664,6 +664,13 @@ void vpanic(const char *fmt, va_list args)
 	 */
 	kgdb_panic(buf);
 
+	/*
+	 * Tell the hypervisor or firmware about the panic. This runs on
+	 * every panic, before a potential crash kexec, so the upcall is
+	 * not lost when a crash kernel is loaded.
+	 */
+	panic_pre_kdump_notify(buf);
+
 	/*
 	 * If we have crashed and we have a crash kernel loaded let it handle
 	 * everything else.
Signature · 1 line
2.53.0
#4 lore raw

RFC PATCH 3/4 misc/pvpanic: notify the host on the pre kdump list

In reply to Bradley Morgan

Writing the panic event to the pvpanic device is a self contained upcall, and pvpanic_send_event() already takes pvpanic_lock with a trylock, so the callback satisfies the contract as is.

The INT_MAX priority is kept so the host hears about the panic before anything else on the list runs.

---
 drivers/misc/pvpanic/pvpanic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/misc/pvpanic/pvpanic.c +2 -3
index 17c0eb549463..c409968228e4 100644
@@ -218,7 +218,7 @@ static int pvpanic_init(void)
 	INIT_LIST_HEAD(&pvpanic_list);
 	spin_lock_init(&pvpanic_lock);
 
-	atomic_notifier_chain_register(&panic_notifier_list, &pvpanic_panic_nb);
+	panic_notifier_register_pre_kdump(&pvpanic_panic_nb);
 
 	return 0;
 }
@@ -226,7 +226,6 @@ module_init(pvpanic_init);
 
 static void pvpanic_exit(void)
 {
-	atomic_notifier_chain_unregister(&panic_notifier_list, &pvpanic_panic_nb);
-
+	panic_notifier_unregister_pre_kdump(&pvpanic_panic_nb);
 }
 module_exit(pvpanic_exit);
Signature · 1 line
2.53.0
#5 lore raw

RFC PATCH 4/4 MAINTAINERS: add an entry for the pre kdump notifier list

In reply to Bradley Morgan

Scoped to the new file and the header so patches touching the callback contract get an extra reviewer.

---
 MAINTAINERS | 6 ++++++
 1 file changed, 6 insertions(+)
MAINTAINERS +6 -0
index 895a87b571c3..44b6318ed692 100644
@@ -20453,6 +20453,12 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/platform/x86/panasonic-laptop.c
 
+PANIC PRE KDUMP NOTIFIER LIST
+M:	Bradley Morgan <[email protected]>
+S:	Maintained
+F:	include/linux/panic_notifier.h
+F:	kernel/panic_notifiers.c
+
 PARALLAX PING IIO SENSOR DRIVER
 M:	Andreas Klinger <[email protected]>
 L:	[email protected]
Signature · 1 line
2.53.0
#6 lore raw
In reply to Bradley Morgan
Quoted 8 lines · Bradley Morgan

When a crash kernel is loaded, panic() jumps to it before the panic notifiers run, unless crash_kexec_post_notifiers is set. So the hypervisor or firmware never finds out the guest panicked. Hyper-V doesn't get the crash registers, gsmi drops its firmware log entry, pvpanic stays silent, SEV-SNP skips its firmware and IOMMU shutdown. Whatever's watching the machine, the host, the BMC, fleet, sees a clean reboot instead of a crash.

Quoted 6 lines

The only way around it today is crash_kexec_post_notifiers, but that runs the whole legacy notifier list before the kdump. That list has slow callbacks in it, IPMI being the obvious one since it talks to a BMC, so turning it on slows down every crash dump on the box. Hyper-V turns it on anyway and eats the cost for one callback. SNP did the same.

IMHO, it is not about speed at all. The main criteria are:

+ what has to be done at which stage
+ reliability

I see that there are 4 situations where crash_kexec_post_notifiers is set to true:

$> git grep "crash_kexec_post_notifiers = true"

arch/powerpc/kernel/fadump.c:   crash_kexec_post_notifiers = true;
arch/x86/hyperv/hv_crash.c:     crash_kexec_post_notifiers = true;
arch/x86/virt/svm/sev.c:        crash_kexec_post_notifiers = true;
drivers/hv/hv_common.c:         crash_kexec_post_notifiers = true;

IMHO, these point to the notifiers have to be called before kdump because otherwise something goes wrong.

The rest are users who do not care. They have happily worked as post-kdump notifiers for years.

Quoted 3 lines

This adds a separate list that runs before the crash kexec no matter what. Callbacks on it have to follow a contract: no locks, no allocation, no sleeping,

This is required by any code called in panic(). It is not special to the pre-kdump notifiers.

Quoted 1 line

other CPUs may still be running

Good question. My upderstanding is that people prefer when kdump catches the system when all CPUs are still running. But it also complicates any lockless solution in the notifiers.

Quoted 2 lines

and it has
to tolerate being entered again if the panic path itself panics.

The panic-in-panic is a dark corner for me. I believe that it might happen but I have never met it. And any panic() code should do its best to avoid it in the first place.

By other words, panic-in-panic is a corner case. We should not focus on it too much.

Quoted 15 lines

This is the minimum: the list, the hook in the panic path, one driver converted so there's a real user, and the MAINTAINERS entry. pvpanic is the first one. Its callback is a self contained upcall that already takes its lock as a trylock, so it fits the contract as is.

The rest
(Hyper-V, Xen, gsmi, SNP) come in a follow on series. gsmi needs a rework to a trylock because its old deadlock guard assumed the other CPUs were stopped, which isn't true on this list. Once SNP's notifier is on the new list, the crash_kexec_post_notifiers forcing in snp_rmptable_init() goes away and SNP gets the early crash kexec back. Hyper-V keeps its forcing for now because its kmsg dump pass also needs to run before kdump. The register report just doesn't depend on it anymore.

Quoted 4 lines

IPMI stays out. Its panic handling assumes the other CPUs have been stopped, and poking a BMC before the crash kexec would slow down every kdump on any box with a BMC. It stays on the legacy list, where kdump skips it like before.

I am not sure if I got it correctly.

IMHO, the ultimate plan should be to remove the "crash_kexec_post_notifiers" option. It is a black magic. Someone has to set it because otherwise crashdump does not work. Others disable it because it breaks or might break crashdump.

Alternative solution would be to move the potentially dangerous notifiers to some optional list, aka, panic_extra_debug_notifiers.

Quoted 7 lines

The legacy list, its position in the panic path and crash_kexec_post_notifiers itself are untouched, so the remaining registrants see zero change and nothing is renamed.

This is on purpose. Piccoli's 2022 series tried to classify every panic notifier and the list split didn't go in. This does the one piece that fixes the kdump versus hypervisor conflict and stops.

I am fine with moving one notifier by one. But we first need to make sure that the split, ordering, and naming makes sense. The good thing about Piccoli's patchset was that we saw the whole picture. And I believe that we need the whole picture to make a reasonable move here.

Quoted 2 lines

One alternative is splitting the legacy list by priority and calling the high priority half before the crash kexec.

I personally prefer two or more notifier lists. Different lists make it clear that they are proceed at different stages. The names might even help to decide which list is the right one. The priority is much harder to maintain.

Best Regards,
Petr