panic, x86/vmware: Preserve crash logs before kdump

This series lets VMware guests copy the end of their kernel log to the host's vmware.log when they crash, before entering kdump. This preserves diagnostic information even if the guest cannot save its own logs or the crash dump fails.

Using the existing panic notifier list for this would require crash_kexec_post_notifiers=1, which also runs unrelated callbacks before kdump. Following Petr's suggestion, v3 instead adds a separate platform panic notifier list for callbacks that need to run early. It runs before kdump and before stopping the other CPUs. It also covers fatal oopses that enter kdump without going through panic(). The callbacks run at most once, and the list can be disabled for debugging with disable_panic_notifiers_platform_list. crash_kexec_post_notifiers, the existing panic notifiers and the kmsg dumpers are unchanged.

The VMware log sender is the first user. It uses a preallocated 8 KiB buffer and limits both the transfer size and the number of retries. Logging is enabled by default in ordinary guests; encrypted guests must opt in through kernel.vmware_record_panic_msg. The copy is best-effort: kdump and reboot do not depend on it succeeding.

The separate VMware guest-crash notification, which can cause the host to terminate the VM, is a kmsg dumper registered late in boot, so it runs after the built-in log dumpers. It is suppressed while a crash kernel is loaded.

Changes since v2:
- Replace the pre-kdump list and postponement parameter with the platform list and a debugging switch to disable it.
- Run the list in vpanic() before the other CPUs are stopped, instead of just before the kmsg dumpers.
- Correct the sysctl documentation heading.
- Rebase onto v7.4-rc1.

Tested on x86-64 guests on VMware Workstation, ESXi (ordinary, SEV-SNP and TDX) and QEMU/KVM.

Platform list proposal:
https://lore.kernel.org/r/[email protected]

v2:
https://lore.kernel.org/r/[email protected]
v1:
https://lore.kernel.org/r/[email protected]

Zack Rusin (4):
panic: Add a platform panic notifier list x86/vmware: Add a bounded panic log sender x86/vmware: Add the vmware_record_panic_msg sysctl x86/vmware: Report guest crashes after kmsg dumpers

 .../admin-guide/kernel-parameters.txt         |  12 +
 Documentation/admin-guide/sysctl/kernel.rst   |  20 ++
 arch/x86/include/asm/vmware.h                 |   2 +
 arch/x86/kernel/cpu/vmware.c                  | 250 ++++++++++++++++++
 include/linux/panic_notifier.h                |   3 +
 kernel/crash_core.c                           |   3 +
 kernel/panic.c                                |  50 ++++
 7 files changed, 340 insertions(+)

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4

Some platforms set crash_kexec_post_notifiers so their crash handlers run before kdump. That also runs every other panic notifier and kmsg dumper before the crash kernel.

Add panic_notifiers_platform_list for platform callbacks that need to run early and are safe while other CPUs are still running. Call it from vpanic() before kdump and before stopping the other CPUs, independently of crash_kexec_post_notifiers.

Also call it from __crash_kexec() after saving the registers, since a fatal oops can enter kdump without panic(). Run the list at most once, including when a callback panics.

Add disable_panic_notifiers_platform_list to skip these callbacks when debugging panic or kdump failures.

---
v3: New. Replaces v2 patches 1-3 (pre-kdump list, crash-core call and panic_pre_kdump_postpone) with the platform list Petr suggested: one early call in vpanic(), one in __crash_kexec(), a shared guard and a disable switch. The guard is a plain bool, because calls are serialized, normally by panic_cpu.

 .../admin-guide/kernel-parameters.txt         | 12 +++++
 include/linux/panic_notifier.h                |  3 ++
 kernel/crash_core.c                           |  3 ++
 kernel/panic.c                                | 50 +++++++++++++++++++
 4 files changed, 68 insertions(+)

index 0b3c140babd9..85ddcad397bf 100644
@@ -1308,6 +1308,18 @@ Kernel parameters
 			MTRR settings.  This parameter disables that behavior,
 			possibly causing your machine to run very slowly.
 
+	disable_panic_notifiers_platform_list=
+			[KNL] Skip the platform panic notifiers. They
+			normally run early in panic(), before kdump, and on
+			crash paths that enter kdump without panic().
+			Intended for debugging panic and kdump failures.
+			Skipping them may lose platform crash handling,
+			such as a copy of the kernel log on the hypervisor.
+			Format: <bool>
+			Default: 0
+			Writable at runtime via
+			/sys/module/kernel/parameters/disable_panic_notifiers_platform_list.
+
 	disable_timer_pin_1 [X86,EARLY]
 			Disable PIN 1 of APIC timer
 			Can be useful to work around chipset bugs.

index 41e32483d7a7..1eaf1ffba0bc 100644
@@ -6,6 +6,9 @@
 #include <linux/types.h>
 
 extern struct atomic_notifier_head panic_notifier_list;
+extern struct atomic_notifier_head panic_notifiers_platform_list;
+
+void panic_notify_platform(void);
 
 extern bool crash_kexec_post_notifiers;
 

index d0bd2d0cf899..10b4384294c2 100644
@@ -23,6 +23,7 @@
 #include <linux/objtool.h>
 #include <linux/delay.h>
 #include <linux/panic.h>
+#include <linux/panic_notifier.h>
 
 #include <asm/page.h>
 #include <asm/sections.h>

@@ -139,6 +140,8 @@ void __noclone __crash_kexec(struct pt_regs *regs)
 			struct pt_regs fixed_regs;
 
 			crash_setup_regs(&fixed_regs, regs);
+			/* For crash paths that bypass panic() */
+			panic_notify_platform();
 			crash_save_vmcoreinfo();
 			machine_crash_shutdown(&fixed_regs);
 			crash_cma_clear_pending_dma();

A VMware guest may be unable to save its own logs after a crash. Copy the newest kernel log records to the host's vmware.log before kdump, so they remain available if the capture kernel fails.

Register the sender on the platform panic notifier list and send the log through the low-bandwidth register interface. Allocate a fixed 8 KiB buffer during early init; 4 KiB can lose useful stack-trace context. The callback allocates no memory and emits no printk messages. Limit the transfer to the buffer size and make at most three attempts, retrying only if a VM checkpoint interrupts the transfer.

Enable logging by default only in guests without memory encryption; encrypted guests should send their log to the host only when an administrator opts in. The copy is best-effort diagnostics; kdump does not depend on it succeeding.

---
v3: Register on the platform list instead of the pre-kdump list. The VMware code is otherwise unchanged from v2.

 arch/x86/include/asm/vmware.h |   1 +
 arch/x86/kernel/cpu/vmware.c  | 173 ++++++++++++++++++++++++++++++++++
 2 files changed, 174 insertions(+)

index 4220dae14a2d..598d4cd448ec 100644
@@ -57,6 +57,7 @@
 #define VMWARE_HYPERVISOR_MAGIC		0x564d5868U
 
 #define VMWARE_CMD_GETVERSION		10
+#define VMWARE_CMD_MESSAGE		30
 #define VMWARE_CMD_GETHZ		45
 #define VMWARE_CMD_GETVCPU_INFO		68
 #define VMWARE_CMD_STEALCLOCK		91

Allow administrators to disable host log transfer on ordinary guests or opt in on encrypted guests. Expose the boolean recording policy through kernel.vmware_record_panic_msg using proc_dobool(), as Joel suggested. Zero disables recording, nonzero integers enable it, and reads return 0 or 1.

Register the sysctl only after the logger is ready. A registration failure leaves the internal default in force. Document that the log is sent before kdump, and that disable_panic_notifiers_platform_list also skips it.

---
v3: Document the platform list ordering and the disable switch, and fix the short section-title underline reported by Sashiko. No code change from v2.

 Documentation/admin-guide/sysctl/kernel.rst | 20 ++++++++++++++++++++
 arch/x86/kernel/cpu/vmware.c                | 15 +++++++++++++++
 2 files changed, 35 insertions(+)

index 608055d7ccc4..be2d5aa52362 100644
@@ -1680,6 +1680,26 @@ entry will default to 2 instead of 0.
 = =============================================================
 
 
+vmware_record_panic_msg
+=======================
+
+Controls whether kmsg data is written to the host's ``vmware.log`` on
+panic. This setting does not control the separate VMware guest-crash
+event. Writing zero disables recording; writing a nonzero integer
+enables it. Reads return 0 or 1.
+
+= ==============================================================
+0 Do not write kmsg data to ``vmware.log``. This is the default
+  for encrypted guests.
+1 Write kmsg data to ``vmware.log``. This is the default for
+  ordinary guests.
+= ==============================================================
+
+The data is written by a platform panic notifier: early in panic,
+before kdump, and also when a fatal oops enters kdump without a panic.
+Setting ``disable_panic_notifiers_platform_list`` skips it as well.
+
+
 warn_limit
 ==========
 

index 0c6f7248a665..8c8b6dd7c3e8 100644
@@ -33,6 +33,7 @@
 #include <linux/reboot.h>
 #include <linux/sizes.h>
 #include <linux/static_call.h>
+#include <linux/sysctl.h>
 #include <linux/wordpart.h>
 #include <linux/sched/cputime.h>
 #include <asm/div64.h>

@@ -260,6 +261,16 @@ static int vmware_log_rpc(const char *buffer, size_t length)
 static struct page *vmware_panic_page;
 static bool vmware_record_panic_msg;
 
+static const struct ctl_table vmware_panic_sysctls[] = {
+	{
+		.procname	= "vmware_record_panic_msg",
+		.data		= &vmware_record_panic_msg,
+		.maxlen		= sizeof(vmware_record_panic_msg),
+		.mode		= 0644,
+		.proc_handler	= proc_dobool,
+	},
+};
+
 static int vmware_panic_log_notify(struct notifier_block *nb,
 				   unsigned long action, void *data)
 {

@@ -311,6 +322,10 @@ static int __init vmware_panic_log_init(void)
 		vmware_panic_page = NULL;
 	}
 
+	if (vmware_panic_page && IS_ENABLED(CONFIG_SYSCTL) &&
+	    !register_sysctl("kernel", vmware_panic_sysctls))
+		pr_err("failed to register panic log sysctl\n");
+
 	return 0;
 }
 early_initcall(vmware_panic_log_init);

VMware accepts a structured guest-crash event, but a host configured with backdoor.coredumpOnCrash may terminate the VM before the hypercall returns. Keep this event separate from the log sender, which runs earlier in panic.

Register a panic-only kmsg dumper with late_initcall_sync(), after built-in dumpers registered by then. Skip the event while a crash kernel is loaded so the host cannot terminate the guest before kdump saves its vmcore.

Issue the event at most once. With CONFIG_PRINTK=n, use a late INT_MIN panic notifier. An unexpected dumper registration failure with printk enabled leaves reporting disabled rather than overtaking working dumpers.

---
v3: Rebase. No code change from v2.

 arch/x86/include/asm/vmware.h |  1 +
 arch/x86/kernel/cpu/vmware.c  | 62 +++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)

index 598d4cd448ec..ea2ce5272a00 100644
@@ -61,6 +61,7 @@
 #define VMWARE_CMD_GETHZ		45
 #define VMWARE_CMD_GETVCPU_INFO		68
 #define VMWARE_CMD_STEALCLOCK		91
+#define VMWARE_CMD_REPORTGUESTCRASH	102
 /*
  * Hypercall command mask:
  *   bits [6:0] command, range [0, 127]