Skip to contentSkip to conversation

arm64/hyperv: Enable kexec reboot support

RFC PATCH In review Shradha Gupta 3 participants 10 messages Aug 14 – Aug 24

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover arm64/hyperv: Enable kexec reboot support

Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup hv_acpi_init() blocks because the hypervisor still holds the old kernel's VMBus session open.

This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to send CHANNELMSG_UNLOAD and disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.

Why existing notification mechanisms don't work:

- Reboot notifiers: Fire too early - they run inside kernel_restart_prepare(), BEFORE device_shutdown(). VMBus UNLOAD must happen AFTER device_shutdown() so that PCI drivers can clean up interrupt mappings (PCI_DELETE_INTERRUPT_MESSAGE) before UNLOAD force-closes channels. Sending UNLOAD first causes interrupt mapping leaks because the mappings aren't released on channel close.

- Device .shutdown callbacks: Too narrow - these handle per-device cleanup (e.g., individual VMBus channel teardown), but cannot perform bus-level operations like sending the global VMBus UNLOAD message or removing the SynIC CPU hotplug state via cpuhp_remove_state().

- VMBus parent device .shutdown callback: Also insufficient - cpuhp_remove_state() for SynIC teardown must run after all device shutdown completes, not during it. A parent .shutdown callback still executes within device_shutdown().

- The required window is: after device_shutdown() completes, after cpu_hotplug_enable(), but before smp_shutdown_nonboot_cpus(). That window exists inside machine_shutdown(), which currently has no hook.

The need for an ARM64 shutdown hook was previously discussed in [1] but lacked a concrete failure case at the time. We now have one.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

This RFC proposes fixing kexec on ARM64 Hyper-V guests with:

Patch 1: A platform hook (arm64_pre_smp_shutdown_hook) in ARM64's
machine_shutdown(), analogous to x86's machine_ops.shutdown. This runs after device_shutdown() and cpu_hotplug_enable(), allowing platform code to inject pre-shutdown logic at the right point in the kexec path.

Design choices I'd like feedback on:
- Single function pointer vs full machine_ops struct: ARM64 uses kernel-wide APIs (register_restart_handler, register_platform_power_off) for restart/poweroff rather than x86's monolithic machine_ops, so a targeted hook seemed more consistent with the ARM64 pattern.
- On ARM64, machine_shutdown() is only called from kernel_kexec(), unlike x86/powerpc where it's also called from restart/halt/poweroff.

Patch 2: Uses this hook to call hv_kexec_handler() which performs:
- vmbus_initiate_unload(): sends CHANNELMSG_UNLOAD to host
- cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all CPUs, ensuring the kexec'd kernel starts with clean state

Tested on ARM64 Azure VMs (Ubuntu 22.04, multiple vCPU configs):
- kexec reboot succeeds, VM comes back online with SSH
- Normal reboot/poweroff unaffected
- Multiple consecutive kexec cycles pass

Looking for feedback on:
1. Is the single function pointer hook acceptable for ARM64, or would the ARM64 maintainers prefer a registration API or __weak function?
2. Should this be arm64-specific or generic kexec infrastructure?

Shradha Gupta (2):
arm64: Add pre-shutdown hook to machine_shutdown() arm64/hyperv: Add kexec handler using machine_shutdown hook

 arch/arm64/hyperv/mshyperv.c         | 37 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/system_misc.h |  2 ++
 arch/arm64/kernel/process.c          | 13 ++++++++++
 3 files changed, 52 insertions(+)
Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH 1/2 arm64: Add pre-shutdown hook to machine_shutdown()

Add a function pointer hook (arm64_pre_smp_shutdown_hook) that is invoked from machine_shutdown() before smp_shutdown_nonboot_cpus(). This allows platform code (e.g., hypervisors) to perform cleanup that must happen after device_shutdown() but before secondary CPUs go offline.

In the kexec path, the call sequence is:
kernel_kexec()
kernel_restart_prepare()
device_shutdown() // drivers shut down here
migrate_to_reboot_cpu()
cpu_hotplug_enable()
machine_shutdown()
arm64_pre_smp_shutdown_hook() // new: platform cleanup
smp_shutdown_nonboot_cpus() // CPUs go offline

x86 achieves this via machine_ops.shutdown; ARM64 currently has no equivalent mechanism. Rather than introducing the full machine_ops structure, add a targeted hook for the shutdown path.

---
 arch/arm64/include/asm/system_misc.h |  2 ++
 arch/arm64/kernel/process.c          | 13 +++++++++++++
 2 files changed, 15 insertions(+)
arch/arm64/include/asm/system_misc.h +2 -0
index d316a804eb38..7c37b9f33e96 100644
@@ -28,6 +28,8 @@ void arm64_notify_die(const char *str, struct pt_regs *regs,
 struct mm_struct;
 extern void __show_regs(struct pt_regs *);
 
+extern void (*arm64_pre_smp_shutdown_hook)(void);
+
 #endif	/* __ASSEMBLER__ */
 
 #endif	/* __ASM_SYSTEM_MISC_H */
arch/arm64/kernel/process.c +13 -0
index 581f80e9b9b7..41c0e9840913 100644
@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)
  * avoid any code or data used by any SW CPU pin loop. The CPU hotplug
  * functionality embodied in smpt_shutdown_nonboot_cpus() to achieve this.
  */
+
+/*
+ * Hook for platform code to perform cleanup after device_shutdown()
+ * but before secondary CPUs are offlined. This runs in the kexec path
+ * from kernel_kexec() after device_shutdown() and cpu_hotplug_enable()
+ * have been called, matching the point at which x86 invokes
+ * machine_ops.shutdown.
+ */
+void (*arm64_pre_smp_shutdown_hook)(void);
+
 void machine_shutdown(void)
 {
+	if (arm64_pre_smp_shutdown_hook)
+		arm64_pre_smp_shutdown_hook();
+
 	smp_shutdown_nonboot_cpus(reboot_cpu);
 }
 
Signature · 1 line
2.43.0
#2 lore raw

Re: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The newly introduced global function pointer `arm64_pre_smp_shutdown_hook` is placed in writable…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly introduced global function pointer arm64_pre_smp_shutdown_hook is placed in writable memory without the __ro_after_init annotation, creating a potential target for control-flow hijacking. --

--- Patch [1]: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown() --- commit 438763f14354dbd91b5fa7bea807c334ada0c642
Author: Shradha Gupta <[email protected]>

arm64: Add pre-shutdown hook to machine_shutdown()

This patch introduces a global function pointer invoked from machine_shutdown() to allow platform-specific cleanup before secondary CPUs go offline during kexec or reboot.

[ ... ]

Quoted 5 lines
arch/arm64/kernel/process.c +0 -0
index 581f80e9b9b73..41c0e98409138 100644
@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)

[ ... ]

arch/arm64/kernel/process.c void __noreturn arch_cpu_idle_dead(void) · line 99

Quoted 8 lines
arch/arm64/kernel/process.c +8 -0
+/*
+ * Hook for platform code to perform cleanup after device_shutdown()
+ * but before secondary CPUs are offlined. This runs in the kexec path
+ * from kernel_kexec() after device_shutdown() and cpu_hotplug_enable()
+ * have been called, matching the point at which x86 invokes
+ * machine_ops.shutdown.
+ */
+void (*arm64_pre_smp_shutdown_hook)(void);

[Severity: Medium]
Should this global function pointer be annotated with __ro_after_init?

Without it, the pointer resides in writable memory, which could make it a reliable target for control-flow hijacking if an arbitrary memory write primitive is obtained elsewhere. Since the hook is presumably only set once during platform initialization, making it read-only after init would improve defense-in-depth on the shutdown/reboot path.

Quoted 8 lines
arch/arm64/kernel/process.c +4 -0
+
 void machine_shutdown(void)
 {
+	if (arm64_pre_smp_shutdown_hook)
+		arm64_pre_smp_shutdown_hook();
+
 	smp_shutdown_nonboot_cpus(reboot_cpu);
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown()

Quoted 36 lines · On Fri, Aug 14, 2026 at 09:38:56AM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly introduced global function pointer arm64_pre_smp_shutdown_hook is placed in writable memory without the __ro_after_init annotation, creating a potential target for control-flow hijacking. --

--- Patch [1]: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown() --- commit 438763f14354dbd91b5fa7bea807c334ada0c642
Author: Shradha Gupta <[email protected]>

arm64: Add pre-shutdown hook to machine_shutdown()

This patch introduces a global function pointer invoked from machine_shutdown() to allow platform-specific cleanup before secondary CPUs go offline during kexec or reboot.

[ ... ]

Quoted 5 lines
arch/arm64/kernel/process.c +0 -0
index 581f80e9b9b73..41c0e98409138 100644
@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)

[ ... ]

arch/arm64/kernel/process.c void __noreturn arch_cpu_idle_dead(void) · line 99

Quoted 8 lines
arch/arm64/kernel/process.c +8 -0
+/*
+ * Hook for platform code to perform cleanup after device_shutdown()
+ * but before secondary CPUs are offlined. This runs in the kexec path
+ * from kernel_kexec() after device_shutdown() and cpu_hotplug_enable()
+ * have been called, matching the point at which x86 invokes
+ * machine_ops.shutdown.
+ */
+void (*arm64_pre_smp_shutdown_hook)(void);

[Severity: Medium]
Should this global function pointer be annotated with __ro_after_init?

Quoted 6 lines

Without it, the pointer resides in writable memory, which could make it a reliable target for control-flow hijacking if an arbitrary memory write primitive is obtained elsewhere. Since the hook is presumably only set once during platform initialization, making it read-only after init would improve defense-in-depth on the shutdown/reboot path.

The pointer is assigned when the VMBus kexec handler is registered and cleared when it is removed, so it cannot be __ro_after_init.

In v1 though, I will use WRITE_ONCE() when publishing/clearing the callback
and READ_ONCE() before invoking it, preventing compiler refetching of the pointer.

Regards,
Shradha

Quoted 11 lines
Quoted 8 lines
arch/arm64/kernel/process.c +4 -0
+
 void machine_shutdown(void)
 {
+	if (arm64_pre_smp_shutdown_hook)
+		arm64_pre_smp_shutdown_hook();
+
 	smp_shutdown_nonboot_cpus(reboot_cpu);
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#4 lore raw

RFC PATCH 2/2 arm64/hyperv: Add kexec handler using machine_shutdown hook

In reply to Shradha Gupta

ARM64 Hyper-V guests currently lack the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). Without this, kexec fails because:

1. The kexec'd kernel's vmbus_connect() hangs because the hypervisor still has the old VMBus connection active.
2. Stale SynIC state (message pending flags, event flags) causes message delivery failures or NULL pointer dereferences in the kexec'd kernel.

The VMBus driver already provides a kexec cleanup callback via the existing hv_setup_kexec_handler() / hv_remove_kexec_handler() API (drivers/hv/vmbus_drv.c). These call into arch-specific overrides; on x86, the override is in mshyperv.c, while ARM64 currently uses the __weak no-op stubs from hv_common.c.

This patch provides the ARM64 override of hv_setup_kexec_handler() and hv_remove_kexec_handler(), wiring the registered hv_kexec_handler to the new arm64_pre_smp_shutdown_hook so that it is invoked during machine_shutdown(). This matches x86's ordering: the handler runs AFTER device_shutdown() (so PCI drivers can clean up interrupt mappings via PCI_DELETE_INTERRUPT_MESSAGE before VMBus UNLOAD force-closes channels) and AFTER cpu_hotplug_enable() (so cpuhp_remove_state() can disable SynIC on all CPUs).

hv_kexec_handler() (registered by vmbus_drv.c via hv_setup_kexec_handler()) performs:
- vmbus_initiate_unload(false): sends CHANNELMSG_UNLOAD to host
- cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all CPUs, ensuring the kexec'd kernel starts with clean SynIC state

---
 arch/arm64/hyperv/mshyperv.c | 37 ++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
arch/arm64/hyperv/mshyperv.c +37 -0
index 4fdc26ade1d7..4176175060d3 100644
@@ -15,10 +15,47 @@
 #include <linux/errno.h>
 #include <linux/version.h>
 #include <linux/cpuhotplug.h>
+#include <linux/kexec.h>
 #include <asm/mshyperv.h>
+#include <asm/system_misc.h>
 
 static bool hyperv_initialized;
 
+/*
+ * Kexec/shutdown handler for ARM64 Hyper-V guests.
+ *
+ * On x86, hv_machine_shutdown() overrides machine_ops.shutdown and
+ * runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses
+ * the arm64_pre_smp_shutdown_hook to achieve the same ordering.
+ *
+ * hv_kexec_handler() (set by vmbus_drv.c) performs:
+ *   1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD
+ *   2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU
+ *
+ * By running after device_shutdown(), PCI drivers (NVMe, MANA) can
+ * send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt
+ * mappings before VMBus channels are force-closed by UNLOAD.
+ */
+static void (*hv_kexec_handler)(void);
+
+static void hv_machine_shutdown(void)
+{
+	if (kexec_in_progress && hv_kexec_handler)
+		hv_kexec_handler();
+}
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+	hv_kexec_handler = handler;
+	arm64_pre_smp_shutdown_hook = hv_machine_shutdown;
+}
+
+void hv_remove_kexec_handler(void)
+{
+	arm64_pre_smp_shutdown_hook = NULL;
+	hv_kexec_handler = NULL;
+}
+
 int hv_get_hypervisor_version(union hv_hypervisor_version_info *info)
 {
 	hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION,
Signature · 1 line
2.43.0
#5 lore raw

Re: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Concurrent module unload and kexec execution causes a Time-of-Check to Time-of-Use (TOCTOU) Use-After-Free/NULL pointer dereference.

Pre-existing issues:
- [High] vmbus_initiate_unload() ignores vmbus_post_msg() errors, hanging kexec indefinitely on wait_for_completion(). --

--- Patch [2]: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook --- commit 49c11bb6b5ac02d64fdd78fc6417342ccc8089ed
Author: Shradha Gupta <[email protected]>

arm64/hyperv: Add kexec handler using machine_shutdown hook

This patch provides the ARM64 override of hv_setup_kexec_handler() and hv_remove_kexec_handler(). It wires the registered hv_kexec_handler to arm64_pre_smp_shutdown_hook to perform VMBus teardown and disable SynIC during kexec.

Quoted 4 lines
arch/arm64/hyperv/mshyperv.c +0 -0
index 4fdc26ade1d74..4176175060d30 100644

[ ... ]

Quoted 21 lines
arch/arm64/hyperv/mshyperv.c +13 -0
@@ -15,10 +15,47 @@
 #include <linux/errno.h>
 #include <linux/version.h>
 #include <linux/cpuhotplug.h>
+#include <linux/kexec.h>
 #include <asm/mshyperv.h>
+#include <asm/system_misc.h>
 
 static bool hyperv_initialized;
 
+/*
+ * Kexec/shutdown handler for ARM64 Hyper-V guests.
+ *
+ * On x86, hv_machine_shutdown() overrides machine_ops.shutdown and
+ * runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses
+ * the arm64_pre_smp_shutdown_hook to achieve the same ordering.
+ *
+ * hv_kexec_handler() (set by vmbus_drv.c) performs:
+ *   1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD
+ *   2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU
+ *

arch/arm64/hyperv/mshyperv.c line 44

Quoted 10 lines
arch/arm64/hyperv/mshyperv.c +10 -0
+ * By running after device_shutdown(), PCI drivers (NVMe, MANA) can
+ * send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt
+ * mappings before VMBus channels are force-closed by UNLOAD.
+ */
+static void (*hv_kexec_handler)(void);
+
+static void hv_machine_shutdown(void)
+{
+	if (kexec_in_progress && hv_kexec_handler)
+		hv_kexec_handler();

[Severity: High]
Is there a race condition here if the hv_vmbus module is unloaded concurrently?

If sys_delete_module() calls hv_remove_kexec_handler(), it clears hv_kexec_handler to NULL. If that happens after the check here but before execution, could this jump to a NULL pointer? Or if the module memory is freed before execution, could this cause a use-after-free panic? A simple READ_ONCE() would prevent a compiler double-fetch, but might not prevent the underlying use-after-free without proper synchronization like RCU or a mutex.

[Severity: High]
This is a pre-existing issue, but enabling this path on ARM64 exposes the system to a potential indefinite hang during kexec.

When hv_kexec_handler() calls vmbus_initiate_unload(false), it ignores errors from vmbus_post_msg() and unconditionally waits for completion:

drivers/hv/channel_mgmt.c:vmbus_initiate_unload() {
	...
	vmbus_post_msg(&hdr, sizeof(struct vmbus_channel_message_header),
		       !crash);

	/*
	 * vmbus_initiate_unload() is also called on crash and the crash can be
	 * happening in an interrupt context, where scheduling is impossible.
	 */
	if (!crash)
		wait_for_completion(&vmbus_connection.unload_event);
	...
}

If vmbus_post_msg() fails (e.g., returning -ENOBUFS if the hypervisor is out of buffers), the unload message is never sent, the completion is never signaled, and the thread waits forever. Is there a way to handle this error instead of hanging the reboot process?

Quoted 13 lines
arch/arm64/hyperv/mshyperv.c +13 -0
+}
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+	hv_kexec_handler = handler;
+	arm64_pre_smp_shutdown_hook = hv_machine_shutdown;
+}
+
+void hv_remove_kexec_handler(void)
+{
+	arm64_pre_smp_shutdown_hook = NULL;
+	hv_kexec_handler = NULL;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#6 lore raw

Re: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook

Quoted 90 lines · On Fri, Aug 14, 2026 at 09:49:43AM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] Concurrent module unload and kexec execution causes a Time-of-Check to Time-of-Use (TOCTOU) Use-After-Free/NULL pointer dereference.

Pre-existing issues:
- [High] vmbus_initiate_unload() ignores vmbus_post_msg() errors, hanging kexec indefinitely on wait_for_completion(). --

--- Patch [2]: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook --- commit 49c11bb6b5ac02d64fdd78fc6417342ccc8089ed
Author: Shradha Gupta <[email protected]>

arm64/hyperv: Add kexec handler using machine_shutdown hook

This patch provides the ARM64 override of hv_setup_kexec_handler() and hv_remove_kexec_handler(). It wires the registered hv_kexec_handler to arm64_pre_smp_shutdown_hook to perform VMBus teardown and disable SynIC during kexec.

Quoted 4 lines
arch/arm64/hyperv/mshyperv.c +0 -0
index 4fdc26ade1d74..4176175060d30 100644

[ ... ]

Quoted 21 lines
arch/arm64/hyperv/mshyperv.c +13 -0
@@ -15,10 +15,47 @@
 #include <linux/errno.h>
 #include <linux/version.h>
 #include <linux/cpuhotplug.h>
+#include <linux/kexec.h>
 #include <asm/mshyperv.h>
+#include <asm/system_misc.h>
 
 static bool hyperv_initialized;
 
+/*
+ * Kexec/shutdown handler for ARM64 Hyper-V guests.
+ *
+ * On x86, hv_machine_shutdown() overrides machine_ops.shutdown and
+ * runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses
+ * the arm64_pre_smp_shutdown_hook to achieve the same ordering.
+ *
+ * hv_kexec_handler() (set by vmbus_drv.c) performs:
+ *   1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD
+ *   2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU
+ *

arch/arm64/hyperv/mshyperv.c line 44

Quoted 10 lines
arch/arm64/hyperv/mshyperv.c +10 -0
+ * By running after device_shutdown(), PCI drivers (NVMe, MANA) can
+ * send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt
+ * mappings before VMBus channels are force-closed by UNLOAD.
+ */
+static void (*hv_kexec_handler)(void);
+
+static void hv_machine_shutdown(void)
+{
+	if (kexec_in_progress && hv_kexec_handler)
+		hv_kexec_handler();

[Severity: High]
Is there a race condition here if the hv_vmbus module is unloaded concurrently?

If sys_delete_module() calls hv_remove_kexec_handler(), it clears hv_kexec_handler to NULL. If that happens after the check here but before execution, could this jump to a NULL pointer? Or if the module memory is freed before execution, could this cause a use-after-free panic? A simple READ_ONCE() would prevent a compiler double-fetch, but might not prevent the underlying use-after-free without proper synchronization like RCU or a mutex.

[Severity: High]
This is a pre-existing issue, but enabling this path on ARM64 exposes the system to a potential indefinite hang during kexec.

When hv_kexec_handler() calls vmbus_initiate_unload(false), it ignores errors from vmbus_post_msg() and unconditionally waits for completion:

drivers/hv/channel_mgmt.c:vmbus_initiate_unload() {
	...
	vmbus_post_msg(&hdr, sizeof(struct vmbus_channel_message_header),
		       !crash);

	/*
	 * vmbus_initiate_unload() is also called on crash and the crash can be
	 * happening in an interrupt context, where scheduling is impossible.
	 */
	if (!crash)
		wait_for_completion(&vmbus_connection.unload_event);
	...
}
Quoted 5 lines

If vmbus_post_msg() fails (e.g., returning -ENOBUFS if the hypervisor is out of buffers), the unload message is never sent, the completion is never signaled, and the thread waits forever. Is there a way to handle this error instead of hanging the reboot process?

The x86 Hyper-V implementation uses the same registration, removal, and dispatch pattern without additional synchronization. That said, I agree this precedent alone does not prove the two paths are serialized.

For v1, I will use a registration helper rather than exporting the raw pointer. The kexec path runs only after device_shutdown() completes and with a single CPU remaining, so in practice unregister cannot race with dispatch. I will document this ordering constraint explicitly.

For second comment:

Agreed that vmbus_initiate_unload() ignores the return value from vmbus_post_msg(). This is a pre-existing issue in the common VMBus code shared with x86. I will address it in a separate cleanup patch outside this series.

Quoted 16 lines
Quoted 13 lines
arch/arm64/hyperv/mshyperv.c +13 -0
+}
+
+void hv_setup_kexec_handler(void (*handler)(void))
+{
+	hv_kexec_handler = handler;
+	arm64_pre_smp_shutdown_hook = hv_machine_shutdown;
+}
+
+void hv_remove_kexec_handler(void)
+{
+	arm64_pre_smp_shutdown_hook = NULL;
+	hv_kexec_handler = NULL;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#7 lore raw
In reply to Shradha Gupta

On Fri, Aug 14, 2026 at 02:31:28AM -0700, Shradha Gupta wrote:

Gentle Reminder

Regards,
Shradha.

Quoted 82 lines

Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup hv_acpi_init() blocks because the hypervisor still holds the old kernel's VMBus session open.

This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to send CHANNELMSG_UNLOAD and disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.

Why existing notification mechanisms don't work:

- Reboot notifiers: Fire too early - they run inside kernel_restart_prepare(), BEFORE device_shutdown(). VMBus UNLOAD must happen AFTER device_shutdown() so that PCI drivers can clean up interrupt mappings (PCI_DELETE_INTERRUPT_MESSAGE) before UNLOAD force-closes channels. Sending UNLOAD first causes interrupt mapping leaks because the mappings aren't released on channel close.

- Device .shutdown callbacks: Too narrow - these handle per-device cleanup (e.g., individual VMBus channel teardown), but cannot perform bus-level operations like sending the global VMBus UNLOAD message or removing the SynIC CPU hotplug state via cpuhp_remove_state().

- VMBus parent device .shutdown callback: Also insufficient - cpuhp_remove_state() for SynIC teardown must run after all device shutdown completes, not during it. A parent .shutdown callback still executes within device_shutdown().

- The required window is: after device_shutdown() completes, after cpu_hotplug_enable(), but before smp_shutdown_nonboot_cpus(). That window exists inside machine_shutdown(), which currently has no hook.

The need for an ARM64 shutdown hook was previously discussed in [1] but lacked a concrete failure case at the time. We now have one.

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

This RFC proposes fixing kexec on ARM64 Hyper-V guests with:

Patch 1: A platform hook (arm64_pre_smp_shutdown_hook) in ARM64's
machine_shutdown(), analogous to x86's machine_ops.shutdown. This runs after device_shutdown() and cpu_hotplug_enable(), allowing platform code to inject pre-shutdown logic at the right point in the kexec path.

Design choices I'd like feedback on:
- Single function pointer vs full machine_ops struct: ARM64 uses kernel-wide APIs (register_restart_handler, register_platform_power_off) for restart/poweroff rather than x86's monolithic machine_ops, so a targeted hook seemed more consistent with the ARM64 pattern.
- On ARM64, machine_shutdown() is only called from kernel_kexec(), unlike x86/powerpc where it's also called from restart/halt/poweroff.

Patch 2: Uses this hook to call hv_kexec_handler() which performs:
- vmbus_initiate_unload(): sends CHANNELMSG_UNLOAD to host
- cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all CPUs, ensuring the kexec'd kernel starts with clean state

Tested on ARM64 Azure VMs (Ubuntu 22.04, multiple vCPU configs):
- kexec reboot succeeds, VM comes back online with SSH
- Normal reboot/poweroff unaffected
- Multiple consecutive kexec cycles pass

Looking for feedback on:
1. Is the single function pointer hook acceptable for ARM64, or would the ARM64 maintainers prefer a registration API or __weak function?
2. Should this be arm64-specific or generic kexec infrastructure?

Shradha Gupta (2):
arm64: Add pre-shutdown hook to machine_shutdown() arm64/hyperv: Add kexec handler using machine_shutdown hook

 arch/arm64/hyperv/mshyperv.c         | 37 ++++++++++++++++++++++++++++
 arch/arm64/include/asm/system_misc.h |  2 ++
 arch/arm64/kernel/process.c          | 13 ++++++++++
 3 files changed, 52 insertions(+)
Signature · 1 line
2.43.0
maintainer
#8 lore raw
In reply to Shradha Gupta
Quoted 9 lines · Shradha Gupta

Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup hv_acpi_init() blocks because the hypervisor still holds the old kernel's VMBus session open.

This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to send CHANNELMSG_UNLOAD and disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.

I'm really not keen on such hook, a bare pointer you set elsewhere. Not sure it was discussed before but can you not use the syscore_shutdown() mechanism? This hook got added in 2023, so the 2022 discussion you referenced precedes it.

Signature · 1 line
Catalin
#9 lore raw
Quoted 11 lines · Catalin Marinas
Quoted 9 lines · Shradha Gupta

Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup hv_acpi_init() blocks because the hypervisor still holds the old kernel's VMBus session open.

This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to send CHANNELMSG_UNLOAD and disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.

Quoted 7 lines

I'm really not keen on such hook, a bare pointer you set elsewhere. Not sure it was discussed before but can you not use the syscore_shutdown() mechanism? This hook got added in 2023, so the 2022 discussion you referenced precedes it.

Signature · 1 line
Catalin

Thanks for the suggestion, Catalin.

I looked into syscore_shutdown() and it should work. The syscore_shutdown() call in the kexec path runs after device_shutdown() and before smp_shutdown_nonboot_cpus(), which is exactly the window where VMBus teardown needs to happen. It does run before cpu_hotplug_enable() (unlike the x86 path where hv_kexec_handler runs inside machine_shutdown, after cpu_hotplug_enable), but that should be safe because cpuhp_remove_state() acquires cpus_read_lock, which should be independent of the cpu_hotplug_disabled flag.

The implementation would override hv_setup_kexec_handler() in arch/arm64/hyperv/mshyperv.c to register a syscore_ops.shutdown callback, guarded by kexec_in_progress. No ARM64 arch changes needed, and other shutdown paths (reboot, halt, poweroff, kdump) should also be unaffected.

I am testing this now and will send a v2 with the updated approach.

Shradha