arm64/hyperv: Enable kexec reboot support
Subsystems (3)
cover arm64/hyperv: Enable kexec reboot support
Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup hv_acpi_init() blocks because the hypervisor still holds the old kernel's VMBus session open.
This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to send CHANNELMSG_UNLOAD and disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.
Why existing notification mechanisms don't work:
- Reboot notifiers: Fire too early - they run inside kernel_restart_prepare(), BEFORE device_shutdown(). VMBus UNLOAD must happen AFTER device_shutdown() so that PCI drivers can clean up interrupt mappings (PCI_DELETE_INTERRUPT_MESSAGE) before UNLOAD force-closes channels. Sending UNLOAD first causes interrupt mapping leaks because the mappings aren't released on channel close.
- Device .shutdown callbacks: Too narrow - these handle per-device cleanup (e.g., individual VMBus channel teardown), but cannot perform bus-level operations like sending the global VMBus UNLOAD message or removing the SynIC CPU hotplug state via cpuhp_remove_state().
- VMBus parent device .shutdown callback: Also insufficient - cpuhp_remove_state() for SynIC teardown must run after all device shutdown completes, not during it. A parent .shutdown callback still executes within device_shutdown().
- The required window is: after device_shutdown() completes, after cpu_hotplug_enable(), but before smp_shutdown_nonboot_cpus(). That window exists inside machine_shutdown(), which currently has no hook.
The need for an ARM64 shutdown hook was previously discussed in [1] but lacked a concrete failure case at the time. We now have one.
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
This RFC proposes fixing kexec on ARM64 Hyper-V guests with:
Patch 1: A platform hook (arm64_pre_smp_shutdown_hook) in ARM64's
machine_shutdown(), analogous to x86's machine_ops.shutdown. This runs after device_shutdown() and cpu_hotplug_enable(), allowing platform code to inject pre-shutdown logic at the right point in the kexec path.
Design choices I'd like feedback on:
- Single function pointer vs full machine_ops struct: ARM64 uses kernel-wide APIs (register_restart_handler, register_platform_power_off) for restart/poweroff rather than x86's monolithic machine_ops, so a targeted hook seemed more consistent with the ARM64 pattern.
- On ARM64, machine_shutdown() is only called from kernel_kexec(), unlike x86/powerpc where it's also called from restart/halt/poweroff.
Patch 2: Uses this hook to call hv_kexec_handler() which performs:
- vmbus_initiate_unload(): sends CHANNELMSG_UNLOAD to host
- cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all CPUs, ensuring the kexec'd kernel starts with clean state
Tested on ARM64 Azure VMs (Ubuntu 22.04, multiple vCPU configs):
- kexec reboot succeeds, VM comes back online with SSH
- Normal reboot/poweroff unaffected
- Multiple consecutive kexec cycles pass
Looking for feedback on:
1. Is the single function pointer hook acceptable for ARM64, or would the ARM64 maintainers prefer a registration API or __weak function?
2. Should this be arm64-specific or generic kexec infrastructure?
Shradha Gupta (2):
arm64: Add pre-shutdown hook to machine_shutdown() arm64/hyperv: Add kexec handler using machine_shutdown hook
arch/arm64/hyperv/mshyperv.c | 37 ++++++++++++++++++++++++++++ arch/arm64/include/asm/system_misc.h | 2 ++ arch/arm64/kernel/process.c | 13 ++++++++++ 3 files changed, 52 insertions(+)
Signature · 1 line
2.43.0
RFC PATCH 1/2 arm64: Add pre-shutdown hook to machine_shutdown()
Add a function pointer hook (arm64_pre_smp_shutdown_hook) that is invoked from machine_shutdown() before smp_shutdown_nonboot_cpus(). This allows platform code (e.g., hypervisors) to perform cleanup that must happen after device_shutdown() but before secondary CPUs go offline.
In the kexec path, the call sequence is:
kernel_kexec()
kernel_restart_prepare()
device_shutdown() // drivers shut down here
migrate_to_reboot_cpu()
cpu_hotplug_enable()
machine_shutdown()
arm64_pre_smp_shutdown_hook() // new: platform cleanup
smp_shutdown_nonboot_cpus() // CPUs go offline
x86 achieves this via machine_ops.shutdown; ARM64 currently has no equivalent mechanism. Rather than introducing the full machine_ops structure, add a targeted hook for the shutdown path.
--- arch/arm64/include/asm/system_misc.h | 2 ++ arch/arm64/kernel/process.c | 13 +++++++++++++ 2 files changed, 15 insertions(+)
arch/arm64/include/asm/system_misc.h +2 -0
@@ -28,6 +28,8 @@ void arm64_notify_die(const char *str, struct pt_regs *regs,struct mm_struct ;extern void __show_regs (struct pt_regs *); +extern void (*arm64_pre_smp_shutdown_hook)(void ); +#endif /* __ASSEMBLER__ */ #endif /* __ASM_SYSTEM_MISC_H */
arch/arm64/kernel/process.c +13 -0
@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)* avoid any code or data used by any SW CPU pin loop. The CPU hotplug * functionality embodied in smpt_shutdown_nonboot_cpus() to achieve this. */ + +/* +* Hook for platform code to perform cleanup after device_shutdown() +* but before secondary CPUs are offlined. This runs in the kexec path +* from kernel_kexec() after device_shutdown() and cpu_hotplug_enable() +* have been called, matching the point at which x86 invokes +* machine_ops.shutdown. +*/ +void (*arm64_pre_smp_shutdown_hook)(void ); +void machine_shutdown (void ) { +if (arm64_pre_smp_shutdown_hook) +arm64_pre_smp_shutdown_hook (); +smp_shutdown_nonboot_cpus (reboot_cpu); }
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The newly introduced global function pointer `arm64_pre_smp_shutdown_hook` is placed in writable…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly introduced global function pointer arm64_pre_smp_shutdown_hook is placed in writable memory without the __ro_after_init annotation, creating a potential target for control-flow hijacking. --
--- Patch [1]: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown() --- commit 438763f14354dbd91b5fa7bea807c334ada0c642
Author: Shradha Gupta <[email protected]>
arm64: Add pre-shutdown hook to machine_shutdown()
This patch introduces a global function pointer invoked from machine_shutdown() to allow platform-specific cleanup before secondary CPUs go offline during kexec or reboot.
[ ... ]
Quoted 5 lines
arch/arm64/kernel/process.c +0
-0@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)
[ ... ]
arch/arm64/kernel/process.c void __noreturn arch_cpu_idle_dead(void) · line 99
Quoted 8 lines
arch/arm64/kernel/process.c +8
-0+/* +* Hook for platform code to perform cleanup after device_shutdown() +* but before secondary CPUs are offlined. This runs in the kexec path +* from kernel_kexec() after device_shutdown() and cpu_hotplug_enable() +* have been called, matching the point at which x86 invokes +* machine_ops.shutdown. +*/ +void (*arm64_pre_smp_shutdown_hook)(void );
[Severity: Medium]
Should this global function pointer be annotated with __ro_after_init?
Without it, the pointer resides in writable memory, which could make it a reliable target for control-flow hijacking if an arbitrary memory write primitive is obtained elsewhere. Since the hook is presumably only set once during platform initialization, making it read-only after init would improve defense-in-depth on the shutdown/reboot path.
Quoted 8 lines
arch/arm64/kernel/process.c +4
-0+void machine_shutdown (void ) { +if (arm64_pre_smp_shutdown_hook) +arm64_pre_smp_shutdown_hook (); +smp_shutdown_nonboot_cpus (reboot_cpu); }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Re: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to machine_shutdown()
Quoted 36 lines · On Fri, Aug 14, 2026 at 09:38:56AM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly introduced global function pointerarm64_pre_smp_shutdown_hookis placed in writable memory without the__ro_after_initannotation, creating a potential target for control-flow hijacking. ----- Patch [1]: [RFC PATCH 1/2] arm64: Add pre-shutdown hook to
machine_shutdown()--- commit438763f14354dbd91b5fa7bea807c334ada0c642
Author: Shradha Gupta <[email protected]>arm64: Add pre-shutdown hook to
machine_shutdown()This patch introduces a global function pointer invoked from
machine_shutdown()to allow platform-specific cleanup before secondary CPUs go offline during kexec or reboot.[ ... ]
Quoted 5 lines
arch/arm64/kernel/process.c +0
-0@@ -88,8 +88,21 @@ void __noreturn arch_cpu_idle_dead(void)[ ... ]
arch/arm64/kernel/process.c void __noreturn arch_cpu_idle_dead(void) · line 99
Quoted 8 lines
arch/arm64/kernel/process.c +8
-0+/* +* Hook for platform code to perform cleanup after device_shutdown() +* but before secondary CPUs are offlined. This runs in the kexec path +* from kernel_kexec() after device_shutdown() and cpu_hotplug_enable() +* have been called, matching the point at which x86 invokes +* machine_ops.shutdown. +*/ +void (*arm64_pre_smp_shutdown_hook)(void );[Severity: Medium]
Should this global function pointer be annotated with__ro_after_init?
Quoted 6 lines
Without it, the pointer resides in writable memory, which could make it a reliable target for control-flow hijacking if an arbitrary memory write primitive is obtained elsewhere. Since the hook is presumably only set once during platform initialization, making it read-only after init would improve defense-in-depth on the shutdown/reboot path.
The pointer is assigned when the VMBus kexec handler is registered and cleared when it is removed, so it cannot be __ro_after_init.
In v1 though, I will use WRITE_ONCE() when publishing/clearing the callback
and READ_ONCE() before invoking it, preventing compiler refetching of the pointer.
Regards,
Shradha
Quoted 11 lines
Quoted 8 lines
arch/arm64/kernel/process.c +4
-0+void machine_shutdown (void ) { +if (arm64_pre_smp_shutdown_hook) +arm64_pre_smp_shutdown_hook (); +smp_shutdown_nonboot_cpus (reboot_cpu); }Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
RFC PATCH 2/2 arm64/hyperv: Add kexec handler using machine_shutdown hook
In reply to Shradha GuptaARM64 Hyper-V guests currently lack the VMBus teardown that x86 performs during kexec via hv_machine_shutdown(). Without this, kexec fails because:
1. The kexec'd kernel's vmbus_connect() hangs because the hypervisor still has the old VMBus connection active.
2. Stale SynIC state (message pending flags, event flags) causes message delivery failures or NULL pointer dereferences in the kexec'd kernel.
The VMBus driver already provides a kexec cleanup callback via the existing hv_setup_kexec_handler() / hv_remove_kexec_handler() API (drivers/hv/vmbus_drv.c). These call into arch-specific overrides; on x86, the override is in mshyperv.c, while ARM64 currently uses the __weak no-op stubs from hv_common.c.
This patch provides the ARM64 override of hv_setup_kexec_handler() and hv_remove_kexec_handler(), wiring the registered hv_kexec_handler to the new arm64_pre_smp_shutdown_hook so that it is invoked during machine_shutdown(). This matches x86's ordering: the handler runs AFTER device_shutdown() (so PCI drivers can clean up interrupt mappings via PCI_DELETE_INTERRUPT_MESSAGE before VMBus UNLOAD force-closes channels) and AFTER cpu_hotplug_enable() (so cpuhp_remove_state() can disable SynIC on all CPUs).
hv_kexec_handler() (registered by vmbus_drv.c via hv_setup_kexec_handler()) performs:
- vmbus_initiate_unload(false): sends CHANNELMSG_UNLOAD to host
- cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all CPUs, ensuring the kexec'd kernel starts with clean SynIC state
--- arch/arm64/hyperv/mshyperv.c | 37 ++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
arch/arm64/hyperv/mshyperv.c +37 -0
@@ -15,10 +15,47 @@#include <linux/errno.h> #include <linux/version.h> #include <linux/cpuhotplug.h> +#include <linux/kexec.h> #include <asm/mshyperv.h> +#include <asm/system_misc.h> static bool hyperv_initialized; +/* +* Kexec/shutdown handler for ARM64 Hyper-V guests. +* +* On x86, hv_machine_shutdown() overrides machine_ops.shutdown and +* runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses +* the arm64_pre_smp_shutdown_hook to achieve the same ordering. +* +* hv_kexec_handler() (set by vmbus_drv.c) performs: +* 1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD +* 2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU +* +* By running after device_shutdown(), PCI drivers (NVMe, MANA) can +* send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt +* mappings before VMBus channels are force-closed by UNLOAD. +*/ +static void (*hv_kexec_handler)(void ); + +static void hv_machine_shutdown (void ) +{ +if (kexec_in_progress && hv_kexec_handler) +hv_kexec_handler (); +} + +void hv_setup_kexec_handler (void (*handler)(void )) +{ + hv_kexec_handler = handler; + arm64_pre_smp_shutdown_hook = hv_machine_shutdown; +} + +void hv_remove_kexec_handler (void ) +{ + arm64_pre_smp_shutdown_hook =NULL ; + hv_kexec_handler =NULL ; +} +int hv_get_hypervisor_version (union hv_hypervisor_version_info *info) { hv_get_vpreg_128(HV_REGISTER_HYPERVISOR_VERSION,
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Concurrent module unload and kexec execution causes a Time-of-Check to Time-of-Use (TOCTOU) Use-After-Free/NULL pointer dereference.
Pre-existing issues:
- [High] vmbus_initiate_unload() ignores vmbus_post_msg() errors, hanging kexec indefinitely on wait_for_completion(). --
--- Patch [2]: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook --- commit 49c11bb6b5ac02d64fdd78fc6417342ccc8089ed
Author: Shradha Gupta <[email protected]>
arm64/hyperv: Add kexec handler using machine_shutdown hook
This patch provides the ARM64 override of hv_setup_kexec_handler() and hv_remove_kexec_handler(). It wires the registered hv_kexec_handler to arm64_pre_smp_shutdown_hook to perform VMBus teardown and disable SynIC during kexec.
Quoted 4 lines
arch/arm64/hyperv/mshyperv.c +0
-0
[ ... ]
Quoted 21 lines
arch/arm64/hyperv/mshyperv.c +13
-0@@ -15,10 +15,47 @@#include <linux/errno.h> #include <linux/version.h> #include <linux/cpuhotplug.h> +#include <linux/kexec.h> #include <asm/mshyperv.h> +#include <asm/system_misc.h> static bool hyperv_initialized; +/* +* Kexec/shutdown handler for ARM64 Hyper-V guests. +* +* On x86, hv_machine_shutdown() overrides machine_ops.shutdown and +* runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses +* the arm64_pre_smp_shutdown_hook to achieve the same ordering. +* +* hv_kexec_handler() (set by vmbus_drv.c) performs: +* 1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD +* 2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU +*
arch/arm64/hyperv/mshyperv.c line 44
Quoted 10 lines
arch/arm64/hyperv/mshyperv.c +10
-0+* By running after device_shutdown(), PCI drivers (NVMe, MANA) can +* send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt +* mappings before VMBus channels are force-closed by UNLOAD. +*/ +static void (*hv_kexec_handler)(void ); + +static void hv_machine_shutdown (void ) +{ +if (kexec_in_progress && hv_kexec_handler) +hv_kexec_handler ();
[Severity: High]
Is there a race condition here if the hv_vmbus module is unloaded concurrently?
If sys_delete_module() calls hv_remove_kexec_handler(), it clears hv_kexec_handler to NULL. If that happens after the check here but before execution, could this jump to a NULL pointer? Or if the module memory is freed before execution, could this cause a use-after-free panic? A simple READ_ONCE() would prevent a compiler double-fetch, but might not prevent the underlying use-after-free without proper synchronization like RCU or a mutex.
[Severity: High]
This is a pre-existing issue, but enabling this path on ARM64 exposes the system to a potential indefinite hang during kexec.
When hv_kexec_handler() calls vmbus_initiate_unload(false), it ignores errors from vmbus_post_msg() and unconditionally waits for completion:
drivers/hv/channel_mgmt.c:vmbus_initiate_unload() {
...
vmbus_post_msg(&hdr, sizeof(struct vmbus_channel_message_header),
!crash);
/*
* vmbus_initiate_unload() is also called on crash and the crash can be
* happening in an interrupt context, where scheduling is impossible.
*/
if (!crash)
wait_for_completion(&vmbus_connection.unload_event);
...
}If vmbus_post_msg() fails (e.g., returning -ENOBUFS if the hypervisor is out of buffers), the unload message is never sent, the completion is never signaled, and the thread waits forever. Is there a way to handle this error instead of hanging the reboot process?
Quoted 13 lines
arch/arm64/hyperv/mshyperv.c +13
-0+} + +void hv_setup_kexec_handler (void (*handler)(void )) +{ + hv_kexec_handler = handler; + arm64_pre_smp_shutdown_hook = hv_machine_shutdown; +} + +void hv_remove_kexec_handler (void ) +{ + arm64_pre_smp_shutdown_hook =NULL ; + hv_kexec_handler =NULL ; +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using machine_shutdown hook
Quoted 90 lines · On Fri, Aug 14, 2026 at 09:49:43AM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Concurrent module unload and kexec execution causes a Time-of-Check to Time-of-Use (TOCTOU) Use-After-Free/NULL pointer dereference.Pre-existing issues:
- [High]vmbus_initiate_unload()ignoresvmbus_post_msg()errors, hanging kexec indefinitely onwait_for_completion(). ----- Patch [2]: [RFC PATCH 2/2] arm64/hyperv: Add kexec handler using
machine_shutdownhook --- commit49c11bb6b5ac02d64fdd78fc6417342ccc8089ed
Author: Shradha Gupta <[email protected]>arm64/hyperv: Add kexec handler using
machine_shutdownhookThis patch provides the ARM64 override of
hv_setup_kexec_handler()andhv_remove_kexec_handler(). It wires the registeredhv_kexec_handlertoarm64_pre_smp_shutdown_hookto perform VMBus teardown and disable SynIC during kexec.Quoted 4 lines
arch/arm64/hyperv/mshyperv.c +0
-0[ ... ]
Quoted 21 lines
arch/arm64/hyperv/mshyperv.c +13
-0@@ -15,10 +15,47 @@#include <linux/errno.h> #include <linux/version.h> #include <linux/cpuhotplug.h> +#include <linux/kexec.h> #include <asm/mshyperv.h> +#include <asm/system_misc.h> static bool hyperv_initialized; +/* +* Kexec/shutdown handler for ARM64 Hyper-V guests. +* +* On x86, hv_machine_shutdown() overrides machine_ops.shutdown and +* runs after device_shutdown() and cpu_hotplug_enable(). ARM64 uses +* the arm64_pre_smp_shutdown_hook to achieve the same ordering. +* +* hv_kexec_handler() (set by vmbus_drv.c) performs: +* 1. vmbus_initiate_unload(false) - sends CHANNELMSG_UNLOAD +* 2. cpuhp_remove_state(hyperv_cpuhp_online) - disables SynIC per CPU +* arch/arm64/hyperv/mshyperv.c line 44
Quoted 10 lines
arch/arm64/hyperv/mshyperv.c +10
-0+* By running after device_shutdown(), PCI drivers (NVMe, MANA) can +* send PCI_DELETE_INTERRUPT_MESSAGE and clean up MMIO/interrupt +* mappings before VMBus channels are force-closed by UNLOAD. +*/ +static void (*hv_kexec_handler)(void ); + +static void hv_machine_shutdown (void ) +{ +if (kexec_in_progress && hv_kexec_handler) +hv_kexec_handler ();[Severity: High]
Is there a race condition here if thehv_vmbusmodule is unloaded concurrently?If
sys_delete_module()callshv_remove_kexec_handler(), it clearshv_kexec_handlerto NULL. If that happens after the check here but before execution, could this jump to a NULL pointer? Or if the module memory is freed before execution, could this cause a use-after-free panic? A simpleREAD_ONCE()would prevent a compiler double-fetch, but might not prevent the underlying use-after-free without proper synchronization like RCU or a mutex.[Severity: High]
This is a pre-existing issue, but enabling this path on ARM64 exposes the system to a potential indefinite hang during kexec.When
hv_kexec_handler()callsvmbus_initiate_unload(false), it ignores errors fromvmbus_post_msg()and unconditionally waits for completion:drivers/hv/channel_mgmt.c:vmbus_initiate_unload() { ... vmbus_post_msg(&hdr, sizeof(struct vmbus_channel_message_header), !crash); /* * vmbus_initiate_unload() is also called on crash and the crash can be * happening in an interrupt context, where scheduling is impossible. */ if (!crash) wait_for_completion(&vmbus_connection.unload_event); ... }
Quoted 5 lines
If
vmbus_post_msg()fails (e.g., returning-ENOBUFSif the hypervisor is out of buffers), the unload message is never sent, the completion is never signaled, and the thread waits forever. Is there a way to handle this error instead of hanging the reboot process?
The x86 Hyper-V implementation uses the same registration, removal, and dispatch pattern without additional synchronization. That said, I agree this precedent alone does not prove the two paths are serialized.
For v1, I will use a registration helper rather than exporting the raw pointer. The kexec path runs only after device_shutdown() completes and with a single CPU remaining, so in practice unregister cannot race with dispatch. I will document this ordering constraint explicitly.
For second comment:
Agreed that vmbus_initiate_unload() ignores the return value from vmbus_post_msg(). This is a pre-existing issue in the common VMBus code shared with x86. I will address it in a separate cleanup patch outside this series.
Quoted 16 lines
Quoted 13 lines
arch/arm64/hyperv/mshyperv.c +13
-0+} + +void hv_setup_kexec_handler (void (*handler)(void )) +{ + hv_kexec_handler = handler; + arm64_pre_smp_shutdown_hook = hv_machine_shutdown; +} + +void hv_remove_kexec_handler (void ) +{ + arm64_pre_smp_shutdown_hook =NULL ; + hv_kexec_handler =NULL ; +}Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
On Fri, Aug 14, 2026 at 02:31:28AM -0700, Shradha Gupta wrote:
Gentle Reminder
Regards,
Shradha.
Quoted 82 lines
Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup
hv_acpi_init()blocks because the hypervisor still holds the old kernel's VMBus session open.This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via
hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to sendCHANNELMSG_UNLOADand disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.Why existing notification mechanisms don't work:
- Reboot notifiers: Fire too early - they run inside
kernel_restart_prepare(), BEFOREdevice_shutdown(). VMBus UNLOAD must happen AFTERdevice_shutdown()so that PCI drivers can clean up interrupt mappings (PCI_DELETE_INTERRUPT_MESSAGE) before UNLOAD force-closes channels. Sending UNLOAD first causes interrupt mapping leaks because the mappings aren't released on channel close.- Device .shutdown callbacks: Too narrow - these handle per-device cleanup (e.g., individual VMBus channel teardown), but cannot perform bus-level operations like sending the global VMBus UNLOAD message or removing the SynIC CPU hotplug state via
cpuhp_remove_state().- VMBus parent device .shutdown callback: Also insufficient -
cpuhp_remove_state()for SynIC teardown must run after all device shutdown completes, not during it. A parent .shutdown callback still executes withindevice_shutdown().- The required window is: after
device_shutdown()completes, aftercpu_hotplug_enable(), but beforesmp_shutdown_nonboot_cpus(). That window exists insidemachine_shutdown(), which currently has no hook.The need for an ARM64 shutdown hook was previously discussed in [1] but lacked a concrete failure case at the time. We now have one.
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
This RFC proposes fixing kexec on ARM64 Hyper-V guests with:
Patch 1: A platform hook (
arm64_pre_smp_shutdown_hook) in ARM64's
machine_shutdown(), analogous to x86's machine_ops.shutdown. This runs afterdevice_shutdown()andcpu_hotplug_enable(), allowing platform code to inject pre-shutdown logic at the right point in the kexec path.Design choices I'd like feedback on:
- Single function pointer vs fullmachine_opsstruct: ARM64 uses kernel-wide APIs (register_restart_handler,register_platform_power_off) for restart/poweroff rather than x86's monolithicmachine_ops, so a targeted hook seemed more consistent with the ARM64 pattern.
- On ARM64,machine_shutdown()is only called fromkernel_kexec(), unlike x86/powerpc where it's also called from restart/halt/poweroff.Patch 2: Uses this hook to call
hv_kexec_handler()which performs:
-vmbus_initiate_unload(): sendsCHANNELMSG_UNLOADto host
-cpuhp_remove_state(): disables SynIC (SIMP, SIEFP, SINT) on all CPUs, ensuring the kexec'd kernel starts with clean stateTested on ARM64 Azure VMs (Ubuntu 22.04, multiple vCPU configs):
- kexec reboot succeeds, VM comes back online with SSH
- Normal reboot/poweroff unaffected
- Multiple consecutive kexec cycles passLooking for feedback on:
1. Is the single function pointer hook acceptable for ARM64, or would the ARM64 maintainers prefer a registration API or__weakfunction?
2. Should this be arm64-specific or generic kexec infrastructure?Shradha Gupta (2):
arm64: Add pre-shutdown hook tomachine_shutdown()arm64/hyperv: Add kexec handler usingmachine_shutdownhookarch/arm64/hyperv/mshyperv.c | 37 ++++++++++++++++++++++++++++ arch/arm64/include/asm/system_misc.h | 2 ++ arch/arm64/kernel/process.c | 13 ++++++++++ 3 files changed, 52 insertions(+)Signature · 1 line
2.43.0
Catalin Marinas
maintainerQuoted 9 lines · Shradha Gupta
Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup
hv_acpi_init()blocks because the hypervisor still holds the old kernel's VMBus session open.This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via
hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to sendCHANNELMSG_UNLOADand disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.
I'm really not keen on such hook, a bare pointer you set elsewhere. Not sure it was discussed before but can you not use the syscore_shutdown() mechanism? This hook got added in 2023, so the 2022 discussion you referenced precedes it.
Signature · 1 line
Catalin
Quoted 11 lines · Catalin Marinas
Quoted 9 lines · Shradha Gupta
Kexec reboot consistently fails on ARM64 Hyper-V guests (Azure VMs). During the kexec shutdown path, VMBus channels are never cleaned up, so in the fresh kexec kernel startup
hv_acpi_init()blocks because the hypervisor still holds the old kernel's VMBus session open.This is because ARM64 lacks the VMBus teardown that x86 performs during kexec via
hv_machine_shutdown(). On x86, machine_ops.shutdown is overridden to sendCHANNELMSG_UNLOADand disable SynIC before CPUs go offline. ARM64 has no equivalent mechanism.
Quoted 7 lines
I'm really not keen on such hook, a bare pointer you set elsewhere. Not sure it was discussed before but can you not use the
syscore_shutdown()mechanism? This hook got added in 2023, so the 2022 discussion you referenced precedes it.Signature · 1 line
Catalin
Thanks for the suggestion, Catalin.
I looked into syscore_shutdown() and it should work. The syscore_shutdown() call in the kexec path runs after device_shutdown() and before smp_shutdown_nonboot_cpus(), which is exactly the window where VMBus teardown needs to happen. It does run before cpu_hotplug_enable() (unlike the x86 path where hv_kexec_handler runs inside machine_shutdown, after cpu_hotplug_enable), but that should be safe because cpuhp_remove_state() acquires cpus_read_lock, which should be independent of the cpu_hotplug_disabled flag.
The implementation would override hv_setup_kexec_handler() in arch/arm64/hyperv/mshyperv.c to register a syscore_ops.shutdown callback, guarded by kexec_in_progress. No ARM64 arch changes needed, and other shutdown paths (reboot, halt, poweroff, kdump) should also be unaffected.
I am testing this now and will send a v2 with the updated approach.
Shradha