KVM: x86: Clear hardware HLT state when userspace makes a vCPU not-halted
Subsystems (4)
PATCH v2 1/1 KVM: x86: Clear hardware HLT state when userspace makes a vCPU not-halted
Force a vCPU out of its hardware-tracked halted state when userspace puts the vCPU into any state other than HALTED via KVM_SET_MP_STATE, i.e. clear VMCS.GUEST_ACTIVITY_STATE if it says the vCPU is halted. Add an optional kvm_x86_ops hook to do the clearing, as SVM has no equivalent VMCB field.
When HLT-exiting is disabled for a VM (KVM_CAP_X86_DISABLE_EXITS with KVM_X86_DISABLE_EXITS_HLT, e.g. QEMU's "-overcommit cpu-pm=on"), a guest HLT halts the physical CPU instead of exiting to KVM, and hardware saves GUEST_ACTIVITY_STATE=HLT into the VMCS on the next VM-Exit. That field is
sticky: it survives VM-Exit/VM-Enter and is only cleared by vmx_clear_hlt() on event injection, or by vmx_vcpu_reset() on INIT / vCPU creation.
Nothing clears it on a userspace-driven state change. KVM_SET_REGS only writes the software register cache and KVM_SET_MP_STATE only writes vcpu->arch.mp_state; kvm_vcpu_running() likewise consults software state only. A VMM that emulates a machine reset therefore ends up with a vCPU that KVM happily VM-Enters while hardware refuses to fetch instructions.
Reproduce with a Linux guest by triggering a panic/kdump on a non-boot
vCPU: nmi_shootdown_cpus() parks the other vCPUs -- including vCPU0 -- in crash_nmi_callback(), which does local_irq_disable() followed by a bare HLT. The capture kernel then resets the machine via port 0xCF9. QEMU rewrites RIP to 0xfff0 and sets mp_state to RUNNABLE, but vCPU0's GUEST_ACTIVITY_STATE is still HLT, so the BSP never executes the reset vector, never sends SIPIs, and the entire VM hangs at "reboot: machine restart" forever. Only destroying and recreating the VM recovers it.
Key off "not HALTED" rather than "is RUNNABLE" as HALTED is the only target state for which retaining the hardware halted state is meaningful; every other transition either can't be reached on VMX, e.g. AP_RESET_HOLD, or goes through vmx_vcpu_reset(), which stuffs GUEST_ACTIVITY_ACTIVE anyways. This also captures the intent more directly: KVM only ever drops the hardware halted state, it never installs it.
Clearing the state is always safe: waking from HLT is architecturally permitted to be spurious, and every HLT in the kernel is inside a loop. Hook KVM_SET_MP_STATE rather than the VM-Enter path so that the clearing is driven by an explicit userspace declaration, and so that no work is added to vmx_vcpu_run(). Gate the call on kvm_hlt_in_guest() even though vmx_clear_hlt() checks it as well, as the check is cheap and makes it obvious in common code when the hardware state can exist at all.
Note, vmx_clear_hlt() loses its "static" qualifier as the kvm_x86_ops table now lives in vmx/main.c. TDX cannot disable HLT-exiting and KVM cannot access a TD's VMCS, so vt_clear_hlt() short-circuits for TD vCPUs, following the existing vt_*() wrapper pattern.
---
v2:
- Gate the call into vendor code on kvm_hlt_in_guest(). (Sean)
- Key off "mp_state->mp_state != KVM_MP_STATE_HALTED" instead of "vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE". (Sean)
- Retitle from "KVM: VMX:" to "KVM: x86:" as the change also touches common x86 code.
- Collect Tested-by and Signed-off-by tags.
v1: https://lore.kernel.org/kvm/[email protected]/
arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 7 +++++++ arch/x86/kvm/vmx/main.c | 13 +++++++++++++ arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/vmx/x86_ops.h | 1 + arch/x86/kvm/x86.c | 15 +++++++++++++++ 6 files changed, 38 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -82,6 +82,7 @@ KVM_X86_OP(interrupt_allowed) KVM_X86_OP(nmi_allowed) KVM_X86_OP(get_nmi_mask) KVM_X86_OP(set_nmi_mask) +KVM_X86_OP_OPTIONAL(clear_hlt) KVM_X86_OP(enable_nmi_window) KVM_X86_OP(enable_irq_window) KVM_X86_OP_OPTIONAL(update_cr8_intercept)
arch/x86/include/asm/kvm_host.h +7 -0
@@ -1607,6 +1607,13 @@ struct kvm_x86_ops {int (*nmi_allowed)(struct kvm_vcpu *vcpu,bool for_injection);bool (*get_nmi_mask)(struct kvm_vcpu *vcpu);void (*set_nmi_mask)(struct kvm_vcpu *vcpu,bool masked); +/* +* Force the vCPU out of any hardware-tracked halted/inactive state so +* that it will fetch and execute instructions on the next VM-Enter. +* Only needed by VMX, where VMCS.GUEST_ACTIVITY_STATE persists across +* VM-Exit/VM-Enter; SVM has no equivalent VMCB field. +*/ +void (*clear_hlt)(struct kvm_vcpu *vcpu);/* Whether or not a virtual NMI is pending in hardware. */ bool (*is_vnmi_pending)(struct kvm_vcpu *vcpu);/*
arch/x86/kvm/vmx/main.c +13 -0
@@ -601,6 +601,18 @@ static void vt_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked)vmx_set_nmi_mask (vcpu, masked); } +static void vt_clear_hlt (struct kvm_vcpu *vcpu) +{ +/* +* TDX doesn't support disabling HLT-exiting, and KVM can't access a +* TD's VMCS, so there is never any hardware halted state to clear. +*/ +if (is_td_vcpu (vcpu)) +return ; + +vmx_clear_hlt (vcpu); +} +static void vt_enable_nmi_window (struct kvm_vcpu *vcpu) {/* Refer to the comments in tdx_inject_nmi(). */
@@ -964,6 +976,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .nmi_allowed = vt_op(nmi_allowed), .get_nmi_mask = vt_op(get_nmi_mask), .set_nmi_mask = vt_op(set_nmi_mask), + .clear_hlt = vt_op(clear_hlt), .enable_nmi_window = vt_op(enable_nmi_window), .enable_irq_window = vt_op(enable_irq_window), .update_cr8_intercept = vt_op(update_cr8_intercept),
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -1909,7 +1909,7 @@ int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu)return skip_emulated_instruction (vcpu); } -static void vmx_clear_hlt (struct kvm_vcpu *vcpu) +void vmx_clear_hlt (struct kvm_vcpu *vcpu) {/* * Ensure that we clear the HLT state in the VMCS. We don't need to
arch/x86/kvm/vmx/x86_ops.h +1 -0
@@ -95,6 +95,7 @@ int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection);int vmx_nmi_allowed (struct kvm_vcpu *vcpu,bool for_injection);bool vmx_get_nmi_mask (struct kvm_vcpu *vcpu);void vmx_set_nmi_mask (struct kvm_vcpu *vcpu,bool masked); +void vmx_clear_hlt (struct kvm_vcpu *vcpu);void vmx_enable_nmi_window (struct kvm_vcpu *vcpu);void vmx_enable_irq_window (struct kvm_vcpu *vcpu);void vmx_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr);
arch/x86/kvm/x86.c +15 -0
@@ -9058,6 +9058,21 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, }kvm_set_mp_state (vcpu, mp_state->mp_state); + +/* +* Force the vCPU out of any hardware-tracked halted state, e.g. VMX's +* GUEST_ACTIVITY_STATE=HLT, when userspace puts the vCPU into a state +* other than HALTED. The hardware state is sticky across VM-Exit and +* VM-Enter and is not touched by any other ioctl, so a vCPU that halted +* with HLT-exiting disabled stays wedged even after userspace rewrites +* its registers, e.g. when a VMM emulates a machine reset. Waking from +* HLT is architecturally allowed to be spurious, so clearing it is +* always safe. +*/ +if (kvm_hlt_in_guest (vcpu->kvm) && + mp_state->mp_state !=KVM_MP_STATE_HALTED ) +kvm_x86_call (clear_hlt)(vcpu); +kvm_make_request (KVM_REQ_EVENT , vcpu); ret =0 ;
base-commit: 1b731e5ded480bd1e5546aed35584238661ce72e
Signature · 1 line
2.24.3
Gentle ping.
The patch still applies cleanly to the latest kvm-x86/next and builds with KVM_INTEL=m KVM_AMD=m KVM_WERROR=y.
No rush given where we are in the cycle -- I just want to make sure this hasn't fallen through the cracks. Please let me know if you'd like any changes, or if I should respin on top of the latest tree.
Thanks!
Quoted 6 lines · keqiang duan <[email protected]> 于2026年8月20日周四 20:44写道：
+Cc Chao Gao
Sorry for the missed Cc. The patch is at
Quoted 4 lines · Keqiang Duan
Force a vCPU out of its hardware-tracked halted state when userspace puts the vCPU into any state other than HALTED via
KVM_SET_MP_STATE, i.e. clear VMCS.GUEST_ACTIVITY_STATE if it says the vCPU is halted. Add an optionalkvm_x86_opshook to do the clearing, as SVM has no equivalent VMCB field.
[snip]
Quoted 6 lines
Fixes caa057a2cad6 ("KVM: X86: Provide a capability to disable HLT intercepts")Tested-by Qinguang Chen <[email protected]>Signed-off-by Zhiping Du <[email protected]>Signed-off-by Keqiang Duan <[email protected]>
Sean Christopherson
maintainerQuoted 19 lines · Keqiang Duan
arch/x86/kvm/vmx/x86_ops.h +1
-0@@ -95,6 +95,7 @@ int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection);int vmx_nmi_allowed (struct kvm_vcpu *vcpu,bool for_injection);bool vmx_get_nmi_mask (struct kvm_vcpu *vcpu);void vmx_set_nmi_mask (struct kvm_vcpu *vcpu,bool masked); +void vmx_clear_hlt (struct kvm_vcpu *vcpu);void vmx_enable_nmi_window (struct kvm_vcpu *vcpu);void vmx_enable_irq_window (struct kvm_vcpu *vcpu);void vmx_update_cr8_intercept (struct kvm_vcpu *vcpu,int tpr,int irr);arch/x86/kvm/x86.c +0
-0@@ -9058,6 +9058,21 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, }
arch/x86/kvm/x86.c int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, · line 9069
Quoted 10 lines
arch/x86/kvm/x86.c +9
-0kvm_set_mp_state (vcpu, mp_state->mp_state); + +/* +* Force the vCPU out of any hardware-tracked halted state, e.g. VMX's +* GUEST_ACTIVITY_STATE=HLT, when userspace puts the vCPU into a state +* other than HALTED. The hardware state is sticky across VM-Exit and +* VM-Enter and is not touched by any other ioctl, so a vCPU that halted +* with HLT-exiting disabled stays wedged even after userspace rewrites +* its registers, e.g. when a VMM emulates a machine reset. Waking from +* HLT is architecturally allowed to be spurious, so clearing it is
Everything looks good except this claim that spurious wakeups is architecturally allowed. I'm 99% certain that is straight up wrong. The SDM explicitly states what will break HLT:
An enabled interrupt (including NMI and SMI), a debug exception, the BINIT# signal, the INIT# signal, or the RESET# signal will resume execution.
and the APM goes a step further, and in addition to listing the wake events:
Execution resumes when an unmasked hardware interrupt (INTR), non-maskable interrupt (NMI), system management interrupt (SMI), RESET, or INIT occurs.
very clearly states that doing HLT with RFLAGS.IF=0 means:
If rFLAGS.IF = 0, the system will remain in a HALT state until an NMI, SMI, RESET, or INIT occurs.
AFAIK, nothing in either the SDM or APM suggests spurious HLT wakeups are allowed. And FWIW, this is not a theoretical issue. A few years back we had a customer issue where a spurious HLT wakeup due to a KVM bug crashed the guest (IIRC, the guest offlined CPUs and put them in HLT, then kexec'd into a new kernel which unmapped the code containing the HLT loop).
Anyways, unless someone cares enough to want to back up the claim that spurious wakeups are ok, I'll just drop that line when applying. I don't see any reason to mention spurious wakes: the vCPU is clearly being moved out of HALTED state, it's on userspace not to screw up (for this particular case; there are other live migration issues that userspace can't solve).
Quoted 15 lines
arch/x86/kvm/x86.c +6
-0+* always safe. +*/ +if (kvm_hlt_in_guest (vcpu->kvm) && + mp_state->mp_state !=KVM_MP_STATE_HALTED ) +kvm_x86_call (clear_hlt)(vcpu); +kvm_make_request (KVM_REQ_EVENT , vcpu); ret =0 ;base-commit:
1b731e5ded480bd1e5546aed35584238661ce72eSignature · 1 line
2.24.3
Quoted 5 lines
Anyways, unless someone cares enough to want to back up the claim that spurious wakeups are ok, I'll just drop that line when applying. I don't see any reason to mention spurious wakes: the vCPU is clearly being moved out of HALTED state, it's on userspace not to screw up (for this particular case; there are other live migration issues that userspace can't solve).
You're right, I was conflating the software convention (every HLT is inside a loop) with
an architectural guarantee. The SDM and APM are clear on what actually wakes HLT,
and neither mentions spurious wakeups.
Please go ahead and drop that line. Thanks for catching it.
Quoted 5 lines · Keqiang Duan
Force a vCPU out of its hardware-tracked halted state when userspace puts the vCPU into any state other than HALTED via
KVM_SET_MP_STATE, i.e. clear VMCS.GUEST_ACTIVITY_STATE if it says the vCPU is halted. Add an optionalkvm_x86_opshook to do the clearing, as SVM has no equivalent VMCB field.
Quoted 8 lines
When HLT-exiting is disabled for a VM (
KVM_CAP_X86_DISABLE_EXITSwithKVM_X86_DISABLE_EXITS_HLT, e.g. QEMU's "-overcommit cpu-pm=on"), a guest HLT halts the physical CPU instead of exiting to KVM, and hardware saves GUEST_ACTIVITY_STATE=HLT into the VMCS on the next VM-Exit. That field is
sticky: it survives VM-Exit/VM-Enter and is only cleared byvmx_clear_hlt()on event injection, or byvmx_vcpu_reset()on INIT / vCPU creation.[...]
Applied to kvm-x86 vmx, thanks!
[1/1] KVM: x86: Clear hardware HLT state when userspace makes a vCPU not-halted
https://github.com/kvm-x86/linux/commit/883719f57f5d