Skip to contentSkip to conversation

KVM: x86: Fix hard lockup with periodic timer in guest

PATCH Applied Sean Christopherson 1 participant 6 messages Nov 13, 2025 – Nov 18, 2025

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

v6 cover KVM: x86: Fix hard lockup with periodic timer in guest

fuqiang's patch/series to fix a bug in KVM's local APIC timer emulation where it can trigger a hard lockup due to restarting an hrtimer with an expired deadline over and over (and over).

v6:
- Split the apic_timer_fn() change to a separate patch (mainly for a bisection point).
- Handle (and WARN on) period=0 in apic_timer_fn().
- Add a patch to grab a pointer to the kvm_timer struct locally.
- Tag the fixes (and prep work) for stable@.

v5:
- https://lore.kernel.org/all/[email protected]
- Add more details in commit messages and letters.

v4:
- https://lore.kernel.org/all/[email protected]
- merge two patch into one

v3:
- https://lore.kernel.org/all/[email protected]
- Fix: advanced SW timer (hrtimer) expiration does not catch up to current time.
- optimize the commit message of patch 2

v2:
- https://lore.kernel.org/all/[email protected]
- Added a bugfix for hardlockup in v2

v1: https://lore.kernel.org/all/[email protected]

Sean Christopherson (2):
KVM: x86: WARN if hrtimer callback for periodic APIC timer fires with
period=0
KVM: x86: Grab lapic_timer in a local variable to cleanup periodic
code

fuqiang wang (2):
KVM: x86: Explicitly set new periodic hrtimer expiration in

    apic_timer_fn()

KVM: x86: Fix VM hard lockup after prolonged inactivity with periodic
HV timer

 arch/x86/kvm/lapic.c | 44 +++++++++++++++++++++++++++++++-------------
 1 file changed, 31 insertions(+), 13 deletions(-)

base-commit: 16ec4fb4ac95d878b879192d280db2baeec43272

Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
author maintainer
#1 lore raw

PATCH v6 1/4 KVM: x86: WARN if hrtimer callback for periodic APIC timer fires with period=0

WARN and don't restart the hrtimer if KVM's callback runs with the guest's APIC timer in periodic mode but with a period of '0', as not advancing the hrtimer's deadline would put the CPU into an infinite loop of hrtimer events. Observing a period of '0' should be impossible, even when the hrtimer is running on a different CPU than the vCPU, as KVM is supposed to cancel the hrtimer before changing (or zeroing) the period, e.g. when switching from periodic to one-shot.

---
 arch/x86/kvm/lapic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/lapic.c +1 -1
index 0ae7f913d782..78b74ba17592 100644
@@ -2970,7 +2970,7 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
 
 	apic_timer_expired(apic, true);
 
-	if (lapic_is_periodic(apic)) {
+	if (lapic_is_periodic(apic) && !WARN_ON_ONCE(!apic->lapic_timer.period)) {
 		advance_periodic_target_expiration(apic);
 		hrtimer_add_expires_ns(&ktimer->timer, ktimer->period);
 		return HRTIMER_RESTART;
Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
author maintainer
#2 lore raw

PATCH v6 2/4 KVM: x86: Explicitly set new periodic hrtimer expiration in apic_timer_fn()

In reply to Sean Christopherson

From: fuqiang wang <[email protected]>

When restarting an hrtimer to emulate a the guest's APIC timer in periodic mode, explicitly set the expiration using the target expiration computed by advance_periodic_target_expiration() instead of adding the period to the existing timer. This will allow making adjustments to the expiration, e.g. to deal with expirations far in the past, without having to implement the same logic in both advance_periodic_target_expiration() and apic_timer_fn().

[sean: split to separate patch, write changelog]
---
 arch/x86/kvm/lapic.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/lapic.c +1 -1
index 78b74ba17592..a5c927e7bae6 100644
@@ -2972,7 +2972,7 @@ static enum hrtimer_restart apic_timer_fn(struct hrtimer *data)
 
 	if (lapic_is_periodic(apic) && !WARN_ON_ONCE(!apic->lapic_timer.period)) {
 		advance_periodic_target_expiration(apic);
-		hrtimer_add_expires_ns(&ktimer->timer, ktimer->period);
+		hrtimer_set_expires(&ktimer->timer, ktimer->target_expiration);
 		return HRTIMER_RESTART;
 	} else
 		return HRTIMER_NORESTART;
Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
author maintainer
#3 lore raw

PATCH v6 3/4 KVM: x86: Fix VM hard lockup after prolonged inactivity with periodic HV timer

In reply to Sean Christopherson

From: fuqiang wang <[email protected]>

When advancing the target expiration for the guest's APIC timer in periodic mode, set the expiration to "now" if the target expiration is in the past (similar to what is done in update_target_expiration()). Blindly adding the period to the previous target expiration can result in KVM generating a practically unbounded number of hrtimer IRQs due to programming an expired timer over and over. In extreme scenarios, e.g. if userspace pauses/suspends a VM for an extended duration, this can even cause hard lockups in the host.

Currently, the bug only affects Intel CPUs when using the hypervisor timer (HV timer), a.k.a. the VMX preemption timer. Unlike the software timer, a.k.a. hrtimer, which KVM keeps running even on exits to userspace, the HV timer only runs while the guest is active. As a result, if the vCPU does not run for an extended duration, there will be a huge gap between the target expiration and the current time the vCPU resumes running. Because the target expiration is incremented by only one period on each timer expiration, this leads to a series of timer expirations occurring rapidly after the vCPU/VM resumes.

More critically, when the vCPU first triggers a periodic HV timer expiration after resuming, advancing the expiration by only one period will result in a target expiration in the past. As a result, the delta may be calculated as a negative value. When the delta is converted into an absolute value (tscdeadline is an unsigned u64), the resulting value can overflow what the HV timer is capable of programming. I.e. the large value will exceed the VMX Preemption Timer's maximum bit width of cpu_preemption_timer_multi + 32, and thus cause KVM to switch from the HV timer to the software timer (hrtimers).

After switching to the software timer, periodic timer expiration callbacks may be executed consecutively within a single clock interrupt handler, because hrtimers honors KVM's request for an expiration in the past and immediately re-invokes KVM's callback after reprogramming. And because the interrupt handler runs with IRQs disabled, restarting KVM's hrtimer over and over until the target expiration is advanced to "now" can result in a hard lockup.

E.g. the following hard lockup was triggered in the host when running a Windows VM (only relevant because it used the APIC timer in periodic mode) after resuming the VM from a long suspend (in the host).

NMI watchdog: Watchdog detected hard LOCKUP on cpu 45
...

  RIP: 0010:advance_periodic_target_expiration+0x4d/0x80 [kvm]
  ...
  RSP: 0018:ff4f88f5d98d8ef0 EFLAGS: 00000046
  RAX: fff0103f91be678e RBX: fff0103f91be678e RCX: 00843a7d9e127bcc
  RDX: 0000000000000002 RSI: 0052ca4003697505 RDI: ff440d5bfbdbd500
  RBP: ff440d5956f99200 R08: ff2ff2a42deb6a84 R09: 000000000002a6c0
  R10: 0122d794016332b3 R11: 0000000000000000 R12: ff440db1af39cfc0
  R13: ff440db1af39cfc0 R14: ffffffffc0d4a560 R15: ff440db1af39d0f8
  FS:  00007f04a6ffd700(0000) GS:ff440db1af380000(0000) knlGS:000000e38a3b8000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 000000d5651feff8 CR3: 000000684e038002 CR4: 0000000000773ee0
  PKRU: 55555554
  Call Trace:
   <IRQ>
   apic_timer_fn+0x31/0x50 [kvm]
   __hrtimer_run_queues+0x100/0x280
   hrtimer_interrupt+0x100/0x210
   ? ttwu_do_wakeup+0x19/0x160
   smp_apic_timer_interrupt+0x6a/0x130
   apic_timer_interrupt+0xf/0x20
   </IRQ>

Moreover, if the suspend duration of the virtual machine is not long enough to trigger a hard lockup in this scenario, since commit 98c25ead5eda ("KVM: VMX: Move preemption timer <=> hrtimer dance to common x86"), KVM will continue using the software timer until the guest reprograms the APIC timer in some way. Since the periodic timer does not require frequent APIC timer register programming, the guest may continue to use the software timer in perpetuity.

Fixes d8f2f498d9ed ("x86/kvm: fix LAPIC timer drift when guest uses periodic mode")
[sean: massage comments and changelog]
---
 arch/x86/kvm/lapic.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
arch/x86/kvm/lapic.c +23 -5
index a5c927e7bae6..8b6ec3304100 100644
@@ -2131,15 +2131,33 @@ static void advance_periodic_target_expiration(struct kvm_lapic *apic)
 	ktime_t delta;
 
 	/*
-	 * Synchronize both deadlines to the same time source or
-	 * differences in the periods (caused by differences in the
-	 * underlying clocks or numerical approximation errors) will
-	 * cause the two to drift apart over time as the errors
-	 * accumulate.
+	 * Use kernel time as the time source for both the hrtimer deadline and
+	 * TSC-based deadline so that they stay synchronized.  Computing each
+	 * deadline independently will cause the two deadlines to drift apart
+	 * over time as differences in the periods accumulate, e.g. due to
+	 * differences in the underlying clocks or numerical approximation errors.
 	 */
 	apic->lapic_timer.target_expiration =
 		ktime_add_ns(apic->lapic_timer.target_expiration,
 				apic->lapic_timer.period);
+
+	/*
+	 * If the new expiration is in the past, e.g. because userspace stopped
+	 * running the VM for an extended duration, then force the expiration
+	 * to "now" and don't try to play catch-up with the missed events.  KVM
+	 * will only deliver a single interrupt regardless of how many events
+	 * are pending, i.e. restarting the timer with an expiration in the
+	 * past will do nothing more than waste host cycles, and can even lead
+	 * to a hard lockup in extreme cases.
+	 */
+	if (ktime_before(apic->lapic_timer.target_expiration, now))
+		apic->lapic_timer.target_expiration = now;
+
+	/*
+	 * Note, ensuring the expiration isn't in the past also prevents delta
+	 * from going negative, which could cause the TSC deadline to become
+	 * excessively large due to it an unsigned value.
+	 */
 	delta = ktime_sub(apic->lapic_timer.target_expiration, now);
 	apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) +
 		nsec_to_cycles(apic->vcpu, delta);
Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
author maintainer
#4 lore raw

PATCH v6 4/4 KVM: x86: Grab lapic_timer in a local variable to cleanup periodic code

In reply to Sean Christopherson

Stash apic->lapic_timer in a local "ktimer" variable in advance_periodic_target_expiration() to eliminate a few unaligned wraps, and to make the code easier to read overall.

No functional change intended.

---
 arch/x86/kvm/lapic.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/kvm/lapic.c +8 -8
index 8b6ec3304100..1597dd0b0cc6 100644
@@ -2126,6 +2126,7 @@ static bool set_target_expiration(struct kvm_lapic *apic, u32 count_reg)
 
 static void advance_periodic_target_expiration(struct kvm_lapic *apic)
 {
+	struct kvm_timer *ktimer = &apic->lapic_timer;
 	ktime_t now = ktime_get();
 	u64 tscl = rdtsc();
 	ktime_t delta;
@@ -2137,9 +2138,8 @@ static void advance_periodic_target_expiration(struct kvm_lapic *apic)
 	 * over time as differences in the periods accumulate, e.g. due to
 	 * differences in the underlying clocks or numerical approximation errors.
 	 */
-	apic->lapic_timer.target_expiration =
-		ktime_add_ns(apic->lapic_timer.target_expiration,
-				apic->lapic_timer.period);
+	ktimer->target_expiration = ktime_add_ns(ktimer->target_expiration,
+						 ktimer->period);
 
 	/*
 	 * If the new expiration is in the past, e.g. because userspace stopped
@@ -2150,17 +2150,17 @@ static void advance_periodic_target_expiration(struct kvm_lapic *apic)
 	 * past will do nothing more than waste host cycles, and can even lead
 	 * to a hard lockup in extreme cases.
 	 */
-	if (ktime_before(apic->lapic_timer.target_expiration, now))
-		apic->lapic_timer.target_expiration = now;
+	if (ktime_before(ktimer->target_expiration, now))
+		ktimer->target_expiration = now;
 
 	/*
 	 * Note, ensuring the expiration isn't in the past also prevents delta
 	 * from going negative, which could cause the TSC deadline to become
 	 * excessively large due to it an unsigned value.
 	 */
-	delta = ktime_sub(apic->lapic_timer.target_expiration, now);
-	apic->lapic_timer.tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) +
-		nsec_to_cycles(apic->vcpu, delta);
+	delta = ktime_sub(ktimer->target_expiration, now);
+	ktimer->tscdeadline = kvm_read_l1_tsc(apic->vcpu, tscl) +
+			      nsec_to_cycles(apic->vcpu, delta);
 }
 
 static void start_sw_period(struct kvm_lapic *apic)
Signature · 1 line
2.52.0.rc1.455.g30608eb744-goog
author maintainer Applied
#5 lore raw
In reply to Sean Christopherson
Quoted 4 lines · Sean Christopherson

fuqiang's patch/series to fix a bug in KVM's local APIC timer emulation where it can trigger a hard lockup due to restarting an hrtimer with an expired deadline over and over (and over).

Quoted 8 lines

v6:
- Split the apic_timer_fn() change to a separate patch (mainly for a bisection point).
- Handle (and WARN on) period=0 in apic_timer_fn().
- Add a patch to grab a pointer to the kvm_timer struct locally.
- Tag the fixes (and prep work) for stable@.

[...]

Applied to kvm-x86 misc, thanks!

[1/4] KVM: x86: WARN if hrtimer callback for periodic APIC timer fires with period=0

      https://github.com/kvm-x86/linux/commit/0ea9494be9c9
[2/4] KVM: x86: Explicitly set new periodic hrtimer expiration in apic_timer_fn()
      https://github.com/kvm-x86/linux/commit/9633f180ce99
[3/4] KVM: x86: Fix VM hard lockup after prolonged inactivity with periodic HV timer
      https://github.com/kvm-x86/linux/commit/18ab3fc8e880
[4/4] KVM: x86: Grab lapic_timer in a local variable to cleanup periodic code
      https://github.com/kvm-x86/linux/commit/a091fe60c2d3

--
https://github.com/kvm-x86/linux/tree/next