Skip to contentSkip to conversation

KVM: x86: Revert to fix apic_hw_disabled underflow

PATCH Applied Sean Christopherson 2 participants 4 messages Oct 13, 2021 – Oct 15, 2021

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

cover KVM: x86: Revert to fix apic_hw_disabled underflow

Revert (mostly) a patch from the vCPU RESET cleanup that open coded some APIC shenanigans to avoid stuffing vcpu->arch.apic_base at vCPU creation, and completely overlooked the side effects on apic_hw_disabled. I went for a revert as I think the original behavior is the least awful solution, just somewhat poorly documented.

The second patch adds WARNs to detect "overflow", where "overflow" means KVM incorrectly increments apic_hw_disabled.

Sean Christopherson (2):
Revert "KVM: x86: Open code necessary bits of kvm_lapic_set_base() at
vCPU RESET"
KVM: x86: WARN if APIC HW/SW disable static keys are non-zero on
unload

 arch/x86/kvm/lapic.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
author maintainer
#1 lore raw

PATCH 1/2 Revert "KVM: x86: Open code necessary bits of kvm_lapic_set_base() at vCPU RESET"

Revert a change to open code bits of kvm_lapic_set_base() when emulating APIC RESET to fix an apic_hw_disabled underflow bug due to arch.apic_base and apic_hw_disabled being unsyncrhonized when the APIC is created. If kvm_arch_vcpu_create() fails after creating the APIC, kvm_free_lapic() will see the initialized-to-zero vcpu->arch.apic_base and decrement apic_hw_disabled without KVM ever having incremented apic_hw_disabled.

Using kvm_lapic_set_base() in kvm_lapic_reset() is also desirable for a potential future where KVM supports RESET outside of vCPU creation, in which case all the side effects of kvm_lapic_set_base() are needed, e.g. to handle the transition from x2APIC => xAPIC.

Alternatively, KVM could temporarily increment apic_hw_disabled (and call kvm_lapic_set_base() at RESET), but that's a waste of cycles and would impact the performance of other vCPUs and VMs. The other subtle side effect is that updating the xAPIC ID needs to be done at RESET regardless of whether the APIC was previously enabled, i.e. kvm_lapic_reset() needs an explicit call to kvm_apic_set_xapic_id() regardless of whether or not kvm_lapic_set_base() also performs the update. That makes stuffing the enable bit at vCPU creation slightly more palatable, as doing so affects only the apic_hw_disabled key.

Opportunistically tweak the comment to explicitly call out the connection between vcpu->arch.apic_base and apic_hw_disabled, and add a comment to call out the need to always do kvm_apic_set_xapic_id() at RESET.

Underflow scenario:

  kvm_vm_ioctl() {
    kvm_vm_ioctl_create_vcpu() {
      kvm_arch_vcpu_create() {
        if (something_went_wrong)
          goto fail_free_lapic;
        /* vcpu->arch.apic_base is initialized when something_went_wrong is false. */
        kvm_vcpu_reset() {
          kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event) {
            vcpu->arch.apic_base = APIC_DEFAULT_PHYS_BASE | MSR_IA32_APICBASE_ENABLE;
          }
        }
        return 0;
      fail_free_lapic:
        kvm_free_lapic() {
          /* vcpu->arch.apic_base is not yet initialized when something_went_wrong is true. */
          if (!(vcpu->arch.apic_base & MSR_IA32_APICBASE_ENABLE))
            static_branch_slow_dec_deferred(&apic_hw_disabled); // <= underflow bug.
        }
        return r;
      }
    }
  }

This (mostly) reverts commit 421221234ada41b4a9f0beeb08e30b07388bd4bd.

Fixes 421221234ada ("KVM: x86: Open code necessary bits of kvm_lapic_set_base() at vCPU RESET")
Reported-by [email protected]
---
 arch/x86/kvm/lapic.c | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
arch/x86/kvm/lapic.c +11 -7
index 76fb00921203..7af25304bda9 100644
@@ -2321,13 +2321,14 @@ EXPORT_SYMBOL_GPL(kvm_apic_update_apicv);
 void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
 {
 	struct kvm_lapic *apic = vcpu->arch.apic;
+	u64 msr_val;
 	int i;
 
 	if (!init_event) {
-		vcpu->arch.apic_base = APIC_DEFAULT_PHYS_BASE |
-				       MSR_IA32_APICBASE_ENABLE;
+		msr_val = APIC_DEFAULT_PHYS_BASE | MSR_IA32_APICBASE_ENABLE;
 		if (kvm_vcpu_is_reset_bsp(vcpu))
-			vcpu->arch.apic_base |= MSR_IA32_APICBASE_BSP;
+			msr_val |= MSR_IA32_APICBASE_BSP;
+		kvm_lapic_set_base(vcpu, msr_val);
 	}
 
 	if (!apic)
@@ -2336,11 +2337,9 @@ void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event)
 	/* Stop the timer in case it's a reset to an active apic */
 	hrtimer_cancel(&apic->lapic_timer.timer);
 
-	if (!init_event) {
-		apic->base_address = APIC_DEFAULT_PHYS_BASE;
-
+	/* The xAPIC ID is set at RESET even if the APIC was already enabled. */
+	if (!init_event)
 		kvm_apic_set_xapic_id(apic, vcpu->vcpu_id);
-	}
 	kvm_apic_set_version(apic->vcpu);
 
 	for (i = 0; i < KVM_APIC_LVT_NUM; i++)
@@ -2481,6 +2480,11 @@ int kvm_create_lapic(struct kvm_vcpu *vcpu, int timer_advance_ns)
 		lapic_timer_advance_dynamic = false;
 	}
 
+	/*
+	 * Stuff the APIC ENABLE bit in lieu of temporarily incrementing
+	 * apic_hw_disabled; the full RESET value is set by kvm_lapic_reset().
+	 */
+	vcpu->arch.apic_base = MSR_IA32_APICBASE_ENABLE;
 	static_branch_inc(&apic_sw_disabled.key); /* sw disabled at reset */
 	kvm_iodevice_init(&apic->dev, &apic_mmio_ops);
 
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
author maintainer
#2 lore raw

PATCH 2/2 KVM: x86: WARN if APIC HW/SW disable static keys are non-zero on unload

In reply to Sean Christopherson

WARN if the static keys used to track if any vCPU has disabled its APIC are left elevated at module exit. Unlike the underflow case, nothing in the static key infrastructure will complain if a key is left elevated, and because an elevated key only affects performance, nothing in KVM will fail if either ey is improperly incremented.

---
 arch/x86/kvm/lapic.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/lapic.c +2 -0
index 7af25304bda9..d6ac32f3f650 100644
@@ -2946,5 +2946,7 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu)
 void kvm_lapic_exit(void)
 {
 	static_key_deferred_flush(&apic_hw_disabled);
+	WARN_ON(static_branch_unlikely(&apic_hw_disabled.key));
 	static_key_deferred_flush(&apic_sw_disabled);
+	WARN_ON(static_branch_unlikely(&apic_sw_disabled.key));
 }
Signature · 1 line
2.33.0.1079.g6e70778dc9-goog
maintainer Applied
#3 lore raw
In reply to Sean Christopherson
Quoted 9 lines · Sean Christopherson

Revert (mostly) a patch from the vCPU RESET cleanup that open coded some APIC shenanigans to avoid stuffing vcpu->arch.apic_base at vCPU creation, and completely overlooked the side effects on apic_hw_disabled. I went for a revert as I think the original behavior is the least awful solution, just somewhat poorly documented.

The second patch adds WARNs to detect "overflow", where "overflow" means KVM incorrectly increments apic_hw_disabled.

Quoted 9 lines

Sean Christopherson (2):
Revert "KVM: x86: Open code necessary bits of kvm_lapic_set_base() at
vCPU RESET"
KVM: x86: WARN if APIC HW/SW disable static keys are non-zero on
unload

  arch/x86/kvm/lapic.c | 20 +++++++++++++-------
  1 file changed, 13 insertions(+), 7 deletions(-)

Queued for 5.15, thanks.

Paolo