Skip to contentSkip to conversation

KVM: x86: Synthesize at most one PMI per VM-exit

PATCH In review Jim Mattson 6 participants 31 messages Sep 1, 2023 – Sep 25, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 KVM: x86: Synthesize at most one PMI per VM-exit

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")
---
 arch/x86/include/asm/kvm_host.h |  1 -
 arch/x86/kvm/pmu.c              | 27 +--------------------------
 arch/x86/kvm/x86.c              |  3 +++
 3 files changed, 4 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 3bc146dfd38d..f6b9e3ae08bf 100644
@@ -528,7 +528,6 @@ struct kvm_pmu {
 	u64 raw_event_mask;
 	struct kvm_pmc gp_counters[KVM_INTEL_PMC_MAX_GENERIC];
 	struct kvm_pmc fixed_counters[KVM_PMC_MAX_FIXED];
-	struct irq_work irq_work;
 
 	/*
 	 * Overlay the bitmap with a 64-bit atomic so that all bits can be
arch/x86/kvm/pmu.c +1 -26
index bf653df86112..0c117cd24077 100644
@@ -93,14 +93,6 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops)
 #undef __KVM_X86_PMU_OP
 }
 
-static void kvm_pmi_trigger_fn(struct irq_work *irq_work)
-{
-	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
-	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
-
-	kvm_pmu_deliver_pmi(vcpu);
-}
-
 static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
 {
 	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -124,20 +116,7 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
 		__set_bit(pmc->idx, (unsigned long *)&pmu->global_status);
 	}
 
-	if (!pmc->intr || skip_pmi)
-		return;
-
-	/*
-	 * Inject PMI. If vcpu was in a guest mode during NMI PMI
-	 * can be ejected on a guest mode re-entry. Otherwise we can't
-	 * be sure that vcpu wasn't executing hlt instruction at the
-	 * time of vmexit and is not going to re-enter guest mode until
-	 * woken up. So we should wake it, but this is impossible from
-	 * NMI context. Do it from irq work instead.
-	 */
-	if (in_pmi && !kvm_handling_nmi_from_guest(pmc->vcpu))
-		irq_work_queue(&pmc_to_pmu(pmc)->irq_work);
-	else
+	if (pmc->intr && !skip_pmi)
 		kvm_make_request(KVM_REQ_PMI, pmc->vcpu);
 }
 
@@ -677,9 +656,6 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
 
 void kvm_pmu_reset(struct kvm_vcpu *vcpu)
 {
-	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
-
-	irq_work_sync(&pmu->irq_work);
 	static_call(kvm_x86_pmu_reset)(vcpu);
 }
 
@@ -689,7 +665,6 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
 
 	memset(pmu, 0, sizeof(*pmu));
 	static_call(kvm_x86_pmu_init)(vcpu);
-	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
 	pmu->event_count = 0;
 	pmu->need_cleanup = false;
 	kvm_pmu_refresh(vcpu);
arch/x86/kvm/x86.c +3 -0
index c381770bcbf1..0732c09fbd2d 100644
@@ -12841,6 +12841,9 @@ static inline bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
 		return true;
 #endif
 
+	if (kvm_test_request(KVM_REQ_PMI, vcpu))
+		return true;
+
 	if (kvm_arch_interrupt_allowed(vcpu) &&
 	    (kvm_cpu_has_interrupt(vcpu) ||
 	    kvm_guest_apic_has_interrupt(vcpu)))
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
#1 lore raw

PATCH 2/2 KVM: x86: Mask LVTPC when handling a PMI

Per the SDM, "When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register."

Add this behavior to KVM's local APIC emulation, to reduce the incidence of "dazed and confused" spurious NMI warnings in Linux guests (at least, those that use a PMI handler with "late_ack").

Fixes 23930f9521c9 ("KVM: x86: Enable NMI Watchdog via in-kernel PIT source")
---
 arch/x86/kvm/lapic.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/lapic.c +2 -0
index a983a16163b1..1a79ec54ae1e 100644
@@ -2743,6 +2743,8 @@ int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type)
 		vector = reg & APIC_VECTOR_MASK;
 		mode = reg & APIC_MODE_MASK;
 		trig_mode = reg & APIC_LVT_LEVEL_TRIGGER;
+		if (lvt_type == APIC_LVTPC)
+			kvm_lapic_set_reg(apic, lvt_type, reg | APIC_LVT_MASKED);
 		return __apic_accept_irq(apic, mode, vector, 1, trig_mode,
 					NULL);
 	}
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
Tested-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] KVM: x86: Mask LVTPC when handling a PMI

Quoted 10 lines · Jim Mattson

Per the SDM, "When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register."

Add this behavior to KVM's local APIC emulation, to reduce the incidence of "dazed and confused" spurious NMI warnings in Linux guests (at least, those that use a PMI handler with "late_ack").

Fixes 23930f9521c9 ("KVM: x86: Enable NMI Watchdog via in-kernel PIT source")

I see consistent number of PMIs and NMIs when running perf on an idle VM.

Quoted 20 lines
---
 arch/x86/kvm/lapic.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/lapic.c +2 -0
index a983a16163b1..1a79ec54ae1e 100644
@@ -2743,6 +2743,8 @@ int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type)
 		vector = reg & APIC_VECTOR_MASK;
 		mode = reg & APIC_MODE_MASK;
 		trig_mode = reg & APIC_LVT_LEVEL_TRIGGER;
+		if (lvt_type == APIC_LVTPC)
+			kvm_lapic_set_reg(apic, lvt_type, reg | APIC_LVT_MASKED);
 		return __apic_accept_irq(apic, mode, vector, 1, trig_mode,
 					NULL);
 	}
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
Reviewed-by Dapeng Mi on v1 2/2
#3 lore raw

RE: [PATCH 2/2] KVM: x86: Mask LVTPC when handling a PMI

In reply to Jim Mattson
Quoted 29 lines

-----Original Message-----
From: Jim Mattson <[email protected]>
Sent: Saturday, September 2, 2023 2:57 AM
To: [email protected]; Christopherson,, Sean <[email protected]>; Paolo Bonzini <[email protected]>
Cc: Xu, Like <[email protected]>; Mingwei Zhang <[email protected]>; Roman Kagan <[email protected]>; Liang, Kan <[email protected]>; Mi, Dapeng1 <[email protected]>; Jim Mattson <[email protected]>
Subject: [PATCH 2/2] KVM: x86: Mask LVTPC when handling a PMI

Per the SDM, "When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register."

Add this behavior to KVM's local APIC emulation, to reduce the incidence of "dazed and confused" spurious NMI warnings in Linux guests (at least, those that use a PMI handler with "late_ack").

Fixes 23930f9521c9 ("KVM: x86: Enable NMI Watchdog via in-kernel PIT source")
---
 arch/x86/kvm/lapic.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

a983a16163b1..1a79ec54ae1e 100644

arch/x86/kvm/lapic.c int kvm_apic_local_deliver(struct kvm_lapic *apic, int · line 2749

Quoted 7 lines
arch/x86/kvm/lapic.c +2 -0
@@ -2743,6 +2743,8 @@ int kvm_apic_local_deliver(struct kvm_lapic *apic, int
 lvt_type)
 		vector = reg & APIC_VECTOR_MASK;
 		mode = reg & APIC_MODE_MASK;
 		trig_mode = reg & APIC_LVT_LEVEL_TRIGGER;
+		if (lvt_type == APIC_LVTPC)
+			kvm_lapic_set_reg(apic, lvt_type, reg |
 APIC_LVT_MASKED);
Quoted 5 lines
arch/x86/kvm/lapic.c +0 -0
 		return __apic_accept_irq(apic, mode, vector, 1, trig_mode,
arch/x86/kvm/lapic.c +0 -1
 					NULL);
 	}
--

2.42.0.283.g2d96d420d3-goog

#4 lore raw

Re: [PATCH 2/2] KVM: x86: Mask LVTPC when handling a PMI

In reply to Jim Mattson
Quoted 7 lines · Jim Mattson

Per the SDM, "When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register."

Add this behavior to KVM's local APIC emulation, to reduce the incidence of "dazed and confused" spurious NMI warnings in Linux guests (at least, those that use a PMI handler with "late_ack").

Hmm, I don't like the "to reduce the incidence" language as that suggests that this isn't a hard requirement. That makes it sound like KVM is doing the guest a favor. This?

Per the SDM, "When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register." Add this behavior to KVM's local APIC emulation.

Failure to mask the LVTPC entry results in spurious PMIs, e.g. when running Linux as a guest, PMI handlers that do a "late_ack" spew a large number of "dazed and confused" spurious NMI warnings.

Quoted 11 lines
Fixes 23930f9521c9 ("KVM: x86: Enable NMI Watchdog via in-kernel PIT source")
---
 arch/x86/kvm/lapic.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kvm/lapic.c +0 -0
index a983a16163b1..1a79ec54ae1e 100644

arch/x86/kvm/lapic.c int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type) · line 2747

Quoted 5 lines
arch/x86/kvm/lapic.c +2 -0
@@ -2743,6 +2743,8 @@ int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type)
 		vector = reg & APIC_VECTOR_MASK;
 		mode = reg & APIC_MODE_MASK;
 		trig_mode = reg & APIC_LVT_LEVEL_TRIGGER;
+		if (lvt_type == APIC_LVTPC)
+			kvm_lapic_set_reg(apic, lvt_type, reg | APIC_LVT_MASKED);

IMO, unconditionally setting the masked bit is wrong. I'm 99% certain KVM should only set the mask bit if an interrupt is actually delivered somehwere.

__apic_accept_irq() "fails" as follows:

APIC_DM_LOWEST and APIC_DM_FIXED
1. Trigger Mode is "level" triggered and level is deasserted. This can't happen because this code hardcodes the level to '1'.

2. APIC is disabled (H/W or S/W). This is a non-issue because the H/W disabled case ignores it entirely, and all masks are defined to be set if the APIC is S/W disabled (per the SDM):

The mask bits for all the LVT entries are set. Attempts to reset these bits will be ignored.

APIC_DM_SMI
1. If SMI injection fails because CONFIG_KVM_SMM=n.

APIC_DM_INIT
1. Trigger Mode is "level" triggered and level is deasserted. As above, this can't happen.

APIC_DM_EXTINT
1. Unconditionally ignored by KVM. This is architecturally correct for the LVTPC as the SDM says:

Not supported for the LVT CMCI register, the LVT thermal monitor register, or the LVT performance counter register.

So basically, failure happens if and only if the guest attempts to send an SMI or ExtINT that KVM ignores. The SDM doesn't explicitly state that mask bit is left unset in these cases, but my reading of

When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register.

is that there has to be an actual interrupt.

I highly doubt any software will ever care, e.g. the guest is going to be unhappy in CONFIG_KVM_SMM=n case no matter what, but setting the mask bit without actually triggering an interrupt seems odd.

This as fixup?

arch/x86/kvm/lapic.c +5 -3
index 71d87b0db0d9..ebfc3d92a266 100644
@@ -2759,15 +2759,17 @@ int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type)
 {
        u32 reg = kvm_lapic_get_reg(apic, lvt_type);
        int vector, mode, trig_mode;
+       int r;
 
        if (kvm_apic_hw_enabled(apic) && !(reg & APIC_LVT_MASKED)) {
                vector = reg & APIC_VECTOR_MASK;
                mode = reg & APIC_MODE_MASK;
                trig_mode = reg & APIC_LVT_LEVEL_TRIGGER;
-               if (lvt_type == APIC_LVTPC)
+
+               r = __apic_accept_irq(apic, mode, vector, 1, trig_mode, NULL);
+               if (r && lvt_type == APIC_LVTPC)
                        kvm_lapic_set_reg(apic, lvt_type, reg | APIC_LVT_MASKED);
-               return __apic_accept_irq(apic, mode, vector, 1, trig_mode,
-                                       NULL);
+               return r;
        }
        return 0;
 }
#5 lore raw

Re: [PATCH 2/2] KVM: x86: Mask LVTPC when handling a PMI

Quoted 7 lines · Sean Christopherson

The SDM doesn't explicitly state that mask bit is left unset in these cases, but my reading of

When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register.

is that there has to be an actual interrupt.

I assume you mean an actual interrupt from the APIC, not an actual PMI to the APIC.

I would argue that one way of "handling" a performance-monitoring counters interrupt is to do nothing with it, but apparently I'm wrong. At least, if I set the delivery mode in LVTPC to the illegal value of 6, I never see the LVTPC.MASK bit get set.

#6 lore raw

Re: [PATCH 2/2] KVM: x86: Mask LVTPC when handling a PMI

Quoted 11 lines · Jim Mattson
Quoted 7 lines · Sean Christopherson

The SDM doesn't explicitly state that mask bit is left unset in these cases, but my reading of

When the local APIC handles a performance-monitoring counters interrupt, it automatically sets the mask flag in the LVT performance counter register.

is that there has to be an actual interrupt.

I assume you mean an actual interrupt from the APIC, not an actual PMI to the APIC.

Yeah.

Quoted 4 lines

I would argue that one way of "handling" a performance-monitoring counters interrupt is to do nothing with it, but apparently I'm wrong. At least, if I set the delivery mode in LVTPC to the illegal value of 6, I never see the LVTPC.MASK bit get set.

Heh, you could complain to Intel and see if they'll change "handles" to "delivers" :-)

Tested-by on v1 1/2
#7 lore raw
In reply to Jim Mattson
Quoted 19 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 6 lines

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")
Quoted 94 lines
---
 arch/x86/include/asm/kvm_host.h |  1 -
 arch/x86/kvm/pmu.c              | 27 +--------------------------
 arch/x86/kvm/x86.c              |  3 +++
 3 files changed, 4 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 3bc146dfd38d..f6b9e3ae08bf 100644
@@ -528,7 +528,6 @@ struct kvm_pmu {
 	u64 raw_event_mask;
 	struct kvm_pmc gp_counters[KVM_INTEL_PMC_MAX_GENERIC];
 	struct kvm_pmc fixed_counters[KVM_PMC_MAX_FIXED];
-	struct irq_work irq_work;
 
 	/*
 	 * Overlay the bitmap with a 64-bit atomic so that all bits can be
arch/x86/kvm/pmu.c +1 -26
index bf653df86112..0c117cd24077 100644
@@ -93,14 +93,6 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops)
 #undef __KVM_X86_PMU_OP
 }
 
-static void kvm_pmi_trigger_fn(struct irq_work *irq_work)
-{
-	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
-	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
-
-	kvm_pmu_deliver_pmi(vcpu);
-}
-
 static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
 {
 	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -124,20 +116,7 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
 		__set_bit(pmc->idx, (unsigned long *)&pmu->global_status);
 	}
 
-	if (!pmc->intr || skip_pmi)
-		return;
-
-	/*
-	 * Inject PMI. If vcpu was in a guest mode during NMI PMI
-	 * can be ejected on a guest mode re-entry. Otherwise we can't
-	 * be sure that vcpu wasn't executing hlt instruction at the
-	 * time of vmexit and is not going to re-enter guest mode until
-	 * woken up. So we should wake it, but this is impossible from
-	 * NMI context. Do it from irq work instead.
-	 */
-	if (in_pmi && !kvm_handling_nmi_from_guest(pmc->vcpu))
-		irq_work_queue(&pmc_to_pmu(pmc)->irq_work);
-	else
+	if (pmc->intr && !skip_pmi)
 		kvm_make_request(KVM_REQ_PMI, pmc->vcpu);
 }
 
@@ -677,9 +656,6 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
 
 void kvm_pmu_reset(struct kvm_vcpu *vcpu)
 {
-	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
-
-	irq_work_sync(&pmu->irq_work);
 	static_call(kvm_x86_pmu_reset)(vcpu);
 }
 
@@ -689,7 +665,6 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
 
 	memset(pmu, 0, sizeof(*pmu));
 	static_call(kvm_x86_pmu_init)(vcpu);
-	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
 	pmu->event_count = 0;
 	pmu->need_cleanup = false;
 	kvm_pmu_refresh(vcpu);
arch/x86/kvm/x86.c +3 -0
index c381770bcbf1..0732c09fbd2d 100644
@@ -12841,6 +12841,9 @@ static inline bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
 		return true;
 #endif
 
+	if (kvm_test_request(KVM_REQ_PMI, vcpu))
+		return true;
+
 	if (kvm_arch_interrupt_allowed(vcpu) &&
 	    (kvm_cpu_has_interrupt(vcpu) ||
 	    kvm_guest_apic_has_interrupt(vcpu)))
Signature · 1 line
2.42.0.283.g2d96d420d3-goog
#8 lore raw
In reply to Jim Mattson
Quoted 4 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

Do we have a way to reproduce this spurious NMI error constantly?

Quoted 112 lines

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")
---
  arch/x86/include/asm/kvm_host.h |  1 -
  arch/x86/kvm/pmu.c              | 27 +--------------------------
  arch/x86/kvm/x86.c              |  3 +++
  3 files changed, 4 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 3bc146dfd38d..f6b9e3ae08bf 100644
@@ -528,7 +528,6 @@ struct kvm_pmu {
  	u64 raw_event_mask;
  	struct kvm_pmc gp_counters[KVM_INTEL_PMC_MAX_GENERIC];
  	struct kvm_pmc fixed_counters[KVM_PMC_MAX_FIXED];
-	struct irq_work irq_work;
 
  	/*
  	 * Overlay the bitmap with a 64-bit atomic so that all bits can be
arch/x86/kvm/pmu.c +1 -26
index bf653df86112..0c117cd24077 100644
@@ -93,14 +93,6 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops)
  #undef __KVM_X86_PMU_OP
  }
 
-static void kvm_pmi_trigger_fn(struct irq_work *irq_work)
-{
-	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
-	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
-
-	kvm_pmu_deliver_pmi(vcpu);
-}
-
  static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
  {
  	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -124,20 +116,7 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
  		__set_bit(pmc->idx, (unsigned long *)&pmu->global_status);
  	}
 
-	if (!pmc->intr || skip_pmi)
-		return;
-
-	/*
-	 * Inject PMI. If vcpu was in a guest mode during NMI PMI
-	 * can be ejected on a guest mode re-entry. Otherwise we can't
-	 * be sure that vcpu wasn't executing hlt instruction at the
-	 * time of vmexit and is not going to re-enter guest mode until
-	 * woken up. So we should wake it, but this is impossible from
-	 * NMI context. Do it from irq work instead.
-	 */
-	if (in_pmi && !kvm_handling_nmi_from_guest(pmc->vcpu))
-		irq_work_queue(&pmc_to_pmu(pmc)->irq_work);
-	else
+	if (pmc->intr && !skip_pmi)
  		kvm_make_request(KVM_REQ_PMI, pmc->vcpu);
  }
 
@@ -677,9 +656,6 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
 
  void kvm_pmu_reset(struct kvm_vcpu *vcpu)
  {
-	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
-
-	irq_work_sync(&pmu->irq_work);
  	static_call(kvm_x86_pmu_reset)(vcpu);
  }
 
@@ -689,7 +665,6 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
 
  	memset(pmu, 0, sizeof(*pmu));
  	static_call(kvm_x86_pmu_init)(vcpu);
-	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
  	pmu->event_count = 0;
  	pmu->need_cleanup = false;
  	kvm_pmu_refresh(vcpu);
arch/x86/kvm/x86.c +3 -0
index c381770bcbf1..0732c09fbd2d 100644
@@ -12841,6 +12841,9 @@ static inline bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
  		return true;
  #endif
 
+	if (kvm_test_request(KVM_REQ_PMI, vcpu))
+		return true;
+
  	if (kvm_arch_interrupt_allowed(vcpu) &&
  	    (kvm_cpu_has_interrupt(vcpu) ||
  	    kvm_guest_apic_has_interrupt(vcpu)))
#9 lore raw
Quoted 9 lines · Dapeng
Quoted 4 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

Do we have a way to reproduce this spurious NMI error constantly?

I think I shared with you in another thread. I also shared the event list and command here:

https://lore.kernel.org/all/[email protected]/

To observe the spurious NMIs, you can just continously look at the NMIs/PMIs in /proc/interrupts and see if you have huge number within 2 minutes when running the above command in a 8-vcpu QEMU VM on Intel SPR machine. Huge here means more than 10K.

In addition, you may observe the following warning from kernel dmesg:

[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43.
[3939579.462836] Dazed and confused, but trying to continue
Quoted 105 lines
Quoted 105 lines

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")
---
  arch/x86/include/asm/kvm_host.h |  1 -
  arch/x86/kvm/pmu.c              | 27 +--------------------------
  arch/x86/kvm/x86.c              |  3 +++
  3 files changed, 4 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 3bc146dfd38d..f6b9e3ae08bf 100644
@@ -528,7 +528,6 @@ struct kvm_pmu {
  	u64 raw_event_mask;
  	struct kvm_pmc gp_counters[KVM_INTEL_PMC_MAX_GENERIC];
  	struct kvm_pmc fixed_counters[KVM_PMC_MAX_FIXED];
-	struct irq_work irq_work;
  	/*
  	 * Overlay the bitmap with a 64-bit atomic so that all bits can be
arch/x86/kvm/pmu.c +1 -26
index bf653df86112..0c117cd24077 100644
@@ -93,14 +93,6 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops)
  #undef __KVM_X86_PMU_OP
  }
-static void kvm_pmi_trigger_fn(struct irq_work *irq_work)
-{
-	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
-	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
-
-	kvm_pmu_deliver_pmi(vcpu);
-}
-
  static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
  {
  	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -124,20 +116,7 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
  		__set_bit(pmc->idx, (unsigned long *)&pmu->global_status);
  	}
-	if (!pmc->intr || skip_pmi)
-		return;
-
-	/*
-	 * Inject PMI. If vcpu was in a guest mode during NMI PMI
-	 * can be ejected on a guest mode re-entry. Otherwise we can't
-	 * be sure that vcpu wasn't executing hlt instruction at the
-	 * time of vmexit and is not going to re-enter guest mode until
-	 * woken up. So we should wake it, but this is impossible from
-	 * NMI context. Do it from irq work instead.
-	 */
-	if (in_pmi && !kvm_handling_nmi_from_guest(pmc->vcpu))
-		irq_work_queue(&pmc_to_pmu(pmc)->irq_work);
-	else
+	if (pmc->intr && !skip_pmi)
  		kvm_make_request(KVM_REQ_PMI, pmc->vcpu);
  }
@@ -677,9 +656,6 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
  void kvm_pmu_reset(struct kvm_vcpu *vcpu)
  {
-	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
-
-	irq_work_sync(&pmu->irq_work);
  	static_call(kvm_x86_pmu_reset)(vcpu);
  }
@@ -689,7 +665,6 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
  	memset(pmu, 0, sizeof(*pmu));
  	static_call(kvm_x86_pmu_init)(vcpu);
-	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
  	pmu->event_count = 0;
  	pmu->need_cleanup = false;
  	kvm_pmu_refresh(vcpu);
arch/x86/kvm/x86.c +3 -0
index c381770bcbf1..0732c09fbd2d 100644
@@ -12841,6 +12841,9 @@ static inline bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
  		return true;
  #endif
+	if (kvm_test_request(KVM_REQ_PMI, vcpu))
+		return true;
+
  	if (kvm_arch_interrupt_allowed(vcpu) &&
  	    (kvm_cpu_has_interrupt(vcpu) ||
  	    kvm_guest_apic_has_interrupt(vcpu)))
Tested-by
#10 lore raw
Quoted 19 lines · Mingwei Zhang
Quoted 8 lines · Dapeng
Quoted 4 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

Do we have a way to reproduce this spurious NMI error constantly?

I think I shared with you in another thread. I also shared the event list and command here:

https://lore.kernel.org/all/[email protected]/

To observe the spurious NMIs, you can just continously look at the NMIs/PMIs in /proc/interrupts and see if you have huge number within 2 minutes when running the above command in a 8-vcpu QEMU VM on Intel SPR machine. Huge here means more than 10K.

Quoted 4 lines

In addition, you may observe the following warning from kernel dmesg:

[3939579.462832] Uhhuh. NMI received for unknown reason 30 on CPU 43.
[3939579.462836] Dazed and confused, but trying to continue

Thanks. I run the perf command which you shared in a VM for 10 minutes on SPR, I indeed see the unknown NMI warning messages, but I didn't see the huge number NMI interrupt burst, instead the NMI number increased averagely and there is no a burst increase.

After applying this patchset, the unknown NMI warning is indeed gone.

Quoted 104 lines
Quoted 104 lines
Quoted 104 lines

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")
---
   arch/x86/include/asm/kvm_host.h |  1 -
   arch/x86/kvm/pmu.c              | 27 +--------------------------
   arch/x86/kvm/x86.c              |  3 +++
   3 files changed, 4 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 3bc146dfd38d..f6b9e3ae08bf 100644
@@ -528,7 +528,6 @@ struct kvm_pmu {
   	u64 raw_event_mask;
   	struct kvm_pmc gp_counters[KVM_INTEL_PMC_MAX_GENERIC];
   	struct kvm_pmc fixed_counters[KVM_PMC_MAX_FIXED];
-	struct irq_work irq_work;
   	/*
   	 * Overlay the bitmap with a 64-bit atomic so that all bits can be
arch/x86/kvm/pmu.c +1 -26
index bf653df86112..0c117cd24077 100644
@@ -93,14 +93,6 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops)
   #undef __KVM_X86_PMU_OP
   }
-static void kvm_pmi_trigger_fn(struct irq_work *irq_work)
-{
-	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
-	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
-
-	kvm_pmu_deliver_pmi(vcpu);
-}
-
   static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
   {
   	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -124,20 +116,7 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
   		__set_bit(pmc->idx, (unsigned long *)&pmu->global_status);
   	}
-	if (!pmc->intr || skip_pmi)
-		return;
-
-	/*
-	 * Inject PMI. If vcpu was in a guest mode during NMI PMI
-	 * can be ejected on a guest mode re-entry. Otherwise we can't
-	 * be sure that vcpu wasn't executing hlt instruction at the
-	 * time of vmexit and is not going to re-enter guest mode until
-	 * woken up. So we should wake it, but this is impossible from
-	 * NMI context. Do it from irq work instead.
-	 */
-	if (in_pmi && !kvm_handling_nmi_from_guest(pmc->vcpu))
-		irq_work_queue(&pmc_to_pmu(pmc)->irq_work);
-	else
+	if (pmc->intr && !skip_pmi)
   		kvm_make_request(KVM_REQ_PMI, pmc->vcpu);
   }
@@ -677,9 +656,6 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
   void kvm_pmu_reset(struct kvm_vcpu *vcpu)
   {
-	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
-
-	irq_work_sync(&pmu->irq_work);
   	static_call(kvm_x86_pmu_reset)(vcpu);
   }
@@ -689,7 +665,6 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
   	memset(pmu, 0, sizeof(*pmu));
   	static_call(kvm_x86_pmu_init)(vcpu);
-	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
   	pmu->event_count = 0;
   	pmu->need_cleanup = false;
   	kvm_pmu_refresh(vcpu);
arch/x86/kvm/x86.c +3 -0
index c381770bcbf1..0732c09fbd2d 100644
@@ -12841,6 +12841,9 @@ static inline bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
   		return true;
   #endif
+	if (kvm_test_request(KVM_REQ_PMI, vcpu))
+		return true;
+
   	if (kvm_arch_interrupt_allowed(vcpu) &&
   	    (kvm_cpu_has_interrupt(vcpu) ||
   	    kvm_guest_apic_has_interrupt(vcpu)))
#11 lore raw
In reply to Jim Mattson
Quoted 6 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to

As you said, that shouldn't be a problem.

Quoted 9 lines

automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Any obstruction issues on fixing in this direction ?

Quoted 98 lines

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

Fixes 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions")
---
  arch/x86/include/asm/kvm_host.h |  1 -
  arch/x86/kvm/pmu.c              | 27 +--------------------------
  arch/x86/kvm/x86.c              |  3 +++
  3 files changed, 4 insertions(+), 27 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
index 3bc146dfd38d..f6b9e3ae08bf 100644
@@ -528,7 +528,6 @@ struct kvm_pmu {
  	u64 raw_event_mask;
  	struct kvm_pmc gp_counters[KVM_INTEL_PMC_MAX_GENERIC];
  	struct kvm_pmc fixed_counters[KVM_PMC_MAX_FIXED];
-	struct irq_work irq_work;
 
  	/*
  	 * Overlay the bitmap with a 64-bit atomic so that all bits can be
arch/x86/kvm/pmu.c +1 -26
index bf653df86112..0c117cd24077 100644
@@ -93,14 +93,6 @@ void kvm_pmu_ops_update(const struct kvm_pmu_ops *pmu_ops)
  #undef __KVM_X86_PMU_OP
  }
 
-static void kvm_pmi_trigger_fn(struct irq_work *irq_work)
-{
-	struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work);
-	struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu);
-
-	kvm_pmu_deliver_pmi(vcpu);
-}
-
  static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
  {
  	struct kvm_pmu *pmu = pmc_to_pmu(pmc);
@@ -124,20 +116,7 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi)
  		__set_bit(pmc->idx, (unsigned long *)&pmu->global_status);
  	}
 
-	if (!pmc->intr || skip_pmi)
-		return;
-
-	/*
-	 * Inject PMI. If vcpu was in a guest mode during NMI PMI
-	 * can be ejected on a guest mode re-entry. Otherwise we can't
-	 * be sure that vcpu wasn't executing hlt instruction at the
-	 * time of vmexit and is not going to re-enter guest mode until
-	 * woken up. So we should wake it, but this is impossible from
-	 * NMI context. Do it from irq work instead.
-	 */
-	if (in_pmi && !kvm_handling_nmi_from_guest(pmc->vcpu))
-		irq_work_queue(&pmc_to_pmu(pmc)->irq_work);
-	else
+	if (pmc->intr && !skip_pmi)
  		kvm_make_request(KVM_REQ_PMI, pmc->vcpu);
  }
 
@@ -677,9 +656,6 @@ void kvm_pmu_refresh(struct kvm_vcpu *vcpu)
 
  void kvm_pmu_reset(struct kvm_vcpu *vcpu)
  {
-	struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
-
-	irq_work_sync(&pmu->irq_work);
  	static_call(kvm_x86_pmu_reset)(vcpu);
  }
 
@@ -689,7 +665,6 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu)
 
  	memset(pmu, 0, sizeof(*pmu));
  	static_call(kvm_x86_pmu_init)(vcpu);
-	init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn);
  	pmu->event_count = 0;
  	pmu->need_cleanup = false;
  	kvm_pmu_refresh(vcpu);
arch/x86/kvm/x86.c +3 -0
index c381770bcbf1..0732c09fbd2d 100644
@@ -12841,6 +12841,9 @@ static inline bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu)
  		return true;
  #endif
 
+	if (kvm_test_request(KVM_REQ_PMI, vcpu))
+		return true;
+
  	if (kvm_arch_interrupt_allowed(vcpu) &&
  	    (kvm_cpu_has_interrupt(vcpu) ||
  	    kvm_guest_apic_has_interrupt(vcpu)))
#12 lore raw
Quoted 9 lines · Like Xu
Quoted 6 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to

As you said, that shouldn't be a problem.

It's still a bug though, overflow should only happen once.

Quoted 9 lines
Quoted 9 lines

automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 5 lines
Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Any obstruction issues on fixing in this direction ?

No, patch 2/2 in this series fixes LVTPC masking bug. I haven't dug through all of this yet, but my gut reaction is that I'm very strongly in favor of not using irq_work just to ensure KVM kicks a vCPU out of HLT. That's just ridiculous.

#13 lore raw
In reply to Jim Mattson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

So, put this patch second, and replace the above paragraphs with something like?

Calling kvm_pmu_deliver_pmi() twice is unlikely to be problematic now that KVM sets the LVTPC mask bit when delivering a PMI. But using IRQ work to trigger the PMI is still broken, albeit very theoretically.

E.g. if the self-IPI to trigger IRQ work is be delayed long enough for the vCPU to be migrated to a different pCPU, then it's possible for kvm_pmi_trigger_fn() to race with the kvm_pmu_deliver_pmi() from KVM_REQ_PMI and still generate two PMIs.

KVM could set the mask bit using an atomic operation, but that'd just be piling on unnecessary code to workaround what is effectively a hack. The *only* reason KVM uses IRQ work is to ensure the PMI is treated as a wake event, e.g. if the vCPU just executed HLT.

Quoted 3 lines

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

#14 lore raw
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 19 lines

So, put this patch second, and replace the above paragraphs with something like?

Calling kvm_pmu_deliver_pmi() twice is unlikely to be problematic now that KVM sets the LVTPC mask bit when delivering a PMI. But using IRQ work to trigger the PMI is still broken, albeit very theoretically.

E.g. if the self-IPI to trigger IRQ work is be delayed long enough for the vCPU to be migrated to a different pCPU, then it's possible for kvm_pmi_trigger_fn() to race with the kvm_pmu_deliver_pmi() from KVM_REQ_PMI and still generate two PMIs.

KVM could set the mask bit using an atomic operation, but that'd just be piling on unnecessary code to workaround what is effectively a hack. The *only* reason KVM uses IRQ work is to ensure the PMI is treated as a wake event, e.g. if the vCPU just executed HLT.

Quoted 3 lines

Remove the irq_work callback for synthesizing a PMI, and all of the logic for invoking it. Instead, to prevent a vcpu from leaving C0 with a PMI pending, add a check for KVM_REQ_PMI to kvm_vcpu_has_events().

#15 lore raw
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

#16 lore raw
Quoted 38 lines · Sean Christopherson
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 5 lines

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Thanks.
-Mingwei

#17 lore raw
Quoted 50 lines · Mingwei Zhang
Quoted 38 lines · Sean Christopherson
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 5 lines

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Quoted 8 lines

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

#18 lore raw
Quoted 56 lines · Sean Christopherson
Quoted 54 lines · Mingwei Zhang
Quoted 38 lines · Sean Christopherson
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 5 lines

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

Quoted 7 lines
Quoted 4 lines

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

Let me step back.

We have the following problem when we run perf inside guest:

[ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3.
[ 1437.487330] Dazed and confused, but trying to continue

This means there are more NMIs that guest PMI could understand. So there are potentially two approaches to solve the problem: 1) fix the PMI injection issue: only one can be injected; 2) fix the code that causes the (incorrect) multiple PMI injection.

I am working on the 2nd one. So, the property of the 2nd one is:
without patches in 1) (Jim's patches), we could still avoid the above warning messages.

Thanks.
-Mingwei

#19 lore raw
Quoted 77 lines · Mingwei Zhang
Quoted 56 lines · Sean Christopherson
Quoted 54 lines · Mingwei Zhang
Quoted 38 lines · Sean Christopherson
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 5 lines

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

Quoted 7 lines
Quoted 4 lines

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

Let me step back.

We have the following problem when we run perf inside guest:

[ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3.
[ 1437.487330] Dazed and confused, but trying to continue

This means there are more NMIs that guest PMI could understand. So there are potentially two approaches to solve the problem: 1) fix the PMI injection issue: only one can be injected; 2) fix the code that causes the (incorrect) multiple PMI injection.

Quoted 6 lines

I am working on the 2nd one. So, the property of the 2nd one is:
without patches in 1) (Jim's patches), we could still avoid the above warning messages.

Thanks.
-Mingwei

This is my draft version. If you don't have full-width counter support, this patch needs be placed on top of this one:
https://lore.kernel.org/all/[email protected]/

My initial testing on both QEMU and our GCP testing environment shows no "Uhhuh..." dmesg in guest.

Please take a look...

From 47e629269d8b0ff65c242334f068300216cb7f91 Mon Sep 17 00:00:00 2001
From: Mingwei Zhang <[email protected]>
Date: Fri, 22 Sep 2023 17:13:55 +0000
Subject: [PATCH] KVM: x86/pmu: Fix emulated counter increment due to
instruction emulation

Fix KVM emulated counter increment due to instruction emulation. KVM pmc->counter is always a snapshot value when counter is running. Therefore, the value does not represent the actual value of counter. Thus it is inappropriate to compare it with other counter values. In existing code KVM directly compares pmc->prev_counter and pmc->counter directly. However, pmc->prev_counter is a snaphot value assigned from pmc->counter when counter may still be running. So this comparison logic in reprogram_counter() will generate incorrect invocations to __kvm_perf_overflow(in_pmi=false) and generate duplicated PMI injection requests.

Fix this issue by adding emulated_counter field and only the do the counter calculation after we pause

Change-Id: I2d59e68557fd35f7bbcfe09ea42ad81bd36776b7

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/pmu.c              | 15 ++++++++-------
 arch/x86/kvm/svm/pmu.c          |  1 +
 arch/x86/kvm/vmx/pmu_intel.c    |  2 ++
 4 files changed, 12 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 1a4def36d5bb..47bbfbc0aa35 100644
@@ -494,6 +494,7 @@ struct kvm_pmc {
 	bool intr;
 	u64 counter;
 	u64 prev_counter;
+	u64 emulated_counter;
 	u64 eventsel;
 	struct perf_event *perf_event;
 	struct kvm_vcpu *vcpu;
arch/x86/kvm/pmu.c +8 -7
index edb89b51b383..47acf3a2b077 100644
@@ -240,12 +240,13 @@ static void pmc_pause_counter(struct kvm_pmc *pmc)
 {
 	u64 counter = pmc->counter;
 
-	if (!pmc->perf_event || pmc->is_paused)
-		return;
-
 	/* update counter, reset event value to avoid redundant accumulation */
-	counter += perf_event_pause(pmc->perf_event, true);
-	pmc->counter = counter & pmc_bitmask(pmc);
+	if (pmc->perf_event && !pmc->is_paused)
+		counter += perf_event_pause(pmc->perf_event, true);
+
+	pmc->prev_counter = counter & pmc_bitmask(pmc);
+	pmc->counter = (counter + pmc->emulated_counter) & pmc_bitmask(pmc);
+	pmc->emulated_counter = 0;
 	pmc->is_paused = true;
 }
 
@@ -452,6 +453,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)
 reprogram_complete:
 	clear_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->reprogram_pmi);
 	pmc->prev_counter = 0;
+	pmc->emulated_counter = 0;
 }
 
 void kvm_pmu_handle_event(struct kvm_vcpu *vcpu)
@@ -725,8 +727,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)
 
 static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
 {
-	pmc->prev_counter = pmc->counter;
-	pmc->counter = (pmc->counter + 1) & pmc_bitmask(pmc);
+	pmc->emulated_counter += 1;
 	kvm_pmu_request_counter_reprogram(pmc);
 }
 
arch/x86/kvm/svm/pmu.c +1 -0
index a25b91ff9aea..b88fab4ae1d7 100644
@@ -243,6 +243,7 @@ static void amd_pmu_reset(struct kvm_vcpu *vcpu)
 
 		pmc_stop_counter(pmc);
 		pmc->counter = pmc->prev_counter = pmc->eventsel = 0;
+		pmc->emulated_counter = 0;
 	}
 
 	pmu->global_ctrl = pmu->global_status = 0;
arch/x86/kvm/vmx/pmu_intel.c +2 -0
index 626df5fdf542..d03c4ec7273d 100644
@@ -641,6 +641,7 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)
 
 		pmc_stop_counter(pmc);
 		pmc->counter = pmc->prev_counter = pmc->eventsel = 0;
+		pmc->emulated_counter = 0;
 	}
 
 	for (i = 0; i < KVM_PMC_MAX_FIXED; i++) {
@@ -648,6 +649,7 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)
 
 		pmc_stop_counter(pmc);
 		pmc->counter = pmc->prev_counter = 0;
+		pmc->emulated_counter = 0;
 	}
 
 	pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status = 0;

--
2.42.0.515.g380fc7ccd1-goog

#20 lore raw
Quoted 5 lines · Mingwei Zhang

On Fri, Sep 22, 2023, Mingwei Zhang wrote:
My initial testing on both QEMU and our GCP testing environment shows no "Uhhuh..." dmesg in guest.

Please take a look...

And now I'm extra confused, I thought the plan was for me to post a proper series for the emulated_counter idea[*], get the code base healthy/functional, and then build on top, e.g. improve performance and whatnot.

The below is just a stripped down version of that, and doesn't look quite right. Specifically, pmc_write_counter() needs to purge emulated_count (my pseudo-patch subtly handled that by pausing the counter).

I totally realize I'm moving sloooow, but I pinky swear I'm getting there. My compile-tested-only branch can be found at

https://github.com/sean-jc/linux x86/pmu_refactor

There's a lot more in there, e.g. it has fixes from Roman and Jim, along with some other found-by-inspection cleanups.

I dropped the "pause on WRMSR" proposal. I still don't love the offset approach, but I agree that pausing and reprogramming counters on writes could introduce an entirely new set of problems.

I'm logging off for the weekend, but I'll pick this back up next (it's at the top of my priority list, assuming guest_memfd doesn't somehow catch fire.

[*] https://lore.kernel.org/all/[email protected]

Quoted 8 lines
arch/x86/kvm/pmu.c +0 -0
index edb89b51b383..47acf3a2b077 100644
@@ -240,12 +240,13 @@ static void pmc_pause_counter(struct kvm_pmc *pmc)
 {
 	u64 counter = pmc->counter;
 

arch/x86/kvm/pmu.c static void pmc_pause_counter(struct kvm_pmc *pmc) · line 247

Quoted 10 lines
arch/x86/kvm/pmu.c +4 -5
-	if (!pmc->perf_event || pmc->is_paused)
-		return;
-
 	/* update counter, reset event value to avoid redundant accumulation */
-	counter += perf_event_pause(pmc->perf_event, true);
-	pmc->counter = counter & pmc_bitmask(pmc);
+	if (pmc->perf_event && !pmc->is_paused)
+		counter += perf_event_pause(pmc->perf_event, true);
+
+	pmc->prev_counter = counter & pmc_bitmask(pmc);

Honest question, is it actually correct/necessary to mask the counter at the intermediate steps? Conceptually, the hardware count and the emulated count are the same thing, just tracked separately.

arch/x86/kvm/pmu.c static void reprogram_counter(struct kvm_pmc *pmc) · line 455

Quoted 9 lines
arch/x86/kvm/pmu.c +2 -0
+	pmc->counter = (counter + pmc->emulated_counter) & pmc_bitmask(pmc);
+	pmc->emulated_counter = 0;
 	pmc->is_paused = true;
 }
 
@@ -452,6 +453,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)
 reprogram_complete:
 	clear_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->reprogram_pmi);
 	pmc->prev_counter = 0;

I don't see any reason to keep kvm_pmc.prev_counter. reprogram_counter() is the only caller of pmc_pause_counter(), and so is effectively the only writer and the only reader. I.e. prev_counter can just be a local variable in reprogram_counter(), no?

#21 lore raw

Hi Sean,

Quoted 16 lines · Sean Christopherson
Quoted 5 lines · Mingwei Zhang

On Fri, Sep 22, 2023, Mingwei Zhang wrote:
My initial testing on both QEMU and our GCP testing environment shows no "Uhhuh..." dmesg in guest.

Please take a look...

And now I'm extra confused, I thought the plan was for me to post a proper series for the emulated_counter idea[*], get the code base healthy/functional, and then build on top, e.g. improve performance and whatnot.

The below is just a stripped down version of that, and doesn't look quite right. Specifically, pmc_write_counter() needs to purge emulated_count (my pseudo-patch subtly handled that by pausing the counter).

Quoted 7 lines

I totally realize I'm moving sloooow, but I pinky swear I'm getting there. My compile-tested-only branch can be found at

https://github.com/sean-jc/linux x86/pmu_refactor

There's a lot more in there, e.g. it has fixes from Roman and Jim, along with some other found-by-inspection cleanups.

Sean, thanks for the work you have done on the thread:
https://lore.kernel.org/all/[email protected]

I think the diff you posted helped quite a lot. In fact, Jim also asked me to try using emulated_counter and I thought that just fixed the issue. I tried my own version as well as yours. However, neither could fix this problem at that time. So, Jim took a further look on the lower level and I was stuck on the performance analysis until recently I came back and discovered the real fix for this.

Your diff (or I should say your patch) covers lots of things including the adding of emulated_counter, some refactoring code on pmu reset and pause-on-wrmsr. In comparison, my code just focuses on the bug fixing for the duplicate PMI, since that's what I care for production. Although the code looks somewhat similar, the thought process and intention are quite different.

Sorry to confuse you. To resolve this issue, I am wondering if adding you into "Co-developed-by:" would be a valid choice? Or the other way around like adding me as a co-developer into one patch of your series.

In addition, I have no interest in further refactoring the existing vPMU code. So for that I will definitely step aside.

Update: it looks like both my patch and Jim's patches (applied separately) break the kvm-unit-test/pmu with the following error on SPR:

FAIL: Intel: emulated instruction: instruction counter overflow
FAIL: Intel: full-width writes: emulated instruction: instruction counter overflow

Not sure whether it is a test error or not.

Quoted 18 lines

I dropped the "pause on WRMSR" proposal. I still don't love the offset approach, but I agree that pausing and reprogramming counters on writes could introduce an entirely new set of problems.

I'm logging off for the weekend, but I'll pick this back up next (it's at the top of my priority list, assuming guest_memfd doesn't somehow catch fire.

[*] https://lore.kernel.org/all/[email protected]

Quoted 8 lines
arch/x86/kvm/pmu.c +0 -0
index edb89b51b383..47acf3a2b077 100644
@@ -240,12 +240,13 @@ static void pmc_pause_counter(struct kvm_pmc *pmc)
 {
      u64 counter = pmc->counter;
 
Quoted 14 lines

arch/x86/kvm/pmu.c static void pmc_pause_counter(struct kvm_pmc *pmc) · line 247

Quoted 10 lines
arch/x86/kvm/pmu.c +4 -5
-     if (!pmc->perf_event || pmc->is_paused)
-             return;
-
      /* update counter, reset event value to avoid redundant accumulation */
-     counter += perf_event_pause(pmc->perf_event, true);
-     pmc->counter = counter & pmc_bitmask(pmc);
+     if (pmc->perf_event && !pmc->is_paused)
+             counter += perf_event_pause(pmc->perf_event, true);
+
+     pmc->prev_counter = counter & pmc_bitmask(pmc);

Honest question, is it actually correct/necessary to mask the counter at the intermediate steps? Conceptually, the hardware count and the emulated count are the same thing, just tracked separately.

It is valid because the counter has stopped, and now pmc->counter contains the valid value. So, it can move into different variables and do the comparison. Regarding the intermediate steps, I don't think this is visible to the guest, no? Otherwise, we may have to use tmp variables and then make the assignment atomic, although I doubt if that is needed.

Quoted 6 lines
Quoted 6 lines
arch/x86/kvm/pmu.c +2 -0
+     pmc->counter = (counter + pmc->emulated_counter) & pmc_bitmask(pmc);
+     pmc->emulated_counter = 0;
      pmc->is_paused = true;
 }
 
Quoted 9 lines

arch/x86/kvm/pmu.c static void reprogram_counter(struct kvm_pmc *pmc) · line 455

Quoted 4 lines
arch/x86/kvm/pmu.c +0 -0
@@ -452,6 +453,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)
 reprogram_complete:
      clear_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->reprogram_pmi);
      pmc->prev_counter = 0;

I don't see any reason to keep kvm_pmc.prev_counter. reprogram_counter() is the only caller of pmc_pause_counter(), and so is effectively the only writer and the only reader. I.e. prev_counter can just be a local variable in reprogram_counter(), no?

You are right, I will update in the next version.

#22 lore raw
In reply to Mingwei Zhang
Quoted 209 lines · Mingwei Zhang
Quoted 77 lines · Mingwei Zhang
Quoted 56 lines · Sean Christopherson
Quoted 54 lines · Mingwei Zhang
Quoted 38 lines · Sean Christopherson
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 5 lines

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

Quoted 7 lines
Quoted 4 lines

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

Let me step back.

We have the following problem when we run perf inside guest:

[ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3.
[ 1437.487330] Dazed and confused, but trying to continue

This means there are more NMIs that guest PMI could understand. So there are potentially two approaches to solve the problem: 1) fix the PMI injection issue: only one can be injected; 2) fix the code that causes the (incorrect) multiple PMI injection.

Quoted 6 lines

I am working on the 2nd one. So, the property of the 2nd one is:
without patches in 1) (Jim's patches), we could still avoid the above warning messages.

Thanks.
-Mingwei

This is my draft version. If you don't have full-width counter support, this patch needs be placed on top of this one:
https://lore.kernel.org/all/[email protected]/

My initial testing on both QEMU and our GCP testing environment shows no "Uhhuh..." dmesg in guest.

Please take a look...

From 47e629269d8b0ff65c242334f068300216cb7f91 Mon Sep 17 00:00:00 2001
From: Mingwei Zhang <[email protected]>
Date: Fri, 22 Sep 2023 17:13:55 +0000
Subject: [PATCH] KVM: x86/pmu: Fix emulated counter increment due to
instruction emulation

Fix KVM emulated counter increment due to instruction emulation. KVM pmc->counter is always a snapshot value when counter is running. Therefore, the value does not represent the actual value of counter. Thus it is inappropriate to compare it with other counter values. In existing code KVM directly compares pmc->prev_counter and pmc->counter directly. However, pmc->prev_counter is a snaphot value assigned from pmc->counter when counter may still be running. So this comparison logic in reprogram_counter() will generate incorrect invocations to __kvm_perf_overflow(in_pmi=false) and generate duplicated PMI injection requests.

Fix this issue by adding emulated_counter field and only the do the counter calculation after we pause

Change-Id: I2d59e68557fd35f7bbcfe09ea42ad81bd36776b7

---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/pmu.c              | 15 ++++++++-------
 arch/x86/kvm/svm/pmu.c          |  1 +
 arch/x86/kvm/vmx/pmu_intel.c    |  2 ++
 4 files changed, 12 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 1a4def36d5bb..47bbfbc0aa35 100644
@@ -494,6 +494,7 @@ struct kvm_pmc {
        bool intr;
        u64 counter;
        u64 prev_counter;
+       u64 emulated_counter;
        u64 eventsel;
        struct perf_event *perf_event;
        struct kvm_vcpu *vcpu;
arch/x86/kvm/pmu.c +8 -7
index edb89b51b383..47acf3a2b077 100644
@@ -240,12 +240,13 @@ static void pmc_pause_counter(struct kvm_pmc *pmc)
 {
        u64 counter = pmc->counter;
 
-       if (!pmc->perf_event || pmc->is_paused)
-               return;
-
        /* update counter, reset event value to avoid redundant accumulation */
-       counter += perf_event_pause(pmc->perf_event, true);
-       pmc->counter = counter & pmc_bitmask(pmc);
+       if (pmc->perf_event && !pmc->is_paused)
+               counter += perf_event_pause(pmc->perf_event, true);
+
+       pmc->prev_counter = counter & pmc_bitmask(pmc);
+       pmc->counter = (counter + pmc->emulated_counter) & pmc_bitmask(pmc);
+       pmc->emulated_counter = 0;
        pmc->is_paused = true;
 }
 
@@ -452,6 +453,7 @@ static void reprogram_counter(struct kvm_pmc *pmc)
 reprogram_complete:
        clear_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->reprogram_pmi);
        pmc->prev_counter = 0;
+       pmc->emulated_counter = 0;
 }
 
 void kvm_pmu_handle_event(struct kvm_vcpu *vcpu)
@@ -725,8 +727,7 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu)
 
 static void kvm_pmu_incr_counter(struct kvm_pmc *pmc)
 {
-       pmc->prev_counter = pmc->counter;
-       pmc->counter = (pmc->counter + 1) & pmc_bitmask(pmc);
+       pmc->emulated_counter += 1;
        kvm_pmu_request_counter_reprogram(pmc);
 }
 
arch/x86/kvm/svm/pmu.c +1 -0
index a25b91ff9aea..b88fab4ae1d7 100644
@@ -243,6 +243,7 @@ static void amd_pmu_reset(struct kvm_vcpu *vcpu)
 
                pmc_stop_counter(pmc);
                pmc->counter = pmc->prev_counter = pmc->eventsel = 0;
+               pmc->emulated_counter = 0;
        }
 
        pmu->global_ctrl = pmu->global_status = 0;
arch/x86/kvm/vmx/pmu_intel.c +2 -1
index 626df5fdf542..d03c4ec7273d 100644
@@ -641,6 +641,7 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)
 
                pmc_stop_counter(pmc);
                pmc->counter = pmc->prev_counter = pmc->eventsel = 0;
+               pmc->emulated_counter = 0;
        }
 
        for (i = 0; i < KVM_PMC_MAX_FIXED; i++) {
@@ -648,6 +649,7 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu)
 
                pmc_stop_counter(pmc);
                pmc->counter = pmc->prev_counter = 0;
+               pmc->emulated_counter = 0;
        }
 
        pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status = 0;
--
Quoted 1 line

2.42.0.515.g380fc7ccd1-goog

#23 lore raw
In reply to Mingwei Zhang
Quoted 24 lines · Mingwei Zhang
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Mingwei Zhang

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

Quoted 7 lines
Quoted 4 lines

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

Let me step back.

We have the following problem when we run perf inside guest:

Quoted 7 lines
[ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3.
[ 1437.487330] Dazed and confused, but trying to continue

This means there are more NMIs that guest PMI could understand. So there are potentially two approaches to solve the problem: 1) fix the PMI injection issue: only one can be injected; 2) fix the code that causes the (incorrect) multiple PMI injection.

No, because the LVTPC masking fix isn't optional, the current KVM behavior is a clear violation of the SDM. And I'm struggling to think of a sane way to fix the IRQ work bug, e.g. KVM would have to busy on the work finishing before resuming the guest, which is rather crazy.

I'm not saying there isn't more work to be done, nor am I saying that we shouldn't further harden KVM against double-injection. I'm just truly confused as to what that has to do with Jim's fixes.

Quoted 3 lines

I am working on the 2nd one. So, the property of the 2nd one is:
without patches in 1) (Jim's patches), we could still avoid the above warning messages.

#24 lore raw
Quoted 34 lines · Sean Christopherson
Quoted 24 lines · Mingwei Zhang
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Mingwei Zhang

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

Quoted 7 lines
Quoted 4 lines

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

Let me step back.

We have the following problem when we run perf inside guest:

Quoted 7 lines
[ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3.
[ 1437.487330] Dazed and confused, but trying to continue

This means there are more NMIs that guest PMI could understand. So there are potentially two approaches to solve the problem: 1) fix the PMI injection issue: only one can be injected; 2) fix the code that causes the (incorrect) multiple PMI injection.

Quoted 9 lines

No, because the LVTPC masking fix isn't optional, the current KVM behavior is a clear violation of the SDM. And I'm struggling to think of a sane way to fix the IRQ work bug, e.g. KVM would have to busy on the work finishing before resuming the guest, which is rather crazy.

I'm not saying there isn't more work to be done, nor am I saying that we shouldn't further harden KVM against double-injection. I'm just truly confused as to what that has to do with Jim's fixes.

hmm, I will take the "two approaches" back. You are right on that. "two directions" is what I mean.

Oh, I think I did not elaborate the full context to you maybe. That might cause confusion and sorry about that.

The context of Jim's patches is to fix the multiple PMI injections when using perf, starting from
https://lore.kernel.org/all/[email protected]/

So, regarding the fix, there are multiple layers and they may or may not be logically connected closely, but we are solving the same problem. In fact, I was asking Jim to help me with this specific issue :)

So yes, they could be put together and they could be put separately. But I don't see why they _cannot_ be together or cause confusion. So, I would like to put them together in the same context with a cover letter fully describing the details.

FYI for reviewers: to reproduce the spurious PMI issue in the guest VM, you need to let KVM emulate some instructions during the runtime, so the function kvm_pmu_incr_counter() will be triggered more. One option is to add a kernel cmdline like "idle=nomwait" to your guest kernel. Regarding the workload in guest vm, please run the perf command specified in
https://lore.kernel.org/all/[email protected]/

Thanks.
-Mingwei

-Mingwei

#25 lore raw
Quoted 2 lines · Mingwei Zhang

So yes, they could be put together and they could be put separately. But I don't see why they _cannot_ be together or cause confusion.

Because they don't need to be put together. Roman's patch kinda sorta overlaps with the prev_counter mess, but Jim's fixes are entirely orthogonal.

If one person initially posted such a series with everything together I probably wouldn't care *too* much, but combining patches and/or series that aren't tightly coupled or dependent in some way usually does more harm than good. E.g. if a maintainer has complaints against only one or two patches in series of unrelated patches, then grabbing the "good" patches is unnecessarily difficult. It's not truly hard on the maintainer's end, but little bits of avoidable friction in the process adds up across hundreds and thousands of patches.

FWIW, my plan is to apply Roman's patch pretty much as-is, grab v2 from Jim, and post my cleanups as a separate series on top (maybe two series, really haven't thought about it yet). The only reason I have them all in a single branch is because there are code conflicts and I know I will apply the patches from Roman and Jim first, i.e. I didn't want to develop on a base that I knew would become stale.

Quoted 2 lines

So, I would like to put them together in the same context with a cover letter fully describing the details.

I certainly won't object to a thorough bug report/analysis, but I'd prefer that Jim's series be posted separately (though I don't care if it's you or Jim that posts it).

#26 lore raw

Hi Sean,

Quoted 23 lines · Sean Christopherson
Quoted 2 lines · Mingwei Zhang

So yes, they could be put together and they could be put separately. But I don't see why they _cannot_ be together or cause confusion.

Because they don't need to be put together. Roman's patch kinda sorta overlaps with the prev_counter mess, but Jim's fixes are entirely orthogonal.

If one person initially posted such a series with everything together I probably wouldn't care *too* much, but combining patches and/or series that aren't tightly coupled or dependent in some way usually does more harm than good. E.g. if a maintainer has complaints against only one or two patches in series of unrelated patches, then grabbing the "good" patches is unnecessarily difficult. It's not truly hard on the maintainer's end, but little bits of avoidable friction in the process adds up across hundreds and thousands of patches.

FWIW, my plan is to apply Roman's patch pretty much as-is, grab v2 from Jim, and post my cleanups as a separate series on top (maybe two series, really haven't thought about it yet). The only reason I have them all in a single branch is because there are code conflicts and I know I will apply the patches from Roman and Jim first, i.e. I didn't want to develop on a base that I knew would become stale.

Quoted 6 lines
Quoted 2 lines

So, I would like to put them together in the same context with a cover letter fully describing the details.

I certainly won't object to a thorough bug report/analysis, but I'd prefer that Jim's series be posted separately (though I don't care if it's you or Jim that posts it).

Thanks for agreeing to put things together. In fact, everything together means all relevant fix patches for the same bug need to be together. But I will put my patch explicitly as _optional_ mentioned in the cover letter.

If the series causes inconvenience, please accept my apology. For the sense of responsibility, I think I could just use this opportunity to send my updated version with your comment fixed. I will also use this chance to update your fix to Jim's patches.

One last thing, breaking the kvm-unit-test/pmu still surprises me. Please test it again when you have a chance. Maybe adding more fixes on top. With the series sent, I will hand it over to you.

Thanks.
-Mingwei

-Mingwei

#27 lore raw
Quoted 39 lines · Mingwei Zhang

Hi Sean,

Quoted 23 lines · Sean Christopherson
Quoted 2 lines · Mingwei Zhang

So yes, they could be put together and they could be put separately. But I don't see why they _cannot_ be together or cause confusion.

Because they don't need to be put together. Roman's patch kinda sorta overlaps with the prev_counter mess, but Jim's fixes are entirely orthogonal.

If one person initially posted such a series with everything together I probably wouldn't care *too* much, but combining patches and/or series that aren't tightly coupled or dependent in some way usually does more harm than good. E.g. if a maintainer has complaints against only one or two patches in series of unrelated patches, then grabbing the "good" patches is unnecessarily difficult. It's not truly hard on the maintainer's end, but little bits of avoidable friction in the process adds up across hundreds and thousands of patches.

FWIW, my plan is to apply Roman's patch pretty much as-is, grab v2 from Jim, and post my cleanups as a separate series on top (maybe two series, really haven't thought about it yet). The only reason I have them all in a single branch is because there are code conflicts and I know I will apply the patches from Roman and Jim first, i.e. I didn't want to develop on a base that I knew would become stale.

Quoted 6 lines
Quoted 2 lines

So, I would like to put them together in the same context with a cover letter fully describing the details.

I certainly won't object to a thorough bug report/analysis, but I'd prefer that Jim's series be posted separately (though I don't care if it's you or Jim that posts it).

Thanks for agreeing to put things together. In fact, everything together means all relevant fix patches for the same bug need to be together. But I will put my patch explicitly as _optional_ mentioned in the cover letter.

Quoted 9 lines

If the series causes inconvenience, please accept my apology. For the sense of responsibility, I think I could just use this opportunity to send my updated version with your comment fixed. I will also use this chance to update your fix to Jim's patches.

One last thing, breaking the kvm-unit-test/pmu still surprises me. Please test it again when you have a chance. Maybe adding more fixes on top. With the series sent, I will hand it over to you.

Never, this is a test failure that we already solved internally. Applying the following fix to kvm-unit-tests/pmu remove the failures:

x86/pmu.c +1 -0
index 0def2869..667e6233 100644
@@ -68,6 +68,7 @@ volatile uint64_t irq_received;
 static void cnt_overflow(isr_regs_t *regs)
 {
        irq_received++;
+       apic_write(APIC_LVTPC, apic_read(APIC_LVTPC) & ~APIC_LVT_MASKED);
        apic_write(APIC_EOI, 0);
 }
 

Since KVM vPMU adds a mask when injecting the PMI, it is the responsibility of the guest PMI handler to remove the mask and allow subsequent PMIs delivered.

We should upstream the above fix some time.

Thanks.
-Mingwei

#28 lore raw
Quoted 31 lines · Mingwei Zhang
Quoted 30 lines · Mingwei Zhang

Hi Sean,

Quoted 26 lines · Sean Christopherson
Quoted 2 lines · Mingwei Zhang

So yes, they could be put together and they could be put separately. But I don't see why they _cannot_ be together or cause confusion.

Because they don't need to be put together. Roman's patch kinda sorta overlaps with the prev_counter mess, but Jim's fixes are entirely orthogonal.

If one person initially posted such a series with everything together I probably wouldn't care *too* much, but combining patches and/or series that aren't tightly coupled or dependent in some way usually does more harm than good. E.g. if a maintainer has complaints against only one or two patches in series of unrelated patches, then grabbing the "good" patches is unnecessarily difficult. It's not truly hard on the maintainer's end, but little bits of avoidable friction in the process adds up across hundreds and thousands of patches.

FWIW, my plan is to apply Roman's patch pretty much as-is, grab v2 from Jim, and post my cleanups as a separate series on top (maybe two series, really haven't thought about it yet). The only reason I have them all in a single branch is because there are code conflicts and I know I will apply the patches from Roman and Jim first, i.e. I didn't want to develop on a base that I knew would become stale.

Quoted 2 lines

So, I would like to put them together in the same context with a cover letter fully describing the details.

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

I certainly won't object to a thorough bug report/analysis, but I'd prefer that Jim's series be posted separately (though I don't care if it's you or Jim that posts it).

Thanks for agreeing to put things together. In fact, everything together means all relevant fix patches for the same bug need to be together. But I will put my patch explicitly as _optional_ mentioned in the cover letter.

No, please do not post your version of the emulated_counter patch. I am more than happy to give you primary author credit (though I need your SoB), all I care about is minimizing the amount of effort and overhead on my end. At this point, posting your version at this time will only generate more noise and make my job harder. To tie everything together in the cover letter, just include lore links to the relevant pseudo-patches.

Assuming you are taking over Jim's series, please post v2 asap. I want to get the critical fixes applied sooner than later.

Quoted 29 lines
Quoted 9 lines

If the series causes inconvenience, please accept my apology. For the sense of responsibility, I think I could just use this opportunity to send my updated version with your comment fixed. I will also use this chance to update your fix to Jim's patches.

One last thing, breaking the kvm-unit-test/pmu still surprises me. Please test it again when you have a chance. Maybe adding more fixes on top. With the series sent, I will hand it over to you.

Never, this is a test failure that we already solved internally. Applying the following fix to kvm-unit-tests/pmu remove the failures:

x86/pmu.c +1 -0
index 0def2869..667e6233 100644
@@ -68,6 +68,7 @@ volatile uint64_t irq_received;
 static void cnt_overflow(isr_regs_t *regs)
 {
        irq_received++;
+       apic_write(APIC_LVTPC, apic_read(APIC_LVTPC) & ~APIC_LVT_MASKED);
        apic_write(APIC_EOI, 0);
 }
 

Since KVM vPMU adds a mask when injecting the PMI, it is the responsibility of the guest PMI handler to remove the mask and allow subsequent PMIs delivered.

Quoted 1 line

We should upstream the above fix some time.

Please post the above asap. And give pmu_pebs.c's cnt_overflow() the same treatment when you do. Or just give my your SoB and I'll write the changelog.

#29 lore raw
In reply to Mingwei Zhang
Quoted 64 lines · Mingwei Zhang
Quoted 34 lines · Sean Christopherson
Quoted 24 lines · Mingwei Zhang
Quoted 11 lines · Sean Christopherson
Quoted 9 lines · Mingwei Zhang

Agree. Both patches are fixing the general potential buggy situation of multiple PMI injection on one vm entry: one software level defense (forcing the usage of KVM_REQ_PMI) and one hardware level defense (preventing PMI injection using mask).

Although neither patch in this series is fixing the root cause of this specific double PMI injection bug, I don't see a reason why we cannot add a "fixes" tag to them, since we may fix it and create it again.

Quoted 7 lines
Quoted 4 lines

I am currently working on it and testing my patch. Please give me some time, I think I could try sending out one version today. Once that is done, I will combine mine with the existing patch and send it out as a series.

Me confused, what patch? And what does this patch have to do with Jim's series? Unless I've missed something, Jim's patches are good to go with my nits addressed.

Let me step back.

We have the following problem when we run perf inside guest:

Quoted 7 lines
[ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3.
[ 1437.487330] Dazed and confused, but trying to continue

This means there are more NMIs that guest PMI could understand. So there are potentially two approaches to solve the problem: 1) fix the PMI injection issue: only one can be injected; 2) fix the code that causes the (incorrect) multiple PMI injection.

Quoted 9 lines

No, because the LVTPC masking fix isn't optional, the current KVM behavior is a clear violation of the SDM. And I'm struggling to think of a sane way to fix the IRQ work bug, e.g. KVM would have to busy on the work finishing before resuming the guest, which is rather crazy.

I'm not saying there isn't more work to be done, nor am I saying that we shouldn't further harden KVM against double-injection. I'm just truly confused as to what that has to do with Jim's fixes.

hmm, I will take the "two approaches" back. You are right on that. "two directions" is what I mean.

Oh, I think I did not elaborate the full context to you maybe. That might cause confusion and sorry about that.

The context of Jim's patches is to fix the multiple PMI injections when using perf, starting from
https://lore.kernel.org/all/[email protected]/

So, regarding the fix, there are multiple layers and they may or may not be logically connected closely, but we are solving the same problem. In fact, I was asking Jim to help me with this specific issue :)

So yes, they could be put together and they could be put separately. But I don't see why they _cannot_ be together or cause confusion. So, I would like to put them together in the same context with a cover letter fully describing the details.

Quoted 7 lines

FYI for reviewers: to reproduce the spurious PMI issue in the guest VM, you need to let KVM emulate some instructions during the runtime, so the function kvm_pmu_incr_counter() will be triggered more. One option is to add a kernel cmdline like "idle=nomwait" to your guest kernel. Regarding the workload in guest vm, please run the perf command specified in
https://lore.kernel.org/all/[email protected]/

Hi Mingwei,

I would encourage you to convert this perf use case into a sequence of MSRs accesses so that it's easier to understand where the emulation fails on KVM.

Quoted 8 lines

Thanks.

(quoted diff) +0 -2
-Mingwei
 
 
 
-Mingwei
#30 lore raw
In reply to Sean Christopherson
Quoted 37 lines · Sean Christopherson
Quoted 30 lines · Jim Mattson
Quoted 21 lines · Sean Christopherson
Quoted 15 lines · Jim Mattson

When the irq_work callback, kvm_pmi_trigger_fn(), is invoked during a VM-exit that also invokes __kvm_perf_overflow() as a result of instruction emulation, kvm_pmu_deliver_pmi() will be called twice before the next VM-entry.

That shouldn't be a problem. The local APIC is supposed to automatically set the mask flag in LVTPC when it handles a PMI, so the second PMI should be inhibited. However, KVM's local APIC emulation fails to set the mask flag in LVTPC when it handles a PMI, so two PMIs are delivered via the local APIC. In the common case, where LVTPC is configured to deliver an NMI, the first NMI is vectored through the guest IDT, and the second one is held pending. When the NMI handler returns, the second NMI is vectored through the IDT. For Linux guests, this results in the "dazed and confused" spurious NMI message.

Quoted 3 lines

Though the obvious fix is to set the mask flag in LVTPC when handling a PMI, KVM's logic around synthesizing a PMI is unnecessarily convoluted.

Quoted 7 lines

To address Like's question about whether not this is necessary, I think we should rephrase this to explicitly state this is a bug irrespective of the whole LVTPC masking thing.

And I think it makes sense to swap the order of the two patches. The LVTPC masking fix is a clearcut architectural violation. This is a bit more of a grey area, though still blatantly buggy.

Quoted 5 lines

The reason I ordered the patches as I did is that when this patch comes first, it actually fixes the problem that was introduced in commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions"). If this patch comes second, it's less clear that it fixes a bug, since the other patch renders this one essentially moot.

Quoted 6 lines

Yeah, but as Like pointed out, the way the changelog is worded just raises the question of why this change is necessary.

I think we should tag them both for stable. They're both bug fixes, regardless of the ordering.

In the semantics of "at most one PMI per VM exit", what if the PMI-caused vm-exit itself triggers a guest counter overflow and triggers vPMI (for example, at this time the L1 guest is counting the number of vm-exit events from the L2 guest), will the latter interrupt be swallowed by L0 KVM ? What is the correct expectation ? It may be different on Intel and AMD.