Skip to contentSkip to conversation

KVM: x86: Optimize "stale" EOI bitmap exiting

PATCH Applied Sean Christopherson 2 participants 10 messages Mar 4, 2025 – Apr 25, 2025

Subsystems (3)
lore
Share & tools
lore

Sean Christopherson

author maintainer
#0 lore raw

v5 cover KVM: x86: Optimize "stale" EOI bitmap exiting

Slightly modified version of the patch (now mini-series) to optimize EOI bitmap exiting for "stale" routes, i.e. for the case where a vCPU has an in-flight IRQ when the routing changes, and needs one final EOI exit to deassert the IRQ.

Kai, I dropped your Reviewed-by as the change was non-trivial.

v5:
- Intercept EOI if and only if the IRQ is level-triggered.
- Add helper to consolidate logic.
- Tweak field name.
- "Reset" field to -1, immediately before scanning routes.

v4: https://lore.kernel.org/all/[email protected]

Sean Christopherson (2):
KVM: x86: Isolate edge vs. level check in userspace I/O APIC route
scanning
KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic

weizijie (1):
KVM: x86: Rescan I/O APIC routes after EOI interception for old
routing

 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/ioapic.c           |  7 ++-----
 arch/x86/kvm/ioapic.h           |  2 ++
 arch/x86/kvm/irq_comm.c         | 37 ++++++++++++++++++++++++++++-----
 arch/x86/kvm/lapic.c            |  8 +++++++
 arch/x86/kvm/x86.c              |  1 +
 6 files changed, 46 insertions(+), 10 deletions(-)

base-commit: ead22377189e55ba0e9b637a8f48578cf84f5b9c

Signature · 1 line
2.48.1.711.g2feabab25a-goog
author maintainer
#1 lore raw

PATCH v5 1/3 KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning

Extract and isolate the trigger mode check in kvm_scan_ioapic_routes() in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).

No functional change intended.

---
 arch/x86/kvm/irq_comm.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
arch/x86/kvm/irq_comm.c +6 -4
index 8136695f7b96..866f84392797 100644
@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 
 			kvm_set_msi_irq(vcpu->kvm, entry, &irq);
 
-			if (irq.trig_mode &&
-			    (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
-						 irq.dest_id, irq.dest_mode) ||
-			     kvm_apic_pending_eoi(vcpu, irq.vector)))
+			if (!irq.trig_mode)
+				continue;
+
+			if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
+						irq.dest_id, irq.dest_mode) ||
+			     kvm_apic_pending_eoi(vcpu, irq.vector))
 				__set_bit(irq.vector, ioapic_handled_vectors);
 		}
 	}
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Reviewed-by on v5 1/3
#2 lore raw

Re: [PATCH v5 1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning

Quoted 7 lines · Sean Christopherson

Extract and isolate the trigger mode check in kvm_scan_ioapic_routes() in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).

No functional change intended.

Quoted 9 lines
---
 arch/x86/kvm/irq_comm.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
arch/x86/kvm/irq_comm.c +0 -0
index 8136695f7b96..866f84392797 100644

arch/x86/kvm/irq_comm.c void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, · line 428

Quoted 9 lines
arch/x86/kvm/irq_comm.c +2 -4
@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 
 			kvm_set_msi_irq(vcpu->kvm, entry, &irq);
 
-			if (irq.trig_mode &&
-			    (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
-						 irq.dest_id, irq.dest_mode) ||
-			     kvm_apic_pending_eoi(vcpu, irq.vector)))
+			if (!irq.trig_mode)
+				continue;

Perhaps take this chance to make it explicit?

			if (irq.trig_mode != IOAPIC_LEVEL_TRIG)
				continue;

kvm_ioapic_scan_entry() also checks against IOAPIC_LEVEL_TRIG explicitly.

Quoted 7 lines
arch/x86/kvm/irq_comm.c +4 -0
+
+			if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
+						irq.dest_id, irq.dest_mode) ||
+			     kvm_apic_pending_eoi(vcpu, irq.vector))
 				__set_bit(irq.vector, ioapic_handled_vectors);
 		}
 	}
author maintainer
#3 lore raw

Re: [PATCH v5 1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning

Quoted 32 lines · Kai Huang
Quoted 7 lines · Sean Christopherson

Extract and isolate the trigger mode check in kvm_scan_ioapic_routes() in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).

No functional change intended.

Quoted 9 lines
---
 arch/x86/kvm/irq_comm.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
arch/x86/kvm/irq_comm.c +0 -0
index 8136695f7b96..866f84392797 100644

arch/x86/kvm/irq_comm.c void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, · line 428

Quoted 9 lines
arch/x86/kvm/irq_comm.c +2 -4
@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 
 			kvm_set_msi_irq(vcpu->kvm, entry, &irq);
 
-			if (irq.trig_mode &&
-			    (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
-						 irq.dest_id, irq.dest_mode) ||
-			     kvm_apic_pending_eoi(vcpu, irq.vector)))
+			if (!irq.trig_mode)
+				continue;

Perhaps take this chance to make it explicit?

Quoted 4 lines
			if (irq.trig_mode != IOAPIC_LEVEL_TRIG)
				continue;

kvm_ioapic_scan_entry() also checks against IOAPIC_LEVEL_TRIG explicitly.

Hmm, I'm leaning "no". kvm_set_msi_irq() isn't I/O APIC specific (and obviously neither is "struct kvm_lapic_irq"). The fact that it sets irq.trig_mode to '0' or '1', and that the '1' value in particular happens to match IOAPIC_LEVEL_TRIG is somewhat of a coincidence.

kvm_ioapic_scan_entry() on the other operates on a "union kvm_ioapic_redirect_entry" object, in which case trig_mode is guaranteed to be '0' or '1', i.e. is exactly IOAPIC_EDGE_TRIG or IOAPIC_LEVEL_TRIG.

	u8 trig_mode:1;

So as much as I advocate for consistency, I think in this case it makes sense to be consistent with __apic_accept_irq(), which only cares about zero vs. non-zero.

#4 lore raw

Re: [PATCH v5 1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning

Quoted 43 lines · Sean Christopherson
Quoted 32 lines · Kai Huang
Quoted 7 lines · Sean Christopherson

Extract and isolate the trigger mode check in kvm_scan_ioapic_routes() in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).

No functional change intended.

Quoted 9 lines
---
  arch/x86/kvm/irq_comm.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
arch/x86/kvm/irq_comm.c +0 -0
index 8136695f7b96..866f84392797 100644

arch/x86/kvm/irq_comm.c void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, · line 428

Quoted 9 lines
arch/x86/kvm/irq_comm.c +2 -4
@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 
  			kvm_set_msi_irq(vcpu->kvm, entry, &irq);
 
-			if (irq.trig_mode &&
-			    (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
-						 irq.dest_id, irq.dest_mode) ||
-			     kvm_apic_pending_eoi(vcpu, irq.vector)))
+			if (!irq.trig_mode)
+				continue;

Perhaps take this chance to make it explicit?

Quoted 4 lines
			if (irq.trig_mode != IOAPIC_LEVEL_TRIG)
				continue;

kvm_ioapic_scan_entry() also checks against IOAPIC_LEVEL_TRIG explicitly.

Hmm, I'm leaning "no". kvm_set_msi_irq() isn't I/O APIC specific (and obviously neither is "struct kvm_lapic_irq"). The fact that it sets irq.trig_mode to '0' or '1', and that the '1' value in particular happens to match IOAPIC_LEVEL_TRIG is somewhat of a coincidence.

Quoted 5 lines

kvm_ioapic_scan_entry() on the other operates on a "union kvm_ioapic_redirect_entry" object, in which case trig_mode is guaranteed to be '0' or '1', i.e. is exactly IOAPIC_EDGE_TRIG or IOAPIC_LEVEL_TRIG.

	u8 trig_mode:1;

This makes sense. Thanks for clarifying.

Quoted 3 lines

So as much as I advocate for consistency, I think in this case it makes sense to be consistent with __apic_accept_irq(), which only cares about zero vs. non-zero.

Yeah LGTM.

author maintainer
#5 lore raw

PATCH v5 2/3 KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic

In reply to Sean Christopherson

Extract the vCPU specific EOI interception logic for I/O APIC emulation into a common helper for userspace and in-kernel emulation in anticipation of optimizing the "pending EOI" case.

No functional change intended.

---
 arch/x86/kvm/ioapic.c   |  7 ++-----
 arch/x86/kvm/ioapic.h   |  2 ++
 arch/x86/kvm/irq_comm.c | 21 +++++++++++++++++----
 3 files changed, 21 insertions(+), 9 deletions(-)
arch/x86/kvm/ioapic.c +2 -5
index 995eb5054360..45dae2d5d2f1 100644
@@ -296,11 +296,8 @@ void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, ulong *ioapic_handled_vectors)
 		    index == RTC_GSI) {
 			u16 dm = kvm_lapic_irq_dest_mode(!!e->fields.dest_mode);
 
-			if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
-						e->fields.dest_id, dm) ||
-			    kvm_apic_pending_eoi(vcpu, e->fields.vector))
-				__set_bit(e->fields.vector,
-					  ioapic_handled_vectors);
+			kvm_scan_ioapic_irq(vcpu, e->fields.dest_id, dm,
+					    e->fields.vector, ioapic_handled_vectors);
 		}
 	}
 	spin_unlock(&ioapic->lock);
arch/x86/kvm/ioapic.h +2 -0
index 539333ac4b38..aa8cb4ac0479 100644
@@ -120,4 +120,6 @@ void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu,
 			   ulong *ioapic_handled_vectors);
 void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 			    ulong *ioapic_handled_vectors);
+void kvm_scan_ioapic_irq(struct kvm_vcpu *vcpu, u32 dest_id, u16 dest_mode,
+			 u8 vector, unsigned long *ioapic_handled_vectors);
 #endif
arch/x86/kvm/irq_comm.c +17 -4
index 866f84392797..14590d9c4a37 100644
@@ -402,6 +402,21 @@ void kvm_arch_post_irq_routing_update(struct kvm *kvm)
 	kvm_make_scan_ioapic_request(kvm);
 }
 
+void kvm_scan_ioapic_irq(struct kvm_vcpu *vcpu, u32 dest_id, u16 dest_mode,
+			 u8 vector, unsigned long *ioapic_handled_vectors)
+{
+	/*
+	 * Intercept EOI if the vCPU is the target of the new IRQ routing, or
+	 * the vCPU has a pending IRQ from the old routing, i.e. if the vCPU
+	 * may receive a level-triggered IRQ in the future, or already received
+	 * level-triggered IRQ.  The EOI needs to be intercepted and forwarded
+	 * to I/O APIC emulation so that the IRQ can be de-asserted.
+	 */
+	if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT, dest_id, dest_mode) ||
+	    kvm_apic_pending_eoi(vcpu, vector))
+		__set_bit(vector, ioapic_handled_vectors);
+}
+
 void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 			    ulong *ioapic_handled_vectors)
 {
@@ -427,10 +442,8 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
 			if (!irq.trig_mode)
 				continue;
 
-			if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT,
-						irq.dest_id, irq.dest_mode) ||
-			     kvm_apic_pending_eoi(vcpu, irq.vector))
-				__set_bit(irq.vector, ioapic_handled_vectors);
+			kvm_scan_ioapic_irq(vcpu, irq.dest_id, irq.dest_mode,
+					    irq.vector, ioapic_handled_vectors);
 		}
 	}
 	srcu_read_unlock(&kvm->irq_srcu, idx);
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Reviewed-by on v5 2/3
#6 lore raw

Re: [PATCH v5 2/3] KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic

Quoted 8 lines · Sean Christopherson

Extract the vCPU specific EOI interception logic for I/O APIC emulation into a common helper for userspace and in-kernel emulation in anticipation of optimizing the "pending EOI" case.

No functional change intended.

author maintainer
#7 lore raw

PATCH v5 3/3 KVM: x86: Rescan I/O APIC routes after EOI interception for old routing

In reply to Sean Christopherson

From: weizijie <[email protected]>

Rescan I/O APIC routes for a vCPU after handling an intercepted I/O APIC EOI for an IRQ that is not targeting said vCPU, i.e. after handling what's effectively a stale EOI VM-Exit. If a level-triggered IRQ is in-flight when IRQ routing changes, e.g. because the guest change routing from its IRQ handler, then KVM intercepts EOIs on both the new and old target vCPUs, so that the in-flight IRQ can be de-asserted when it's EOI'd.

However, only the EOI for the in-flight IRQ needs to intercepted, as IRQs on the same vector with the new routing are coincidental, i.e. occur only if the guest is reusing the vector for multiple interrupt sources. If the I/O APIC routes aren't rescanned, KVM will unnecessarily intercept EOIs for the vector and negative impact the vCPU's interrupt performance.

Note, both commit db2bdcbbbd32 ("KVM: x86: fix edge EOI and IOAPIC reconfig race") and commit 0fc5a36dd6b3 ("KVM: x86: ioapic: Fix level-triggered EOI and IOAPIC reconfigure race") mentioned this issue, but it was considered a "rare" occurrence thus was not addressed. However in real environments, this issue can happen even in a well-behaved guest.

Cc Kai Huang <[email protected]>
[sean: massage changelog and comments, use int/-1, reset at scan]
---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/irq_comm.c         | 16 ++++++++++++++--
 arch/x86/kvm/lapic.c            |  8 ++++++++
 arch/x86/kvm/x86.c              |  1 +
 4 files changed, 24 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 44007a351e88..b378414c3104 100644
@@ -1025,6 +1025,7 @@ struct kvm_vcpu_arch {
 
 	int pending_ioapic_eoi;
 	int pending_external_vector;
+	int highest_stale_pending_ioapic_eoi;
 
 	/* be preempted when it's in kernel-mode(cpl=0) */
 	bool preempted_in_kernel;
arch/x86/kvm/irq_comm.c +14 -2
index 14590d9c4a37..d6d792b5d1bd 100644
@@ -412,9 +412,21 @@ void kvm_scan_ioapic_irq(struct kvm_vcpu *vcpu, u32 dest_id, u16 dest_mode,
 	 * level-triggered IRQ.  The EOI needs to be intercepted and forwarded
 	 * to I/O APIC emulation so that the IRQ can be de-asserted.
 	 */
-	if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT, dest_id, dest_mode) ||
-	    kvm_apic_pending_eoi(vcpu, vector))
+	if (kvm_apic_match_dest(vcpu, NULL, APIC_DEST_NOSHORT, dest_id, dest_mode)) {
 		__set_bit(vector, ioapic_handled_vectors);
+	} else if (kvm_apic_pending_eoi(vcpu, vector)) {
+		__set_bit(vector, ioapic_handled_vectors);
+
+		/*
+		 * Track the highest pending EOI for which the vCPU is NOT the
+		 * target in the new routing.  Only the EOI for the IRQ that is
+		 * in-flight (for the old routing) needs to be intercepted, any
+		 * future IRQs that arrive on this vCPU will be coincidental to
+		 * the level-triggered routing and don't need to be intercepted.
+		 */
+		if ((int)vector > vcpu->arch.highest_stale_pending_ioapic_eoi)
+			vcpu->arch.highest_stale_pending_ioapic_eoi = vector;
+	}
 }
 
 void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
arch/x86/kvm/lapic.c +8 -0
index 9dbc0f5d9865..6af84a0f84f3 100644
@@ -1459,6 +1459,14 @@ static void kvm_ioapic_send_eoi(struct kvm_lapic *apic, int vector)
 	if (!kvm_ioapic_handles_vector(apic, vector))
 		return;
 
+	/*
+	 * If the intercepted EOI is for an IRQ that was pending from previous
+	 * routing, then re-scan the I/O APIC routes as EOIs for the IRQ likely
+	 * no longer need to be intercepted.
+	 */
+	if (apic->vcpu->arch.highest_stale_pending_ioapic_eoi == vector)
+		kvm_make_request(KVM_REQ_SCAN_IOAPIC, apic->vcpu);
+
 	/* Request a KVM exit to inform the userspace IOAPIC. */
 	if (irqchip_split(apic->vcpu->kvm)) {
 		apic->vcpu->arch.pending_ioapic_eoi = vector;
arch/x86/kvm/x86.c +1 -0
index 7d4b9e2f1a38..a40b09dfb36a 100644
@@ -10650,6 +10650,7 @@ static void vcpu_scan_ioapic(struct kvm_vcpu *vcpu)
 		return;
 
 	bitmap_zero(vcpu->arch.ioapic_handled_vectors, 256);
+	vcpu->arch.highest_stale_pending_ioapic_eoi = -1;
 
 	kvm_x86_call(sync_pir_to_irr)(vcpu);
 
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Reviewed-by on v5 3/3
#8 lore raw

Re: [PATCH v5 3/3] KVM: x86: Rescan I/O APIC routes after EOI interception for old routing

Quoted 6 lines · Sean Christopherson

From: weizijie <[email protected]>

Rescan I/O APIC routes for a vCPU after handling an intercepted I/O APIC EOI for an IRQ that is not targeting said vCPU, i.e. after handling what's effectively a stale EOI VM-Exit. If a level-triggered IRQ is in-flight when IRQ routing changes, e.g. because the guest change routing from its

						   ^
						   changes ?
Quoted 4 lines

IRQ handler, then KVM intercepts EOIs on both the new and old target vCPUs, so that the in-flight IRQ can be de-asserted when it's EOI'd.

However, only the EOI for the in-flight IRQ needs to intercepted, as IRQs

							^
							be intercepted
Quoted 11 lines

on the same vector with the new routing are coincidental, i.e. occur only if the guest is reusing the vector for multiple interrupt sources. If the I/O APIC routes aren't rescanned, KVM will unnecessarily intercept EOIs for the vector and negative impact the vCPU's interrupt performance.

Note, both commit db2bdcbbbd32 ("KVM: x86: fix edge EOI and IOAPIC reconfig race") and commit 0fc5a36dd6b3 ("KVM: x86: ioapic: Fix level-triggered EOI and IOAPIC reconfigure race") mentioned this issue, but it was considered a "rare" occurrence thus was not addressed. However in real environments, this issue can happen even in a well-behaved guest.

Quoted 6 lines
Cc Kai Huang <[email protected]>
[sean: massage changelog and comments, use int/-1, reset at scan]
author maintainer Applied
#9 lore raw
In reply to Sean Christopherson
Quoted 8 lines · Sean Christopherson

Slightly modified version of the patch (now mini-series) to optimize EOI bitmap exiting for "stale" routes, i.e. for the case where a vCPU has an in-flight IRQ when the routing changes, and needs one final EOI exit to deassert the IRQ.

Kai, I dropped your Reviewed-by as the change was non-trivial.

[...]

Applied to kvm-x86 misc, thanks!

[1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning

      commit: b1f7723a5a5b018f4bc3fb8e234510be7c44ad00
[2/3] KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic
      commit: c2207bbc0c0f4b6ae8dbb73ec26e17aa0c45a3ca
[3/3] KVM: x86: Rescan I/O APIC routes after EOI interception for old routing
      commit: 87e4951e250bbccac47a8822f6f023a1de8b96ec

--
https://github.com/kvm-x86/linux/tree/next