KVM: x86: Optimize "stale" EOI bitmap exiting
Sean Christopherson
maintainerv5 cover KVM: x86: Optimize "stale" EOI bitmap exiting
Slightly modified version of the patch (now mini-series) to optimize EOI bitmap exiting for "stale" routes, i.e. for the case where a vCPU has an in-flight IRQ when the routing changes, and needs one final EOI exit to deassert the IRQ.
Kai, I dropped your Reviewed-by as the change was non-trivial.
v5:
- Intercept EOI if and only if the IRQ is level-triggered.
- Add helper to consolidate logic.
- Tweak field name.
- "Reset" field to -1, immediately before scanning routes.
v4: https://lore.kernel.org/all/[email protected]
Sean Christopherson (2):
KVM: x86: Isolate edge vs. level check in userspace I/O APIC route
scanning
KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic
weizijie (1):
KVM: x86: Rescan I/O APIC routes after EOI interception for old
routing
arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/ioapic.c | 7 ++----- arch/x86/kvm/ioapic.h | 2 ++ arch/x86/kvm/irq_comm.c | 37 ++++++++++++++++++++++++++++----- arch/x86/kvm/lapic.c | 8 +++++++ arch/x86/kvm/x86.c | 1 + 6 files changed, 46 insertions(+), 10 deletions(-)
base-commit: ead22377189e55ba0e9b637a8f48578cf84f5b9c
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Sean Christopherson
maintainerPATCH v5 1/3 KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning
Extract and isolate the trigger mode check in kvm_scan_ioapic_routes() in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).
No functional change intended.
--- arch/x86/kvm/irq_comm.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
arch/x86/kvm/irq_comm.c +6 -4
@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,kvm_set_msi_irq (vcpu->kvm, entry, &irq); -if (irq.trig_mode && - (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , - irq.dest_id, irq.dest_mode) || -kvm_apic_pending_eoi (vcpu, irq.vector))) +if (!irq.trig_mode) +continue ; + +if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , + irq.dest_id, irq.dest_mode) || +kvm_apic_pending_eoi (vcpu, irq.vector))__set_bit (irq.vector, ioapic_handled_vectors); } }
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Re: [PATCH v5 1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning
Quoted 7 lines · Sean Christopherson
Extract and isolate the trigger mode check in
kvm_scan_ioapic_routes()in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>
Quoted 9 lines
--- arch/x86/kvm/irq_comm.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)arch/x86/kvm/irq_comm.c +0
-0
arch/x86/kvm/irq_comm.c void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, · line 428
Quoted 9 lines
arch/x86/kvm/irq_comm.c +2
-4@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,kvm_set_msi_irq (vcpu->kvm, entry, &irq); -if (irq.trig_mode && - (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , - irq.dest_id, irq.dest_mode) || -kvm_apic_pending_eoi (vcpu, irq.vector))) +if (!irq.trig_mode) +continue ;
Perhaps take this chance to make it explicit?
if (irq.trig_mode != IOAPIC_LEVEL_TRIG) continue;
kvm_ioapic_scan_entry() also checks against IOAPIC_LEVEL_TRIG explicitly.
Quoted 7 lines
arch/x86/kvm/irq_comm.c +4
-0+ +if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , + irq.dest_id, irq.dest_mode) || +kvm_apic_pending_eoi (vcpu, irq.vector))__set_bit (irq.vector, ioapic_handled_vectors); } }
Sean Christopherson
maintainerRe: [PATCH v5 1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning
Quoted 32 lines · Kai Huang
Quoted 7 lines · Sean Christopherson
Extract and isolate the trigger mode check in
kvm_scan_ioapic_routes()in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Reviewed-by Kai Huang <[email protected]>Quoted 9 lines
--- arch/x86/kvm/irq_comm.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)arch/x86/kvm/irq_comm.c +0
-0arch/x86/kvm/irq_comm.c void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, · line 428
Quoted 9 lines
arch/x86/kvm/irq_comm.c +2
-4@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,kvm_set_msi_irq (vcpu->kvm, entry, &irq); -if (irq.trig_mode && - (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , - irq.dest_id, irq.dest_mode) || -kvm_apic_pending_eoi (vcpu, irq.vector))) +if (!irq.trig_mode) +continue ;Perhaps take this chance to make it explicit?
Quoted 4 lines
if (irq.trig_mode != IOAPIC_LEVEL_TRIG) continue;
kvm_ioapic_scan_entry()also checks againstIOAPIC_LEVEL_TRIGexplicitly.
Hmm, I'm leaning "no". kvm_set_msi_irq() isn't I/O APIC specific (and obviously neither is "struct kvm_lapic_irq"). The fact that it sets irq.trig_mode to '0' or '1', and that the '1' value in particular happens to match IOAPIC_LEVEL_TRIG is somewhat of a coincidence.
kvm_ioapic_scan_entry() on the other operates on a "union kvm_ioapic_redirect_entry" object, in which case trig_mode is guaranteed to be '0' or '1', i.e. is exactly IOAPIC_EDGE_TRIG or IOAPIC_LEVEL_TRIG.
u8 trig_mode:1;
So as much as I advocate for consistency, I think in this case it makes sense to be consistent with __apic_accept_irq(), which only cares about zero vs. non-zero.
Re: [PATCH v5 1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning
Quoted 43 lines · Sean Christopherson
Quoted 32 lines · Kai Huang
Quoted 7 lines · Sean Christopherson
Extract and isolate the trigger mode check in
kvm_scan_ioapic_routes()in anticipation of moving destination matching logic to a common helper (for userspace vs. in-kernel I/O APIC emulation).No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>Reviewed-by Kai Huang <[email protected]>Quoted 9 lines
--- arch/x86/kvm/irq_comm.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)arch/x86/kvm/irq_comm.c +0
-0arch/x86/kvm/irq_comm.c void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, · line 428
Quoted 9 lines
arch/x86/kvm/irq_comm.c +2
-4@@ -424,10 +424,12 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,kvm_set_msi_irq (vcpu->kvm, entry, &irq); -if (irq.trig_mode && - (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , - irq.dest_id, irq.dest_mode) || -kvm_apic_pending_eoi (vcpu, irq.vector))) +if (!irq.trig_mode) +continue ;Perhaps take this chance to make it explicit?
Quoted 4 lines
if (irq.trig_mode != IOAPIC_LEVEL_TRIG) continue;
kvm_ioapic_scan_entry()also checks againstIOAPIC_LEVEL_TRIGexplicitly.Hmm, I'm leaning "no".
kvm_set_msi_irq()isn't I/O APIC specific (and obviously neither is "structkvm_lapic_irq"). The fact that it sets irq.trig_mode to '0' or '1', and that the '1' value in particular happens to matchIOAPIC_LEVEL_TRIGis somewhat of a coincidence.
Quoted 5 lines
kvm_ioapic_scan_entry()on the other operates on a "unionkvm_ioapic_redirect_entry" object, in which casetrig_modeis guaranteed to be '0' or '1', i.e. is exactlyIOAPIC_EDGE_TRIGorIOAPIC_LEVEL_TRIG.u8 trig_mode:1;
This makes sense. Thanks for clarifying.
Quoted 3 lines
So as much as I advocate for consistency, I think in this case it makes sense to be consistent with
__apic_accept_irq(), which only cares about zero vs. non-zero.
Yeah LGTM.
Sean Christopherson
maintainerPATCH v5 2/3 KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic
In reply to Sean ChristophersonExtract the vCPU specific EOI interception logic for I/O APIC emulation into a common helper for userspace and in-kernel emulation in anticipation of optimizing the "pending EOI" case.
No functional change intended.
--- arch/x86/kvm/ioapic.c | 7 ++----- arch/x86/kvm/ioapic.h | 2 ++ arch/x86/kvm/irq_comm.c | 21 +++++++++++++++++---- 3 files changed, 21 insertions(+), 9 deletions(-)
arch/x86/kvm/ioapic.c +2 -5
@@ -296,11 +296,8 @@ void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, ulong *ioapic_handled_vectors) index == RTC_GSI) {u16 dm =kvm_lapic_irq_dest_mode (!!e->fields.dest_mode); -if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , - e->fields.dest_id, dm) || -kvm_apic_pending_eoi (vcpu, e->fields.vector)) -__set_bit (e->fields.vector, - ioapic_handled_vectors); +kvm_scan_ioapic_irq (vcpu, e->fields.dest_id, dm, + e->fields.vector, ioapic_handled_vectors); } }spin_unlock (&ioapic->lock);
arch/x86/kvm/ioapic.h +2 -0
@@ -120,4 +120,6 @@ void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, ulong *ioapic_handled_vectors);void kvm_scan_ioapic_routes (struct kvm_vcpu *vcpu,ulong *ioapic_handled_vectors); +void kvm_scan_ioapic_irq (struct kvm_vcpu *vcpu,u32 dest_id,u16 dest_mode, +u8 vector,unsigned long *ioapic_handled_vectors);#endif
arch/x86/kvm/irq_comm.c +17 -4
@@ -402,6 +402,21 @@ void kvm_arch_post_irq_routing_update(struct kvm *kvm)kvm_make_scan_ioapic_request (kvm); } +void kvm_scan_ioapic_irq (struct kvm_vcpu *vcpu,u32 dest_id,u16 dest_mode, +u8 vector,unsigned long *ioapic_handled_vectors) +{ +/* +* Intercept EOI if the vCPU is the target of the new IRQ routing, or +* the vCPU has a pending IRQ from the old routing, i.e. if the vCPU +* may receive a level-triggered IRQ in the future, or already received +* level-triggered IRQ. The EOI needs to be intercepted and forwarded +* to I/O APIC emulation so that the IRQ can be de-asserted. +*/ +if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , dest_id, dest_mode) || +kvm_apic_pending_eoi (vcpu, vector)) +__set_bit (vector, ioapic_handled_vectors); +} +void kvm_scan_ioapic_routes (struct kvm_vcpu *vcpu,ulong *ioapic_handled_vectors) {
@@ -427,10 +442,8 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,if (!irq.trig_mode)continue ; -if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , - irq.dest_id, irq.dest_mode) || -kvm_apic_pending_eoi (vcpu, irq.vector)) -__set_bit (irq.vector, ioapic_handled_vectors); +kvm_scan_ioapic_irq (vcpu, irq.dest_id, irq.dest_mode, + irq.vector, ioapic_handled_vectors); } }srcu_read_unlock (&kvm->irq_srcu, idx);
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Re: [PATCH v5 2/3] KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic
Quoted 8 lines · Sean Christopherson
Extract the vCPU specific EOI interception logic for I/O APIC emulation into a common helper for userspace and in-kernel emulation in anticipation of optimizing the "pending EOI" case.
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerPATCH v5 3/3 KVM: x86: Rescan I/O APIC routes after EOI interception for old routing
In reply to Sean ChristophersonFrom: weizijie <[email protected]>
Rescan I/O APIC routes for a vCPU after handling an intercepted I/O APIC EOI for an IRQ that is not targeting said vCPU, i.e. after handling what's effectively a stale EOI VM-Exit. If a level-triggered IRQ is in-flight when IRQ routing changes, e.g. because the guest change routing from its IRQ handler, then KVM intercepts EOIs on both the new and old target vCPUs, so that the in-flight IRQ can be de-asserted when it's EOI'd.
However, only the EOI for the in-flight IRQ needs to intercepted, as IRQs on the same vector with the new routing are coincidental, i.e. occur only if the guest is reusing the vector for multiple interrupt sources. If the I/O APIC routes aren't rescanned, KVM will unnecessarily intercept EOIs for the vector and negative impact the vCPU's interrupt performance.
Note, both commit db2bdcbbbd32 ("KVM: x86: fix edge EOI and IOAPIC reconfig race") and commit 0fc5a36dd6b3 ("KVM: x86: ioapic: Fix level-triggered EOI and IOAPIC reconfigure race") mentioned this issue, but it was considered a "rare" occurrence thus was not addressed. However in real environments, this issue can happen even in a well-behaved guest.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/irq_comm.c | 16 ++++++++++++++-- arch/x86/kvm/lapic.c | 8 ++++++++ arch/x86/kvm/x86.c | 1 + 4 files changed, 24 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1025,6 +1025,7 @@ struct kvm_vcpu_arch {int pending_ioapic_eoi;int pending_external_vector; +int highest_stale_pending_ioapic_eoi;/* be preempted when it's in kernel-mode(cpl=0) */ bool preempted_in_kernel;
arch/x86/kvm/irq_comm.c +14 -2
@@ -412,9 +412,21 @@ void kvm_scan_ioapic_irq(struct kvm_vcpu *vcpu, u32 dest_id, u16 dest_mode,* level-triggered IRQ. The EOI needs to be intercepted and forwarded * to I/O APIC emulation so that the IRQ can be de-asserted. */ -if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , dest_id, dest_mode) || -kvm_apic_pending_eoi (vcpu, vector)) +if (kvm_apic_match_dest (vcpu,NULL ,APIC_DEST_NOSHORT , dest_id, dest_mode)) {__set_bit (vector, ioapic_handled_vectors); + }else if (kvm_apic_pending_eoi (vcpu, vector)) { +__set_bit (vector, ioapic_handled_vectors); + +/* +* Track the highest pending EOI for which the vCPU is NOT the +* target in the new routing. Only the EOI for the IRQ that is +* in-flight (for the old routing) needs to be intercepted, any +* future IRQs that arrive on this vCPU will be coincidental to +* the level-triggered routing and don't need to be intercepted. +*/ +if ((int )vector > vcpu->arch.highest_stale_pending_ioapic_eoi) + vcpu->arch.highest_stale_pending_ioapic_eoi = vector; + } } void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu,
arch/x86/kvm/lapic.c +8 -0
@@ -1459,6 +1459,14 @@ static void kvm_ioapic_send_eoi(struct kvm_lapic *apic, int vector)if (!kvm_ioapic_handles_vector (apic, vector))return ; +/* +* If the intercepted EOI is for an IRQ that was pending from previous +* routing, then re-scan the I/O APIC routes as EOIs for the IRQ likely +* no longer need to be intercepted. +*/ +if (apic->vcpu->arch.highest_stale_pending_ioapic_eoi == vector) +kvm_make_request (KVM_REQ_SCAN_IOAPIC , apic->vcpu); +/* Request a KVM exit to inform the userspace IOAPIC. */ if (irqchip_split (apic->vcpu->kvm)) { apic->vcpu->arch.pending_ioapic_eoi = vector;
arch/x86/kvm/x86.c +1 -0
@@ -10650,6 +10650,7 @@ static void vcpu_scan_ioapic(struct kvm_vcpu *vcpu)return ;bitmap_zero (vcpu->arch.ioapic_handled_vectors,256 ); + vcpu->arch.highest_stale_pending_ioapic_eoi =-1 ;kvm_x86_call (sync_pir_to_irr)(vcpu);
Signature · 1 line
2.48.1.711.g2feabab25a-goog
Re: [PATCH v5 3/3] KVM: x86: Rescan I/O APIC routes after EOI interception for old routing
Quoted 6 lines · Sean Christopherson
From: weizijie <[email protected]>
Rescan I/O APIC routes for a vCPU after handling an intercepted I/O APIC EOI for an IRQ that is not targeting said vCPU, i.e. after handling what's effectively a stale EOI VM-Exit. If a level-triggered IRQ is in-flight when IRQ routing changes, e.g. because the guest change routing from its
^ changes ?
Quoted 4 lines
IRQ handler, then KVM intercepts EOIs on both the new and old target vCPUs, so that the in-flight IRQ can be de-asserted when it's EOI'd.
However, only the EOI for the in-flight IRQ needs to intercepted, as IRQs
^ be intercepted
Quoted 11 lines
on the same vector with the new routing are coincidental, i.e. occur only if the guest is reusing the vector for multiple interrupt sources. If the I/O APIC routes aren't rescanned, KVM will unnecessarily intercept EOIs for the vector and negative impact the vCPU's interrupt performance.
Note, both commit
db2bdcbbbd32("KVM: x86: fix edge EOI and IOAPIC reconfig race") and commit0fc5a36dd6b3("KVM: x86: ioapic: Fix level-triggered EOI and IOAPIC reconfigure race") mentioned this issue, but it was considered a "rare" occurrence thus was not addressed. However in real environments, this issue can happen even in a well-behaved guest.
Quoted 6 lines
Cc Kai Huang <[email protected]>Co-developed-by xuyun <[email protected]>Signed-off-by xuyun <[email protected]>Signed-off-by weizijie <[email protected]>[sean: massage changelog and comments, use int/-1, reset at scan]Signed-off-by Sean Christopherson <[email protected]>
Quoted 8 lines · Sean Christopherson
Slightly modified version of the patch (now mini-series) to optimize EOI bitmap exiting for "stale" routes, i.e. for the case where a vCPU has an in-flight IRQ when the routing changes, and needs one final EOI exit to deassert the IRQ.
Kai, I dropped your Reviewed-by as the change was non-trivial.
[...]
Applied to kvm-x86 misc, thanks!
[1/3] KVM: x86: Isolate edge vs. level check in userspace I/O APIC route scanning
commit: b1f7723a5a5b018f4bc3fb8e234510be7c44ad00
[2/3] KVM: x86: Add a helper to deduplicate I/O APIC EOI interception logic
commit: c2207bbc0c0f4b6ae8dbb73ec26e17aa0c45a3ca
[3/3] KVM: x86: Rescan I/O APIC routes after EOI interception for old routing
commit: 87e4951e250bbccac47a8822f6f023a1de8b96ec