KVM: arm64: Reduce occurence of GICv4 doorbells on non-oversubscribed systems
Subsystems (5)
Marc Zyngier
maintainercover KVM: arm64: Reduce occurence of GICv4 doorbells on non-oversubscribed systems
As I was cleaning up some of the GICv4 code to make way for GICv4.1 it occured to me that we could drastically reduce the impact of the GICv4 doorbells on systems that are not oversubscribed (each vcpu "owns" a physical CPU).
The technique borrows its logic from the way we disable WFE trapping when a vcpu is the only process on the CPU run-queue. If this vcpu is the target of VLPIs, it is then beneficial not to trap blocking WFIs and to leave the vcpu waiting for interrupts in guest state.
All we need to do here is to track whether VLPIs are associated to a vcpu (which is easily done by using a counter that we update on MAPI, DISCARD and MOVI).
It has been *very lightly* tested on a D05, and behaved pretty well in my limited test cases (I get almost no doorbell at all in the non oversubscribed case, and the usual hailstorm as soon as there is oversubscription). I'd welcome some testing on more current HW.
Marc Zyngier (2):
KVM: vgic-v4: Track the number of VLPIs per vcpu
KVM: arm64: Opportunistically turn off WFI trapping when using direct
LPI injection
arch/arm/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_emulate.h | 9 +++++++-- include/linux/irqchip/arm-gic-v4.h | 2 ++ virt/kvm/arm/arm.c | 4 ++-- virt/kvm/arm/vgic/vgic-init.c | 1 + virt/kvm/arm/vgic/vgic-its.c | 3 +++ virt/kvm/arm/vgic/vgic-v4.c | 2 ++ 7 files changed, 19 insertions(+), 6 deletions(-)
Signature · 1 line
2.20.1
Marc Zyngier
maintainerPATCH 2/2 KVM: arm64: Opportunistically turn off WFI trapping when using direct LPI injection
Just like we do for WFE trapping, it can be useful to turn off WFI trapping when the physical CPU is not oversubscribed (that is, the vcpu is the only runnable process on this CPU) *and* that we're using direct injection of interrupts.
The conditions are reevaluated on each vcpu_load(), ensuring that we don't switch to this mode on a busy system.
On a GICv4 system, this has the effect of reducing the generation of doorbell interrupts to zero when the right conditions are met, which is a huge improvement over the current situation (where the doorbells are screaming if the CPU ever hits a blocking WFI).
--- arch/arm/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_emulate.h | 9 +++++++-- virt/kvm/arm/arm.c | 4 ++-- 3 files changed, 11 insertions(+), 6 deletions(-)
arch/arm/include/asm/kvm_emulate.h +2 -2
@@ -95,12 +95,12 @@ static inline unsigned long *vcpu_hcr(const struct kvm_vcpu *vcpu)return (unsigned long *)&vcpu->arch.hcr; } -static inline void vcpu_clear_wfe_traps (struct kvm_vcpu *vcpu) +static inline void vcpu_clear_wfx_traps (struct kvm_vcpu *vcpu) { vcpu->arch.hcr &= ~HCR_TWE ; } -static inline void vcpu_set_wfe_traps (struct kvm_vcpu *vcpu) +static inline void vcpu_set_wfx_traps (struct kvm_vcpu *vcpu) { vcpu->arch.hcr |=HCR_TWE ; }
arch/arm64/include/asm/kvm_emulate.h +7 -2
@@ -87,14 +87,19 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)return (unsigned long *)&vcpu->arch.hcr_el2; } -static inline void vcpu_clear_wfe_traps (struct kvm_vcpu *vcpu) +static inline void vcpu_clear_wfx_traps (struct kvm_vcpu *vcpu) { vcpu->arch.hcr_el2 &= ~HCR_TWE ; +if (atomic_read (&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count)) + vcpu->arch.hcr_el2 &= ~HCR_TWI ; +else + vcpu->arch.hcr_el2 |=HCR_TWI ; } -static inline void vcpu_set_wfe_traps (struct kvm_vcpu *vcpu) +static inline void vcpu_set_wfx_traps (struct kvm_vcpu *vcpu) { vcpu->arch.hcr_el2 |=HCR_TWE ; + vcpu->arch.hcr_el2 |=HCR_TWI ; }static inline void vcpu_ptrauth_enable (struct kvm_vcpu *vcpu)
virt/kvm/arm/arm.c +2 -2
@@ -416,9 +416,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)kvm_make_request (KVM_REQ_RECORD_STEAL , vcpu);if (single_task_running ()) -vcpu_clear_wfe_traps (vcpu); +vcpu_clear_wfx_traps (vcpu);else -vcpu_set_wfe_traps (vcpu); +vcpu_set_wfx_traps (vcpu);vcpu_ptrauth_setup_lazy (vcpu); }
Signature · 1 line
2.20.1
Marc Zyngier
maintainerPATCH 1/2 KVM: vgic-v4: Track the number of VLPIs per vcpu
In reply to Marc ZyngierIn order to find out whether a vcpu is likely to be the target of VLPIs (and to further optimize the way we deal with those), let's track the number of VLPIs a vcpu can receive.
This gets implemented with an atomic variable that gets incremented or decremented on map, unmap and move of a VLPI.
--- include/linux/irqchip/arm-gic-v4.h | 2 ++ virt/kvm/arm/vgic/vgic-init.c | 1 + virt/kvm/arm/vgic/vgic-its.c | 3 +++ virt/kvm/arm/vgic/vgic-v4.c | 2 ++ 4 files changed, 8 insertions(+)
include/linux/irqchip/arm-gic-v4.h +2 -0
@@ -32,6 +32,8 @@ struct its_vm {struct its_vpe {struct page *vpt_page;struct its_vm *its_vm; +/* per-vPE VLPI tracking */ +atomic_t vlpi_count;/* Doorbell interrupt */ int irq;irq_hw_number_t vpe_db_lpi;
virt/kvm/arm/vgic/vgic-init.c +1 -0
@@ -203,6 +203,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)INIT_LIST_HEAD (&vgic_cpu->ap_list_head);raw_spin_lock_init (&vgic_cpu->ap_list_lock); +atomic_set (&vgic_cpu->vgic_v3.its_vpe.vlpi_count,0 );/* * Enable and configure all SGIs to be edge-triggered and
virt/kvm/arm/vgic/vgic-its.c +3 -0
@@ -360,7 +360,10 @@ static int update_affinity(struct vgic_irq *irq, struct kvm_vcpu *vcpu)if (ret)return ret; +if (map.vpe) +atomic_dec (&map.vpe->vlpi_count); map.vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe; +atomic_inc (&map.vpe->vlpi_count); ret =its_map_vlpi (irq->host_irq, &map); }
virt/kvm/arm/vgic/vgic-v4.c +2 -0
@@ -309,6 +309,7 @@ int kvm_vgic_v4_set_forwarding(struct kvm *kvm, int virq, irq->hw = true; irq->host_irq = virq; +atomic_inc (&map.vpe->vlpi_count); out:mutex_unlock (&its->its_lock);
@@ -342,6 +343,7 @@ int kvm_vgic_v4_unset_forwarding(struct kvm *kvm, int virq,WARN_ON (!(irq->hw && irq->host_irq == virq));if (irq->hw) { +atomic_dec (&irq->target_vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count); irq->hw = false; ret =its_unmap_vlpi (virq); }
Signature · 1 line
2.20.1
Quoted 10 lines · Marc Zyngier
As I was cleaning up some of the GICv4 code to make way for GICv4.1 it occured to me that we could drastically reduce the impact of the GICv4 doorbells on systems that are not oversubscribed (each vcpu "owns" a physical CPU).
The technique borrows its logic from the way we disable WFE trapping when a vcpu is the only process on the CPU run-queue. If this vcpu is the target of VLPIs, it is then beneficial not to trap blocking WFIs and to leave the vcpu waiting for interrupts in guest state.
Quoted 9 lines
All we need to do here is to track whether VLPIs are associated to a vcpu (which is easily done by using a counter that we update on MAPI, DISCARD and MOVI).
It has been *very lightly* tested on a D05, and behaved pretty well in my limited test cases (I get almost no doorbell at all in the non oversubscribed case, and the usual hailstorm as soon as there is oversubscription). I'd welcome some testing on more current HW.
Re: [PATCH 2/2] KVM: arm64: Opportunistically turn off WFI trapping when using direct LPI injection
In reply to Marc ZyngierHi Marc,
Quoted 8 lines · Marc Zyngier
Just like we do for WFE trapping, it can be useful to turn off WFI trapping when the physical CPU is not oversubscribed (that is, the vcpu is the only runnable process on this CPU) *and* that we're using direct injection of interrupts.
The conditions are reevaluated on each
vcpu_load(), ensuring that we don't switch to this mode on a busy system.
Quoted 7 lines
On a GICv4 system, this has the effect of reducing the generation of doorbell interrupts to zero when the right conditions are met, which is a huge improvement over the current situation (where the doorbells are screaming if the CPU ever hits a blocking WFI).
Signed-off-by Marc Zyngier <[email protected]>
Re: [PATCH 1/2] KVM: vgic-v4: Track the number of VLPIs per vcpu
In reply to Marc ZyngierHi Marc,
Quoted 8 lines · Marc Zyngier
In order to find out whether a vcpu is likely to be the target of VLPIs (and to further optimize the way we deal with those), let's track the number of VLPIs a vcpu can receive.
This gets implemented with an atomic variable that gets incremented or decremented on map, unmap and move of a VLPI.
Signed-off-by Marc Zyngier <[email protected]>