KVM: arm64: vgic-v3: MMIO-based LPI invalidation and co
Subsystems (4)
Marc Zyngier
maintainerPATCH v2 1/4 irqchip/gic-v3: Exposes bit values for GICR_CTLR.{IR,CES}
In reply to Marc ZyngierAs we're about to expose GICR_CTLR.{IR,CES} to guests, populate the include file with the architectural values.
--- include/linux/irqchip/arm-gic-v3.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/irqchip/arm-gic-v3.h +2 -0
@@ -127,6 +127,8 @@#define GICR_PIDR2 GICD_PIDR2#define GICR_CTLR_ENABLE_LPIS (1UL << 0) +#define GICR_CTLR_CES (1UL << 1) +#define GICR_CTLR_IR (1UL << 2)#define GICR_CTLR_RWP (1UL << 3)#define GICR_TYPER_CPU_NUMBER (r) (((r) >> 8) & 0xffff)
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerv2 cover KVM: arm64: vgic-v3: MMIO-based LPI invalidation and co
Since revision IHI0069G of the GICv3 spec, an implementation is allowed to implement MMIO-based LPI invalidation, without having to support RVPEI (which is essentially a GICv4.1 feature).
This has the potential to make workloads using heavy LPI invalidation fare a bit better, as they don't need to lock the access to the command queue.
Similarly, an implementation can now expose that it allows LPIs to be turned off, something that we always supported.
This series implements both these features, exposing the new GICR_{INVLPIR,INVALLR,SYNCR} registers, transitions of the GICR_CTLR.RWP bit on LPI disabling, and finally exposes these to userspace and the guest with a new GICD_IIDR revision (and the ability to save/restore it).
This series has been extremely useful to debug related GIC features, and will be complemented by a few GIC driver patches.
* From v1 [1]:
- Fixed CES and IR bit numbers (shrug...)
- Plenty of small fixes all over the shop thanks to Oliver
- Rebased on top of 5.18-rc1
[1] https://lore.kernel.org/r/[email protected]
Marc Zyngier (4):
irqchip/gic-v3: Exposes bit values for GICR_CTLR.{IR,CES}
KVM: arm64: vgic-v3: Expose GICR_CTLR.RWP when disabling LPIs
KVM: arm64: vgic-v3: Implement MMIO-based LPI invalidation
KVM: arm64: vgic-v3: Advertise GICR_CTLR.{IR,CES} as a new GICD_IIDR
revision
arch/arm64/kvm/vgic/vgic-init.c | 7 +- arch/arm64/kvm/vgic/vgic-its.c | 64 ++++++++++----- arch/arm64/kvm/vgic/vgic-mmio-v2.c | 18 ++++- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 125 ++++++++++++++++++++++++++--- arch/arm64/kvm/vgic/vgic.h | 10 +++ include/kvm/arm_vgic.h | 8 +- include/linux/irqchip/arm-gic-v3.h | 2 + 7 files changed, 195 insertions(+), 39 deletions(-)
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 2/4 KVM: arm64: vgic-v3: Expose GICR_CTLR.RWP when disabling LPIs
When disabling LPIs, a guest needs to poll GICR_CTLR.RWP in order to be sure that the write has taken effect. We so far reported it as 0, as we didn't advertise that LPIs could be turned off the first place.
Start tracking this state during which LPIs are being disabled, and expose the 'in progress' state via the RWP bit.
We also take this opportunity to disallow enabling LPIs and programming GICR_{PEND,PROP}BASER while LPI disabling is in progress, as allowed by the architecture (UNPRED behaviour).
We don't advertise the feature to the guest yet (which is allowed by the architecture).
--- arch/arm64/kvm/vgic/vgic-its.c | 2 +- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 36 ++++++++++++++++++++++-------- arch/arm64/kvm/vgic/vgic.h | 1 + include/kvm/arm_vgic.h | 4 ++-- 4 files changed, 31 insertions(+), 12 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +1 -1
@@ -683,7 +683,7 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,if (!vcpu)return E_ITS_INT_UNMAPPED_INTERRUPT ; -if (!vcpu->arch.vgic_cpu.lpis_enabled) +if (!vgic_lpis_enabled (vcpu))return -EBUSY ;vgic_its_cache_translation (kvm, its, devid, eventid, ite->irq);
arch/arm64/kvm/vgic/vgic-mmio-v3.c +27 -9
@@ -221,6 +221,13 @@ static void vgic_mmio_write_irouter(struct kvm_vcpu *vcpu,vgic_put_irq (vcpu->kvm, irq); } +bool vgic_lpis_enabled (struct kvm_vcpu *vcpu) +{ +struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + +return atomic_read (&vgic_cpu->ctlr) ==GICR_CTLR_ENABLE_LPIS ; +} +static unsigned long vgic_mmio_read_v3r_ctlr (struct kvm_vcpu *vcpu,gpa_t addr,unsigned int len) {
@@ -229,26 +236,38 @@ static unsigned long vgic_mmio_read_v3r_ctlr(struct kvm_vcpu *vcpu,return vgic_cpu->lpis_enabled ?GICR_CTLR_ENABLE_LPIS :0 ; } -static void vgic_mmio_write_v3r_ctlr (struct kvm_vcpu *vcpu,gpa_t addr,unsigned int len,unsigned long val) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; -bool was_enabled = vgic_cpu->lpis_enabled; +u32 ctlr;if (!vgic_has_its (vcpu->kvm))return ; - vgic_cpu->lpis_enabled = val &GICR_CTLR_ENABLE_LPIS ; +if (!(val &GICR_CTLR_ENABLE_LPIS )) { +/* +* Don't disable if RWP is set, as there already an +* ongoing disable. Funky guest... +*/ + ctlr =atomic_cmpxchg_acquire (&vgic_cpu->ctlr, +GICR_CTLR_ENABLE_LPIS , +GICR_CTLR_RWP ); +if (ctlr !=GICR_CTLR_ENABLE_LPIS ) +return ; -if (was_enabled && !vgic_cpu->lpis_enabled) {vgic_flush_pending_lpis (vcpu);vgic_its_invalidate_cache (vcpu->kvm); - } +atomic_set_release (&vgic_cpu->ctlr,0 ); + }else { + ctlr =atomic_cmpxchg_acquire (&vgic_cpu->ctlr,0 , +GICR_CTLR_ENABLE_LPIS ); +if (ctlr !=0 ) +return ; -if (!was_enabled && vgic_cpu->lpis_enabled)vgic_enable_lpis (vcpu); + } }static bool vgic_mmio_vcpu_rdist_is_last (struct kvm_vcpu *vcpu)
@@ -478,11 +497,10 @@ static void vgic_mmio_write_propbase(struct kvm_vcpu *vcpu, unsigned long val) {struct vgic_dist *dist = &vcpu->kvm->arch.vgic; -struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;u64 old_propbaser, propbaser;/* Storing a value with LPIs already enabled is undefined */ -if (vgic_cpu->lpis_enabled) +if (vgic_lpis_enabled (vcpu))return ; do {
@@ -513,7 +531,7 @@ static void vgic_mmio_write_pendbase(struct kvm_vcpu *vcpu,u64 old_pendbaser, pendbaser;/* Storing a value with LPIs already enabled is undefined */ -if (vgic_cpu->lpis_enabled) +if (vgic_lpis_enabled (vcpu))return ; do {
arch/arm64/kvm/vgic/vgic.h +1 -0
@@ -308,6 +308,7 @@ static inline bool vgic_dist_overlap(struct kvm *kvm, gpa_t base, size_t size) (base < d->vgic_dist_base +KVM_VGIC_V3_DIST_SIZE ); } +bool vgic_lpis_enabled (struct kvm_vcpu *vcpu);int vgic_copy_lpi_list (struct kvm *kvm,struct kvm_vcpu *vcpu,u32 **intid_ptr);int vgic_its_resolve_lpi (struct kvm *kvm,struct vgic_its *its,u32 devid,u32 eventid,struct vgic_irq **irq);
include/kvm/arm_vgic.h +2 -2
@@ -347,8 +347,8 @@ struct vgic_cpu {/* Contains the attributes and gpa of the LPI pending tables. */ u64 pendbaser; - -bool lpis_enabled; +/* GICR_CTLR.{ENABLE_LPIS,RWP} */ +atomic_t ctlr;/* Cache guest priority bits */ u32 num_pri_bits;
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 3/4 KVM: arm64: vgic-v3: Implement MMIO-based LPI invalidation
In reply to Marc ZyngierSince GICv4.1, it has become legal for an implementation to advertise GICR_{INVLPIR,INVALLR,SYNCR} while having an ITS, allowing for a more efficient invalidation scheme (no guest command queue contention when multiple CPUs are generating invalidations).
Provide the invalidation registers as a primitive to their ITS counterpart. Note that we don't advertise them to the guest yet (the architecture allows an implementation to do this).
--- arch/arm64/kvm/vgic/vgic-its.c | 62 ++++++++++++++++++---------- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 66 ++++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic.h | 4 ++ include/kvm/arm_vgic.h | 1 + 4 files changed, 112 insertions(+), 21 deletions(-)
arch/arm64/kvm/vgic/vgic-its.c +41 -21
@@ -1272,6 +1272,11 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,return 0 ; } +int vgic_its_inv_lpi (struct kvm *kvm,struct vgic_irq *irq) +{ +return update_lpi_config (kvm, irq,NULL , true); +} +/* * The INV command syncs the configuration bits from the memory table. * Must be called with the its_lock mutex held.
@@ -1288,7 +1293,41 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,if (!ite)return E_ITS_INV_UNMAPPED_INTERRUPT ; -return update_lpi_config (kvm, ite->irq,NULL , true); +return vgic_its_inv_lpi (kvm, ite->irq); +} + +/** +* vgic_its_invall - invalidate all LPIs targetting a given vcpu +* @vcpu: the vcpu for which the RD is targetted by an invalidation +* +* Contrary to the INVALL command, this targets a RD instead of a +* collection, and we don't need to hold the its_lock, since no ITS is +* involved here. +*/ +int vgic_its_invall (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +int irq_count, i =0 ; +u32 *intids; + + irq_count =vgic_copy_lpi_list (kvm, vcpu, &intids); +if (irq_count <0 ) +return irq_count; + +for (i =0 ; i < irq_count; i++) { +struct vgic_irq *irq =vgic_get_irq (kvm,NULL , intids[i]); +if (!irq) +continue ; +update_lpi_config (kvm, irq, vcpu, false); +vgic_put_irq (kvm, irq); + } + +kfree (intids); + +if (vcpu->arch.vgic_cpu.vgic_v3.its_vpe.its_vm) +its_invall_vpe (&vcpu->arch.vgic_cpu.vgic_v3.its_vpe); + +return 0 ; }/*
@@ -1305,32 +1344,13 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,u32 coll_id =its_cmd_get_collection (its_cmd);struct its_collection *collection;struct kvm_vcpu *vcpu; -struct vgic_irq *irq; -u32 *intids; -int irq_count, i; collection =find_collection (its, coll_id);if (!its_is_collection_mapped (collection))return E_ITS_INVALL_UNMAPPED_COLLECTION ; vcpu =kvm_get_vcpu (kvm, collection->target_addr); - - irq_count =vgic_copy_lpi_list (kvm, vcpu, &intids); -if (irq_count <0 ) -return irq_count; - -for (i =0 ; i < irq_count; i++) { - irq =vgic_get_irq (kvm,NULL , intids[i]); -if (!irq) -continue ; -update_lpi_config (kvm, irq, vcpu, false); -vgic_put_irq (kvm, irq); - } - -kfree (intids); - -if (vcpu->arch.vgic_cpu.vgic_v3.its_vpe.its_vm) -its_invall_vpe (&vcpu->arch.vgic_cpu.vgic_v3.its_vpe); +vgic_its_invall (vcpu);return 0 ; }
arch/arm64/kvm/vgic/vgic-mmio-v3.c +66 -0
78 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.h +4 -0
@@ -318,6 +318,10 @@ void vgic_lpi_translation_cache_init(struct kvm *kvm);void vgic_lpi_translation_cache_destroy (struct kvm *kvm);void vgic_its_invalidate_cache (struct kvm *kvm); +/* GICv4.1 MMIO interface */ +int vgic_its_inv_lpi (struct kvm *kvm,struct vgic_irq *irq); +int vgic_its_invall (struct kvm_vcpu *vcpu); +bool vgic_supports_direct_msis (struct kvm *kvm);int vgic_v4_init (struct kvm *kvm);void vgic_v4_teardown (struct kvm *kvm);
include/kvm/arm_vgic.h +1 -0
@@ -344,6 +344,7 @@ struct vgic_cpu {struct vgic_io_device rd_iodev;struct vgic_redist_region *rdreg;u32 rdreg_index; +atomic_t syncr_busy;/* Contains the attributes and gpa of the LPI pending tables. */ u64 pendbaser;
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v2 4/4 KVM: arm64: vgic-v3: Advertise GICR_CTLR.{IR,CES} as a new GICD_IIDR revision
In reply to Marc ZyngierSince adversising GICR_CTLR.{IC,CES} is directly observable from a guest, we need to make it selectable from userspace.
For that, bump the default GICD_IIDR revision and let userspace downgrade it to the previous default. For GICv2, the two distributor revisions are strictly equivalent.
--- arch/arm64/kvm/vgic/vgic-init.c | 7 ++++++- arch/arm64/kvm/vgic/vgic-mmio-v2.c | 18 +++++++++++++++--- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 23 +++++++++++++++++++++-- arch/arm64/kvm/vgic/vgic.h | 5 +++++ include/kvm/arm_vgic.h | 3 +++ 5 files changed, 50 insertions(+), 6 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +6 -1
@@ -319,7 +319,12 @@ int vgic_init(struct kvm *kvm)vgic_debug_init (kvm); - dist->implementation_rev =2 ; +/* +* If userspace didn't set the GIC implementation revision, +* default to the latest and greatest. You know want it. +*/ +if (!dist->implementation_rev) + dist->implementation_rev =KVM_VGIC_IMP_REV_LATEST ; dist->initialized = true; out:
arch/arm64/kvm/vgic/vgic-mmio-v2.c +15 -3
@@ -73,9 +73,13 @@ static int vgic_mmio_uaccess_write_v2_misc(struct kvm_vcpu *vcpu, gpa_t addr, unsigned int len, unsigned long val) { + struct vgic_dist *dist = &vcpu->kvm->arch.vgic; +u32 reg; +switch (addr &0x0c ) {case GIC_DIST_IIDR : -if (val !=vgic_mmio_read_v2_misc (vcpu, addr, len)) + reg =vgic_mmio_read_v2_misc (vcpu, addr, len); +if ((reg ^ val) & ~GICD_IIDR_REVISION_MASK )return -EINVAL ;/*
@@ -87,8 +91,16 @@ static int vgic_mmio_uaccess_write_v2_misc(struct kvm_vcpu *vcpu,* migration from old kernels to new kernels with legacy * userspace. */ - vcpu->kvm->arch.vgic.v2_groups_user_writable = true; -return 0 ; + reg =FIELD_GET (GICD_IIDR_REVISION_MASK , reg); +switch (reg) { +case KVM_VGIC_IMP_REV_2 : +case KVM_VGIC_IMP_REV_3 : + vcpu->kvm->arch.vgic.v2_groups_user_writable = true; + dist->implementation_rev = reg; +return 0 ; +default : +return -EINVAL ; + } }vgic_mmio_write_v2_misc (vcpu, addr, len, val);
arch/arm64/kvm/vgic/vgic-mmio-v3.c +21 -2
@@ -155,13 +155,27 @@ static int vgic_mmio_uaccess_write_v3_misc(struct kvm_vcpu *vcpu, unsigned long val) {struct vgic_dist *dist = &vcpu->kvm->arch.vgic; +u32 reg;switch (addr &0x0c ) {case GICD_TYPER2 : -case GICD_IIDR :if (val !=vgic_mmio_read_v3_misc (vcpu, addr, len))return -EINVAL ;return 0 ; +case GICD_IIDR : + reg =vgic_mmio_read_v3_misc (vcpu, addr, len); +if ((reg ^ val) & ~GICD_IIDR_REVISION_MASK ) +return -EINVAL ; + + reg =FIELD_GET (GICD_IIDR_REVISION_MASK , reg); +switch (reg) { +case KVM_VGIC_IMP_REV_2 : +case KVM_VGIC_IMP_REV_3 : + dist->implementation_rev = reg; +return 0 ; +default : +return -EINVAL ; + }case GICD_CTLR :/* Not a GICv4.1? No HW SGIs */ if (!kvm_vgic_global_state.has_gicv4_1)
@@ -232,8 +246,13 @@ static unsigned long vgic_mmio_read_v3r_ctlr(struct kvm_vcpu *vcpu, gpa_t addr, unsigned int len) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; +unsigned long val; + + val =atomic_read (&vgic_cpu->ctlr); +if (vgic_get_implementation_rev (vcpu) >=KVM_VGIC_IMP_REV_3 ) + val |=GICR_CTLR_IR |GICR_CTLR_CES ; -return vgic_cpu->lpis_enabled ?GICR_CTLR_ENABLE_LPIS :0 ; +return val; } static void vgic_mmio_write_v3r_ctlr(struct kvm_vcpu *vcpu,
arch/arm64/kvm/vgic/vgic.h +5 -0
@@ -98,6 +98,11 @@#define DEBUG_SPINLOCK_BUG_ON (p)#endif +static inline u32 vgic_get_implementation_rev (struct kvm_vcpu *vcpu) +{ +return vcpu->kvm->arch.vgic.implementation_rev; +} +/* Requires the irq_lock to be held by the caller. */ static inline bool irq_is_pending (struct vgic_irq *irq) {
include/kvm/arm_vgic.h +3 -0
@@ -231,6 +231,9 @@ struct vgic_dist {/* Implementation revision as reported in the GICD_IIDR */ u32 implementation_rev; +#define KVM_VGIC_IMP_REV_2 2/* GICv2 restorable groups */ +#define KVM_VGIC_IMP_REV_3 3/* GICv3 GICR_CTLR.{IW,CES,RWP} */ +#define KVM_VGIC_IMP_REV_LATEST KVM_VGIC_IMP_REV_3/* Userspace can write to GICv2 IGROUPR */ bool v2_groups_user_writable;
Signature · 6 lines
2.34.1 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 1/4] irqchip/gic-v3: Exposes bit values for GICR_CTLR.{IR,CES}
In reply to Marc ZyngierQuoted 4 lines · Marc Zyngier
As we're about to expose GICR_CTLR.{IR,CES} to guests, populate the include file with the architectural values.
Signed-off-by Marc Zyngier <[email protected]>
Quoted 20 lines
--- include/linux/irqchip/arm-gic-v3.h | 2 ++ 1 file changed, 2 insertions(+)include/linux/irqchip/arm-gic-v3.h +2
-0@@ -127,6 +127,8 @@#define GICR_PIDR2 GICD_PIDR2#define GICR_CTLR_ENABLE_LPIS (1UL << 0) +#define GICR_CTLR_CES (1UL << 1) +#define GICR_CTLR_IR (1UL << 2)#define GICR_CTLR_RWP (1UL << 3)#define GICR_TYPER_CPU_NUMBER (r) (((r) >> 8) & 0xffff)Signature · 1 line
2.34.1
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 3/4] KVM: arm64: vgic-v3: Implement MMIO-based LPI invalidation
In reply to Marc ZyngierQuoted 9 lines · Marc Zyngier
Since GICv4.1, it has become legal for an implementation to advertise GICR_{INVLPIR,INVALLR,SYNCR} while having an ITS, allowing for a more efficient invalidation scheme (no guest command queue contention when multiple CPUs are generating invalidations).
Provide the invalidation registers as a primitive to their ITS counterpart. Note that we don't advertise them to the guest yet (the architecture allows an implementation to do this).
I'll admit that this part tripped me up a bit, odd stuff.
Quoted 1 line
Signed-off-by Marc Zyngier <[email protected]>
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[irqchip: irq/irqchip-next] irqchip/gic-v3: Exposes bit values for GICR_CTLR.{IR, CES}
In reply to Marc ZyngierThe following commit has been merged into the irq/irqchip-next branch of irqchip:
The following commit has been merged into the irq/irqchip-next branch of irqchip:
Commit-ID: 34453c2e9f799d02f5f379519495208bbd96a935 Gitweb: https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/34453c2e9f799d02f5f379519495208bbd96a935 Author: Marc Zyngier <[email protected]> AuthorDate: Tue, 05 Apr 2022 19:23:24 +01:00 Committer: Marc Zyngier <[email protected]> CommitterDate: Wed, 04 May 2022 14:09:52 +01:00
irqchip/gic-v3: Exposes bit values for GICR_CTLR.{IR, CES}
As we're about to expose GICR_CTLR.{IR,CES} to guests, populate the include file with the architectural values.
--- include/linux/irqchip/arm-gic-v3.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/irqchip/arm-gic-v3.h +2 -0
@@ -127,6 +127,8 @@#define GICR_PIDR2 GICD_PIDR2#define GICR_CTLR_ENABLE_LPIS (1UL << 0) +#define GICR_CTLR_CES (1UL << 1) +#define GICR_CTLR_IR (1UL << 2)#define GICR_CTLR_RWP (1UL << 3)#define GICR_TYPER_CPU_NUMBER (r) (((r) >> 8) & 0xffff)