KVM: arm64: Add LR overflow infrastructure
Subsystems (6)
Marc Zyngier
maintainercover KVM: arm64: Add LR overflow infrastructure
Despite being an awfully complex piece of software, the KVM vgic implementation is not doing what it should in a number of cases:
- It behaves very badly when the number of in-flight interrupts targeting a specific CPU exceed the number of list registers (LRs). This is catastrophic with NV (Volodymyr triggered that one with a Xen guest), but can easily be triggered without it. You just have to activate more interrupts than the number of LRs to end-up with a guest live-lock.
- Similar issues exist when making high-priority interrupts targeting a group that isn't enabled.
Overall, this stems from the vgic implementation not sorting the interrupts presented to the guest in the correct order (i.e. the one expected by the architecture), and allow deactivation outside of the window exposed by the LRs.
Unfortunately, the cure for this is pretty involved, and involves adding a significant amount of complexity:
- Order the list as the architecture requires it, by starting with placing the HPPIs at the forefront of the ap_list
- Handle deactivation for EOImode being 0 or 1, which is specially annoying as we cannot easily find out which mode we're in, and we need to handle both concurrently
- Deal with individual interrupt groups being enabled/disabled
- Deal with SPIs being acked on one CPU and deactivated on another
Implementation wise, this is about extracting the relevant primitive from the existing code, making guest state available to the emulation more eagerly, dealing with a lot more MI sources, and be creative about trapping stuff. Oh, and of course dealing with broken HW, because nothing would be fun if we didn't.
The eagle eyed reviewer will notice that this series doesn't provide any selftest. That's on purpose: I'm still working on those, and I could do with some reviewing while I'm finishing them, and plan to post them with v2.
Marc Zyngier (33):
irqchip/gic: Add missing GICH_HCR control bits irqchip/gic: Expose CPU interface VA to KVM irqchip/apple-aic: Spit out ICH_MIDR_EL2 value on spurious vGIC MI
KVM: arm64: Turn vgic-v3 errata traps into a patched-in constant
KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1
trapping
KVM: arm64: Repack struct vgic_irq fields
KVM: arm64: Add tracking of vgic_irq being present in a LR
KVM: arm64: Add LR overflow handling documentation
KVM: arm64: GICv3: Drop LPI active state when folding LRs
KVM: arm64: GICv3: Preserve EOIcount on exit
KVM: arm64: GICv3: Decouple ICH_HCR_EL2 programming from LRs
KVM: arm64: GICv3: Extract LR folding primitive
KVM: arm64: GICv3: Extract LR computing primitive
KVM: arm64: GICv2: Preserve EOIcount on exit
KVM: arm64: GICv2: Decouple GICH_HCR programming from LRs being loaded
KVM: arm64: GICv2: Extract LR folding primitive
KVM: arm64: GICv2: Extract LR computing primitive
KVM: arm64: Compute vgic state irrespective of the number of
interrupts
KVM: arm64: Eagerly save VMCR on exit
KVM: arm64: Revamp vgic maintenance interrupt configuration
KVM: arm64: Make vgic_target_oracle() globally available
KVM: arm64: Invert ap_list sorting to push active interrupts out
KVM: arm64: Move undeliverable interrupts to the end of ap_list
KVM: arm64: Use MI to detect groups being enabled/disabled
KVM: arm64: Add AP-list overflow split/splice
KVM: arm64: GICv3: Handle LR overflow when EOImode==0
KVM: arm64: GICv3: Handle deactivation via ICV_DIR_EL1 traps
KVM: arm64: GICv3: Add GICv2 SGI handling to deactivation primitive
KVM: arm64: GICv3: Set ICH_HCR_EL2.TDIR when interrupts overflow LR
capacity
KVM: arm64: GICv2: Handle LR overflow when EOImode==0
KVM: arm64: GICv2: Handle deactivation via GICV_DIR traps
KVM: arm64: GICv2: Always trap GICV_DIR register
KVM: arm64: GICv3: Add SPI tracking to handle asymmetric deactivation
arch/arm64/include/asm/kvm_asm.h | 2 +- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/kvm_hyp.h | 2 +- arch/arm64/include/asm/virt.h | 7 +- arch/arm64/kernel/cpufeature.c | 34 ++ arch/arm64/kernel/hyp-stub.S | 5 + arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kvm/arm.c | 7 +- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 7 +- arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 + arch/arm64/kvm/hyp/vgic-v3-sr.c | 55 +-- arch/arm64/kvm/sys_regs.c | 19 +- arch/arm64/kvm/vgic/vgic-init.c | 3 + arch/arm64/kvm/vgic/vgic-mmio-v2.c | 24 ++ arch/arm64/kvm/vgic/vgic-mmio.h | 1 + arch/arm64/kvm/vgic/vgic-v2.c | 311 +++++++++++++---- arch/arm64/kvm/vgic/vgic-v3-nested.c | 11 +- arch/arm64/kvm/vgic/vgic-v3.c | 419 ++++++++++++++++++----- arch/arm64/kvm/vgic/vgic-v4.c | 5 +- arch/arm64/kvm/vgic/vgic.c | 328 ++++++++++++------ arch/arm64/kvm/vgic/vgic.h | 44 ++- arch/arm64/tools/cpucaps | 1 + drivers/irqchip/irq-apple-aic.c | 7 +- drivers/irqchip/irq-gic.c | 3 + include/kvm/arm_vgic.h | 35 +- include/linux/irqchip/arm-gic.h | 6 + include/linux/irqchip/arm-vgic-info.h | 2 + 27 files changed, 1012 insertions(+), 332 deletions(-)
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 1/33 irqchip/gic: Add missing GICH_HCR control bits
The GICH_HCR description is missing a bunch of additional control bits for the maintenance interrupt. Add them.
--- include/linux/irqchip/arm-gic.h | 6 ++++++ 1 file changed, 6 insertions(+)
include/linux/irqchip/arm-gic.h +6 -0
@@ -86,7 +86,13 @@#define GICH_HCR_EN (1 << 0)#define GICH_HCR_UIE (1 << 1) +#define GICH_HCR_LRENPIE (1 << 2)#define GICH_HCR_NPIE (1 << 3) +#define GICH_HCR_VGrp0EIE (1 << 4) +#define GICH_HCR_VGrp0DIE (1 << 5) +#define GICH_HCR_VGrp1EIE (1 << 6) +#define GICH_HCR_VGrp1DIE (1 << 7) +#define GICH_HCR_EOICOUNT GENMASK(31, 27)#define GICH_LR_VIRTUALID (0x3ff << 0)#define GICH_LR_PHYSID_CPUID_SHIFT (10)
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 2/33 irqchip/gic: Expose CPU interface VA to KVM
In reply to Marc ZyngierFuture changes will require KVM to be able to perform deactivations by writing to the physical CPU interface. Add the corresponding VA to the kvm_info structure, and let KVM stash it.
--- arch/arm64/kvm/vgic/vgic-v2.c | 1 + drivers/irqchip/irq-gic.c | 3 +++ include/kvm/arm_vgic.h | 3 +++ include/linux/irqchip/arm-vgic-info.h | 2 ++ 4 files changed, 9 insertions(+)
arch/arm64/kvm/vgic/vgic-v2.c +1 -0
@@ -385,6 +385,7 @@ int vgic_v2_probe(const struct gic_kvm_info *info) kvm_vgic_global_state.can_emulate_gicv2 = true; kvm_vgic_global_state.vcpu_base = info->vcpu.start; + kvm_vgic_global_state.gicc_base = info->gicc_base; kvm_vgic_global_state.type =VGIC_V2 ; kvm_vgic_global_state.max_gic_vcpus =VGIC_V2_MAX_CPUS ;
drivers/irqchip/irq-gic.c +3 -0
@@ -1459,6 +1459,8 @@ static void __init gic_of_setup_kvm_info(struct device_node *node)if (ret)return ; + gic_v2_kvm_info.gicc_base = gic_data[0 ].cpu_base.common_base; +if (static_branch_likely (&supports_deactivate_key))vgic_set_kvm_info (&gic_v2_kvm_info); }
@@ -1620,6 +1622,7 @@ static void __init gic_acpi_setup_kvm_info(void)return ; gic_v2_kvm_info.maint_irq = irq; + gic_v2_kvm_info.gicc_base = gic_data[0 ].cpu_base.common_base;vgic_set_kvm_info (&gic_v2_kvm_info); }
include/kvm/arm_vgic.h +3 -0
@@ -59,6 +59,9 @@ struct vgic_global {/* virtual control interface mapping, HYP VA */ void __iomem *vctrl_hyp; +/* Physical CPU interface, kernel VA */ +void __iomem *gicc_base; +/* Number of implemented list registers */ int nr_lr;
include/linux/irqchip/arm-vgic-info.h +2 -0
@@ -24,6 +24,8 @@ struct gic_kvm_info {enum gic_type type;/* Virtual CPU interface */ struct resource vcpu; +/* GICv2 GICC VA */ +void __iomem *gicc_base;/* Interrupt number */ unsigned int maint_irq;/* No interrupt mask, no need to use the above field */
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 3/33 irqchip/apple-aic: Spit out ICH_MIDR_EL2 value on spurious vGIC MI
In reply to Marc ZyngierIt is all good and well to scream about spurious vGIC maintenance interrupts. It would be even better to output the reason why, which is already checked, but not printed out.
The unsuspecting kernel tinkerer thanks you.
--- drivers/irqchip/irq-apple-aic.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/irqchip/irq-apple-aic.c +5 -2
@@ -411,12 +411,15 @@ static void __exception_irq_entry aic_handle_irq(struct pt_regs *regs)if (is_kernel_in_hyp_mode () && (read_sysreg_s (SYS_ICH_HCR_EL2 ) & ICH_HCR_EL2_En) &&read_sysreg_s (SYS_ICH_MISR_EL2 ) !=0 ) { +u64 val; +generic_handle_domain_irq (aic_irqc->hw_domain,AIC_FIQ_HWIRQ (AIC_VGIC_MI ));if (unlikely ((read_sysreg_s (SYS_ICH_HCR_EL2 ) & ICH_HCR_EL2_En) && -read_sysreg_s (SYS_ICH_MISR_EL2 ))) { -pr_err_ratelimited ("vGIC IRQ fired and not handled by KVM, disabling.\n" ); + (val =read_sysreg_s (SYS_ICH_MISR_EL2 )))) { +pr_err_ratelimited ("vGIC IRQ fired and not handled by KVM (MISR=%llx), disabling.\n" , + val);sysreg_clear_set_s (SYS_ICH_HCR_EL2 , ICH_HCR_EL2_En,0 ); } }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 5/33 KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping
In reply to Marc ZyngierA long time ago, an unsuspecting architect forgot to add a trap bit for ICV_DIR_EL1 in ICH_HCR_EL2. Which was unfortunate, but what's a bit of spec between friends? Thankfully, this was fixed in a later revision, and ARM "deprecates" the lack of trapping ability.
Unfortuantely, a few (billion) CPUs went out with that defect, anything ARMv8.0 from ARM, give or take. And on these CPUs, you can't trap DIR on its own, full stop.
As the next best thing, we can trap everything in the common group, which is a tad expensive, but hey ho, that's what you get. You can otherwise recycle the HW in the neaby bin.
--- arch/arm64/include/asm/virt.h | 7 ++++++- arch/arm64/kernel/cpufeature.c | 34 ++++++++++++++++++++++++++++++++++ arch/arm64/kernel/hyp-stub.S | 5 +++++ arch/arm64/kvm/vgic/vgic-v3.c | 3 +++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 49 insertions(+), 1 deletion(-)
arch/arm64/include/asm/virt.h +6 -1
@@ -40,8 +40,13 @@*/ #define HVC_FINALISE_EL2 3 +/* +* HVC_GET_ICH_VTR_EL2 - Retrieve the ICH_VTR_EL2 value +*/ +#define HVC_GET_ICH_VTR_EL2 4 +/* Max number of HYP stub hypercalls */ -#define HVC_STUB_HCALL_NR 4 +#define HVC_STUB_HCALL_NR 5/* Error returned when an invalid stub number is passed into x0 */ #define HVC_STUB_ERR 0xbadca11
arch/arm64/kernel/cpufeature.c +34 -0
@@ -2303,6 +2303,31 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry }#endif +static bool can_trap_icv_dir_el1 (const struct arm64_cpu_capabilities *entry, +int scope) +{ +struct arm_smccc_res res = {}; + +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV3_CPUIF ); +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV5_LEGACY ); +if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ) || + !cpus_have_cap (ARM64_HAS_GICV3_CPUIF )) +return false; + +if (!is_hyp_mode_available ()) +return false; + +if (is_kernel_in_hyp_mode ()) + res.a1 =read_sysreg_s (SYS_ICH_VTR_EL2 ); +else +arm_smccc_1_1_hvc (HVC_GET_ICH_VTR_EL2 , &res); + +if (res.a0 ==HVC_STUB_ERR ) +return false; + +return res.a1 &ICH_VTR_EL2_TDS ; +} +#ifdef CONFIG_ARM64_BTI static void bti_enable (const struct arm64_cpu_capabilities *__unused) {
@@ -2814,6 +2839,15 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_gic_prio_relaxed_sync, },#endif + { + /* + * Depends on having GICv3 + */ + .desc = "ICV_DIR_EL1 trapping", + .capability = ARM64_HAS_ICH_HCR_EL2_TDS, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = can_trap_icv_dir_el1, + }, #ifdef CONFIG_ARM64_E0PD { .desc = "E0PD",
arch/arm64/kernel/hyp-stub.S +5 -0
@@ -54,6 +54,11 @@ SYM_CODE_START_LOCAL(elx_sync) 1: cmp x0, #HVC_FINALISE_EL2 b.eq __finalise_el2 + cmp x0, #HVC_GET_ICH_VTR_EL2 + b.ne 2f + mrs x1, ich_vtr_el2 + b 9f + 2: cmp x0, #HVC_SOFT_RESTART b.ne 3f mov x0, x2
arch/arm64/kvm/vgic/vgic-v3.c +3 -0
@@ -648,6 +648,9 @@ void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt, dir_trap = true; } +if (!cpus_have_cap (ARM64_HAS_ICH_HCR_EL2_TDS )) + common_trap = true; +if (group0_trap) hcr |=ICH_HCR_EL2_TALL0 ;if (group1_trap)
arch/arm64/tools/cpucaps +1 -0
@@ -40,6 +40,7 @@ HAS_GICV5_CPUIF HAS_GICV5_LEGACY HAS_GIC_PRIO_MASKING HAS_GIC_PRIO_RELAXED_SYNC +HAS_ICH_HCR_EL2_TDS HAS_HCR_NV1 HAS_HCX HAS_LDAPR
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 4/33 KVM: arm64: Turn vgic-v3 errata traps into a patched-in constant
In reply to Marc ZyngierThe trap bits are currently only set to manage CPU errata. However, we are about to make use of them for purposes beyond beating broken CPUs into submission.
For this purpose, turn these errata-driven bits into a patched-in constant that is merged with the KVM-driven value at the point of programming the ICH_HCR_EL2 register, rather than being directly stored with with the shadow value..
This allows the KVM code to distinguish between a trap being handled for the purpose of an erratum workaround, or for KVM's own need.
--- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kvm/hyp/vgic-v3-sr.c | 21 +++++--- arch/arm64/kvm/vgic/vgic-v3-nested.c | 9 ---- arch/arm64/kvm/vgic/vgic-v3.c | 81 +++++++++++++++++----------- arch/arm64/kvm/vgic/vgic.h | 16 ++++++ 5 files changed, 82 insertions(+), 46 deletions(-)
arch/arm64/kernel/image-vars.h +1 -0
@@ -91,6 +91,7 @@ KVM_NVHE_ALIAS(spectre_bhb_patch_loop_mitigation_enable);KVM_NVHE_ALIAS (spectre_bhb_patch_wa3);KVM_NVHE_ALIAS (spectre_bhb_patch_clearbhb);KVM_NVHE_ALIAS (alt_cb_patch_nops); +KVM_NVHE_ALIAS (kvm_compute_ich_hcr_trap_bits);/* Global kernel state accessed by nVHE hyp code. */ KVM_NVHE_ALIAS (kvm_vgic_global_state);
arch/arm64/kvm/hyp/vgic-v3-sr.c +14 -7
@@ -14,6 +14,8 @@#include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> +#include "../../vgic/vgic.h" +#define vtr_to_max_lr_idx (v) ((v) & 0xf)#define vtr_to_nr_pre_bits (v) ((((u32)(v) >> 26) & 7) + 1)#define vtr_to_nr_apr_regs (v) (1 << (vtr_to_nr_pre_bits(v) - 5))
@@ -196,6 +198,11 @@ static u32 __vgic_v3_read_ap1rn(int n)return val; } +static u64 compute_ich_hcr (struct vgic_v3_cpu_if *cpu_if) +{ +return cpu_if->vgic_hcr |vgic_ich_hcr_trap_bits (); +} +void __vgic_v3_save_state (struct vgic_v3_cpu_if *cpu_if) {u64 used_lrs = cpu_if->used_lrs;
@@ -218,7 +225,7 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) elrsr =read_gicreg (ICH_ELRSR_EL2 ); -write_gicreg (cpu_if->vgic_hcr & ~ICH_HCR_EL2_En,ICH_HCR_EL2 ); +write_gicreg (compute_ich_hcr (cpu_if) & ~ICH_HCR_EL2_En,ICH_HCR_EL2 );for (i =0 ; i < used_lrs; i++) {if (elrsr & (1 << i))
@@ -237,7 +244,7 @@ void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if)int i;if (used_lrs || cpu_if->its_vpe.its_vm) { -write_gicreg (cpu_if->vgic_hcr,ICH_HCR_EL2 ); +write_gicreg (compute_ich_hcr (cpu_if),ICH_HCR_EL2 );for (i =0 ; i < used_lrs; i++)__gic_v3_set_lr (cpu_if->vgic_lr[i], i);
@@ -307,14 +314,14 @@ void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if) }/* -* If we need to trap system registers, we must write -* ICH_HCR_EL2 anyway, even if no interrupts are being -* injected. Note that this also applies if we don't expect -* any system register access (no vgic at all). +* If we need to trap system registers, we must write ICH_HCR_EL2 +* anyway, even if no interrupts are being injected. Note that this +* also applies if we don't expect any system register access (no +* vgic at all). In any case, no need to provide MI configuration. */ if (static_branch_unlikely (&vgic_v3_cpuif_trap) || cpu_if->its_vpe.its_vm || !cpu_if->vgic_sre) -write_gicreg (cpu_if->vgic_hcr,ICH_HCR_EL2 ); +write_gicreg (vgic_ich_hcr_trap_bits () | ICH_HCR_EL2_En,ICH_HCR_EL2 ); }void __vgic_v3_deactivate_traps (struct vgic_v3_cpu_if *cpu_if)
arch/arm64/kvm/vgic/vgic-v3-nested.c +0 -9
@@ -301,15 +301,6 @@ static void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu,u64 val =0 ;int i; -/* -* If we're on a system with a broken vgic that requires -* trapping, propagate the trapping requirements. -* -* Ah, the smell of rotten fruits... -*/ -if (static_branch_unlikely (&vgic_v3_cpuif_trap)) - val = host_if->vgic_hcr & (ICH_HCR_EL2_TALL0 |ICH_HCR_EL2_TALL1 | -ICH_HCR_EL2_TC |ICH_HCR_EL2_TDIR ); s_cpu_if->vgic_hcr =__vcpu_sys_reg (vcpu,ICH_HCR_EL2 ) | val; s_cpu_if->vgic_vmcr =__vcpu_sys_reg (vcpu,ICH_VMCR_EL2 ); s_cpu_if->vgic_sre = host_if->vgic_sre;
arch/arm64/kvm/vgic/vgic-v3.c +51 -30
116 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.h +16 -0
@@ -432,4 +432,20 @@ static inline bool vgic_is_v3(struct kvm *kvm)int vgic_its_debug_init (struct kvm_device *dev);void vgic_its_debug_destroy (struct kvm_device *dev); +void kvm_compute_ich_hcr_trap_bits (struct alt_instr *alt, +__le32 *origptr,__le32 *updptr,int nr_inst); + +static inline u64 vgic_ich_hcr_trap_bits (void ) +{ +u64 hcr; + +/* All the traps are in the bottom 16bits */ + asm volatile(ALTERNATIVE_CB("movz %0, #0\n", + ARM64_ALWAYS_SYSTEM, + kvm_compute_ich_hcr_trap_bits) + : "=r" (hcr)); + +return hcr; +} +#endif
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 6/33 KVM: arm64: Repack struct vgic_irq fields
In reply to Marc Zyngierstruct vgic_irq has grown over the years, in a rather bad way. Repack it using bitfields so that the individual flags, and move things around a bit so that it a bit smaller.
--- arch/arm64/kvm/vgic/vgic-v4.c | 5 ++++- include/kvm/arm_vgic.h | 20 ++++++++++---------- 2 files changed, 14 insertions(+), 11 deletions(-)
arch/arm64/kvm/vgic/vgic-v4.c +4 -1
@@ -163,6 +163,7 @@ static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu)struct vgic_irq *irq =vgic_get_vcpu_irq (vcpu, i);struct irq_desc *desc;unsigned long flags; +bool pending;int ret;raw_spin_lock_irqsave (&irq->irq_lock, flags);
@@ -173,9 +174,11 @@ static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu) irq->hw = false; ret =irq_get_irqchip_state (irq->host_irq,IRQCHIP_STATE_PENDING , - &irq->pending_latch); + &pending);WARN_ON (ret); + irq->pending_latch = pending; + desc =irq_to_desc (irq->host_irq);irq_domain_deactivate_irq (irq_desc_get_irq_data (desc)); unlock:
include/kvm/arm_vgic.h +10 -10
@@ -123,6 +123,7 @@ struct irq_ops {struct vgic_irq {raw_spinlock_t irq_lock;/* Protects the content of the struct */ +u32 intid;/* Guest visible INTID */ struct rcu_head rcu;struct list_head ap_list;
@@ -137,17 +138,17 @@ struct vgic_irq {* affinity reg (v3). */ -u32 intid;/* Guest visible INTID */ -bool line_level;/* Level only */ -bool pending_latch;/* The pending latch state used to calculate -* the pending state for both level -* and edge triggered IRQs. */ -bool active; -bool pending_release;/* Used for LPIs only, unreferenced IRQ + bool pending_release:1; /* Used for LPIs only, unreferenced IRQ* pending a release */ -bool enabled; -bool hw;/* Tied to HW IRQ */ + bool pending_latch:1; /* The pending latch state used to calculate +* the pending state for both level +* and edge triggered IRQs. */ + enum vgic_irq_config config:1; /* Level or edge */ + bool line_level:1; /* Level only */ + bool enabled:1; + bool active:1; + bool hw:1; /* Tied to HW IRQ */refcount_t refcount;/* Used for LPIs */ u32 hwintid;/* HW INTID number */ unsigned int host_irq;/* linux irq corresponding to hwintid */
@@ -159,7 +160,6 @@ struct vgic_irq {u8 active_source;/* GICv2 SGIs only */ u8 priority;u8 group;/* 0 == group 0, 1 == group 1 */ -enum vgic_irq_config config;/* Level or edge */ struct irq_ops *ops;
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 7/33 KVM: arm64: Add tracking of vgic_irq being present in a LR
In reply to Marc ZyngierWe currently cannot identify whether an interrupt is queued into a LR. It wasn't needed until now, but that's about to change.
Add yet another flag to track that state.
--- arch/arm64/kvm/vgic/vgic-v2.c | 6 ++++++ arch/arm64/kvm/vgic/vgic-v3.c | 6 ++++++ include/kvm/arm_vgic.h | 1 + 3 files changed, 13 insertions(+)
arch/arm64/kvm/vgic/vgic-v2.c +6 -0
@@ -101,6 +101,8 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu)/* Handle resampling for mapped interrupts if required */ vgic_irq_handle_resampling (irq, deactivated, val &GICH_LR_PENDING_BIT ); + irq->on_lr = false; +raw_spin_unlock (&irq->irq_lock);vgic_put_irq (vcpu->kvm, irq); }
@@ -124,6 +126,8 @@ void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)u32 val = irq->intid;bool allow_pending = true; +WARN_ON (irq->on_lr); +if (irq->active) { val |=GICH_LR_ACTIVE_BIT ;if (vgic_irq_is_sgi (irq->intid))
@@ -194,6 +198,8 @@ void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)/* The GICv2 LR only holds five bits of priority. */ val |= (irq->priority >>3 ) <<GICH_LR_PRIORITY_SHIFT ; + irq->on_lr = true; + vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val; }
arch/arm64/kvm/vgic/vgic-v3.c +6 -0
@@ -97,6 +97,8 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu)/* Handle resampling for mapped interrupts if required */ vgic_irq_handle_resampling (irq, deactivated, val &ICH_LR_PENDING_BIT ); + irq->on_lr = false; +raw_spin_unlock (&irq->irq_lock);vgic_put_irq (vcpu->kvm, irq); }
@@ -111,6 +113,8 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)u64 val = irq->intid;bool allow_pending = true, is_v2_sgi; +WARN_ON (irq->on_lr); + is_v2_sgi = (vgic_irq_is_sgi (irq->intid) && model ==KVM_DEV_TYPE_ARM_VGIC_V2 );
@@ -185,6 +189,8 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) val |= (u64 )irq->priority <<ICH_LR_PRIORITY_SHIFT ; vcpu->arch.vgic_cpu.vgic_v3.vgic_lr[lr] = val; + + irq->on_lr = true; }void vgic_v3_clear_lr (struct kvm_vcpu *vcpu,int lr)
include/kvm/arm_vgic.h +1 -0
@@ -149,6 +149,7 @@ struct vgic_irq { bool enabled:1; bool active:1; bool hw:1; /* Tied to HW IRQ */ + bool on_lr:1; /* Present in a CPU LR */refcount_t refcount;/* Used for LPIs */ u32 hwintid;/* HW INTID number */ unsigned int host_irq;/* linux irq corresponding to hwintid */
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 8/33 KVM: arm64: Add LR overflow handling documentation
In reply to Marc ZyngierAdd a bit of documentation describing how we are dealing with LR overflow. This is mostly a braindump of how things are expected to work. For now anyway.
--- arch/arm64/kvm/vgic/vgic.c | 81 +++++++++++++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic.c +80 -1
@@ -825,7 +825,86 @@ static int compute_ap_list_depth(struct kvm_vcpu *vcpu,return count; } -/* Requires the VCPU's ap_list_lock to be held. */ +/* +* Dealing with LR overflow is close to black magic -- dress accordingly. +* +* We have to present an almost infinite number of interrupts through a very +* limited number of registers. Therefore crucial decisions must be made to +* ensure we feed the most relevant interrupts into the LRs, and yet have +* some facilities to let the guest interact with those that are not there. +* +* All considerations below are in the context of interrupts targeting a +* single vcpu with non-idle state (either pending, active, or both), +* colloquially called the ap_list: +* +* - Pending interrupts must have priority over active interrupts. This also +* excludes pending+active interrupts. This ensures that a guest can +* perform priority drops on any number of interrupts, and yet be +* presented the next pending one. +* +* - Deactivation of interrupts outside of the LRs must be tracked by using +* either the EOIcount-driven maintenance interrupt, and sometimes by +* trapping the DIR register. +* +* - For EOImode=0, a non-zero EOIcount means walking the ap_list past the +* point that made it into the LRs, and deactivate interrupts that would +* have made it onto the LRs if we had the space. +* +* - The MI-generation bits must be used to try and force an exit when the +* guest has done enough changes to the LRs that we want to reevaluate the +* situation: +* +* - if the total number of pending interrupts exceeds the number of +* LR, NPIE must be set in order to exit once no pending interrupts +* are present in the LRs, allowing us to populate the next batch. +* +* - if there are active interrupts outside of the LRs, then LRENPIE +* must be set so that we exit on deactivation of one of these, and +* work out which one is to be deactivated. Note that this is not +* enough to deal with EOImode=1, see below. +* +* - if the overall number of interrupts exceeds the number of LRs, +* then UIE must be set to allow refilling of the LRs once the +* majority of them has been processed. +* +* - as usual, MI triggers are only an optimisation, since we cannot +* rely on the MI being delivered in timely manner... +* +* - EOImode=1 creates some additional problems: +* +* - deactivation can happen in any order, and we cannot rely on +* EOImode=0's coupling of priority-drop and deactivation which +* imposes strict reverse Ack order. This means that DIR must be set +* if we have active interrupts outside of the LRs. +* +* - deactivation of SPIs can occur on any CPU, while the SPI is only +* present in the ap_list of the CPU that actually ack-ed it. In that +* case, EOIcount doesn't provide enough information, and we must +* resort to trapping DIR even if we don't overflow the LRs. Bonus +* point for not trapping DIR when no SPIs are pending or active in +* the whole VM. +* +* - LPIs do not suffer the same problem as SPIs on deactivation, as we +* have to essentially discard the active state, see below. +* +* - Virtual LPIs have an active state (surprise!), which gets removed on +* priority drop (EOI). However, EOIcount doesn't get bumped when the LPI +* is not present in the LR (surprise again!). Special care must therefore +* be taken to remove the active state from any activated LPI when exiting +* from the guest. This is in a way no different from what happens on the +* physical side. We still rely on the running priority to have been +* removed from the APRs, irrespective of the LPI being present in the LRs +* or not. +* +* - Virtual SGIs directly injected via GICv4.1 must not affect EOIcount, as +* they are not managed in SW and don't have a true active state. So only +* set vSGIEOICount when no SGIs are in the ap_list. +* +* - GICv2 SGIs with multiple sources are injected one source at a time, as +* if they were made pending sequentially. This may mean that we don't +* always present the HPPI if other interrupts with lower priority are +* pending in the LRs. Big deal. +*/ static void vgic_flush_lr_state (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 9/33 KVM: arm64: GICv3: Drop LPI active state when folding LRs
In reply to Marc ZyngierDespite LPIs not having an active state, *virtual* LPIs do have one, which gets cleared on EOI. So far, so good.
However, this leads to a small problem: when an active LPI is not in the LRs, that EOImode==0 and that the guest EOIs it, EOIcount doesn't get bumped up. Which means that in these condition, the LPI would stay active forever.
Clearly, we can't have that. So if we spot an active LPI, we drop that state. It's pretty pointless anyway, and only serves as a way to trip SW over.
--- arch/arm64/kvm/vgic/vgic-v3.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-v3.c +3 -1
@@ -72,7 +72,9 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu)raw_spin_lock (&irq->irq_lock); -/* Always preserve the active bit, note deactivation */ +/* Always preserve the active bit for !LPIs, note deactivation */ +if (irq->intid >=VGIC_MIN_LPI ) + val &= ~ICH_LR_ACTIVE_BIT ; deactivated = irq->active && !(val &ICH_LR_ACTIVE_BIT ); irq->active = !!(val &ICH_LR_ACTIVE_BIT );
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 10/33 KVM: arm64: GICv3: Preserve EOIcount on exit
In reply to Marc ZyngierEOIcount is how the virtual CPU interface signals that the guest is deactivating interrupts outside of the LRs when EOImode==0.
We therefore need to preserve that information so that we can find out what actually needs deactivating.
--- arch/arm64/kvm/hyp/vgic-v3-sr.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/kvm/hyp/vgic-v3-sr.c +6 -0
@@ -225,6 +225,12 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) elrsr =read_gicreg (ICH_ELRSR_EL2 ); +if (cpu_if->vgic_hcr &ICH_HCR_EL2_LRENPIE ) { +u64 val =read_gicreg (ICH_HCR_EL2 ); + cpu_if->vgic_hcr &= ~ICH_HCR_EL2_EOIcount; + cpu_if->vgic_hcr |= val & ICH_HCR_EL2_EOIcount; + } +write_gicreg (compute_ich_hcr (cpu_if) & ~ICH_HCR_EL2_En,ICH_HCR_EL2 );for (i =0 ; i < used_lrs; i++) {
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 11/33 KVM: arm64: GICv3: Decouple ICH_HCR_EL2 programming from LRs
In reply to Marc ZyngierNot programming ICH_HCR_EL2 while no LRs are populated is a bit of an issue, as we otherwise don't see any maintenance interrupt when the guest interacts with the LRs.
Decouple the two and always program the control register, even when we don't have to touch the LRs.
--- arch/arm64/kvm/hyp/vgic-v3-sr.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-)
arch/arm64/kvm/hyp/vgic-v3-sr.c +12 -14
@@ -219,20 +219,12 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) } } -if (used_lrs || cpu_if->its_vpe.its_vm) { +if (used_lrs) {int i;u32 elrsr; elrsr =read_gicreg (ICH_ELRSR_EL2 ); -if (cpu_if->vgic_hcr &ICH_HCR_EL2_LRENPIE ) { -u64 val =read_gicreg (ICH_HCR_EL2 ); - cpu_if->vgic_hcr &= ~ICH_HCR_EL2_EOIcount; - cpu_if->vgic_hcr |= val & ICH_HCR_EL2_EOIcount; - } - -write_gicreg (compute_ich_hcr (cpu_if) & ~ICH_HCR_EL2_En,ICH_HCR_EL2 ); -for (i =0 ; i < used_lrs; i++) {if (elrsr & (1 << i)) cpu_if->vgic_lr[i] &= ~ICH_LR_STATE ;
@@ -242,6 +234,14 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if)__gic_v3_set_lr (0 , i); } } + +if (cpu_if->vgic_hcr &ICH_HCR_EL2_LRENPIE ) { +u64 val =read_gicreg (ICH_HCR_EL2 ); + cpu_if->vgic_hcr &= ~ICH_HCR_EL2_EOIcount; + cpu_if->vgic_hcr |= val & ICH_HCR_EL2_EOIcount; + } + +write_gicreg (compute_ich_hcr (cpu_if) & ~ICH_HCR_EL2_En,ICH_HCR_EL2 ); }void __vgic_v3_restore_state (struct vgic_v3_cpu_if *cpu_if)
@@ -249,12 +249,10 @@ void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if)u64 used_lrs = cpu_if->used_lrs;int i; -if (used_lrs || cpu_if->its_vpe.its_vm) { -write_gicreg (compute_ich_hcr (cpu_if),ICH_HCR_EL2 ); +write_gicreg (compute_ich_hcr (cpu_if),ICH_HCR_EL2 ); -for (i =0 ; i < used_lrs; i++) -__gic_v3_set_lr (cpu_if->vgic_lr[i], i); - } +for (i =0 ; i < used_lrs; i++) +__gic_v3_set_lr (cpu_if->vgic_lr[i], i);/* * Ensure that writes to the LRs, and on non-VHE systems ensure that
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 12/33 KVM: arm64: GICv3: Extract LR folding primitive
In reply to Marc ZyngierAs we are going to need to handle deactivation for interrupts that are not in the LRs, split vgic_v3_fold_lr_state() into a helper that deals with a single interrupt, and the function that loops over the used LRs.
--- arch/arm64/kvm/vgic/vgic-v3.c | 88 +++++++++++++++++------------------ 1 file changed, 43 insertions(+), 45 deletions(-)
arch/arm64/kvm/vgic/vgic-v3.c +43 -45
@@ -33,78 +33,76 @@ static bool lr_signals_eoi_mi(u64 lr_val) !(lr_val &ICH_LR_HW ); } -void vgic_v3_fold_lr_state (struct kvm_vcpu *vcpu) +static void vgic_v3_fold_lr (struct kvm_vcpu *vcpu,u64 val) { -struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; -struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; -u32 model = vcpu->kvm->arch.vgic.vgic_model; -int lr; - -DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ()); - - cpuif->vgic_hcr &= ~ICH_HCR_EL2_UIE ; - -for (lr =0 ; lr < cpuif->used_lrs; lr++) { -u64 val = cpuif->vgic_lr[lr]; -u32 intid, cpuid; -struct vgic_irq *irq; -bool is_v2_sgi = false; -bool deactivated; - - cpuid = val &GICH_LR_PHYSID_CPUID ; - cpuid >>=GICH_LR_PHYSID_CPUID_SHIFT ; - -if (model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) { - intid = val &ICH_LR_VIRTUAL_ID_MASK ; - }else { - intid = val &GICH_LR_VIRTUALID ; - is_v2_sgi =vgic_irq_is_sgi (intid); - } +struct vgic_irq *irq; +bool is_v2_sgi = false; +bool deactivated; +u32 intid; -/* Notify fds when the guest EOI'ed a level-triggered IRQ */ -if (lr_signals_eoi_mi (val) &&vgic_valid_spi (vcpu->kvm, intid)) -kvm_notify_acked_irq (vcpu->kvm,0 , - intid -VGIC_NR_PRIVATE_IRQS ); +if (vcpu->kvm->arch.vgic.vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ) { + intid = val &ICH_LR_VIRTUAL_ID_MASK ; + }else { + intid = val &GICH_LR_VIRTUALID ; + is_v2_sgi =vgic_irq_is_sgi (intid); + } - irq =vgic_get_vcpu_irq (vcpu, intid); -if (!irq)/* An LPI could have been unmapped. */ -continue ; + irq =vgic_get_vcpu_irq (vcpu, intid); +if (!irq)/* An LPI could have been unmapped. */ +return ; -raw_spin_lock (&irq->irq_lock); +/* Notify fds when the guest EOI'ed a level-triggered IRQ */ +if (lr_signals_eoi_mi (val) &&vgic_valid_spi (vcpu->kvm, intid)) +kvm_notify_acked_irq (vcpu->kvm,0 , + intid -VGIC_NR_PRIVATE_IRQS ); + scoped_guard(raw_spinlock, &irq->irq_lock) {/* Always preserve the active bit for !LPIs, note deactivation */ if (irq->intid >=VGIC_MIN_LPI ) val &= ~ICH_LR_ACTIVE_BIT ; deactivated = irq->active && !(val &ICH_LR_ACTIVE_BIT ); irq->active = !!(val &ICH_LR_ACTIVE_BIT ); -if (irq->active && is_v2_sgi) - irq->active_source = cpuid; -/* Edge is the only case where we preserve the pending bit */ if (irq->config ==VGIC_CONFIG_EDGE && - (val &ICH_LR_PENDING_BIT )) { + (val &ICH_LR_PENDING_BIT )) irq->pending_latch = true; -if (is_v2_sgi) - irq->source |= (1 << cpuid); - } -/* * Clear soft pending state when level irqs have been acked. */ if (irq->config ==VGIC_CONFIG_LEVEL && !(val &ICH_LR_STATE )) irq->pending_latch = false; +if (is_v2_sgi) { +u8 cpuid =FIELD_GET (GICH_LR_PHYSID_CPUID , val); + +if (irq->active) + irq->active_source = cpuid; + +if (val &ICH_LR_PENDING_BIT ) + irq->source |=BIT (cpuid); + } +/* Handle resampling for mapped interrupts if required */ vgic_irq_handle_resampling (irq, deactivated, val &ICH_LR_PENDING_BIT ); irq->on_lr = false; - -raw_spin_unlock (&irq->irq_lock); -vgic_put_irq (vcpu->kvm, irq); } +vgic_put_irq (vcpu->kvm, irq); +} + +void vgic_v3_fold_lr_state (struct kvm_vcpu *vcpu) +{ +struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; +struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; + +DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ()); + +for (int lr =0 ; lr < cpuif->used_lrs; lr++) +vgic_v3_fold_lr (vcpu, cpuif->vgic_lr[lr]); + cpuif->used_lrs =0 ; }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 13/33 KVM: arm64: GICv3: Extract LR computing primitive
In reply to Marc ZyngierSplit vgic_v3_populate_lr() into two, so that we have another primitive that computes the LR from a vgic_irq, but doesn't update anything in the shadow structure.
--- arch/arm64/kvm/vgic/vgic-v3.c | 49 +++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 17 deletions(-)
arch/arm64/kvm/vgic/vgic-v3.c +32 -17
@@ -107,7 +107,7 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) }/* Requires the irq to be locked already */ -void vgic_v3_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr) +static u64 vgic_v3_compute_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq) {u32 model = vcpu->kvm->arch.vgic.vgic_model;u64 val = irq->intid;
@@ -154,6 +154,35 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)if (allow_pending &&irq_is_pending (irq)) { val |=ICH_LR_PENDING_BIT ; +if (is_v2_sgi) { +u32 src =ffs (irq->source); + +if (WARN_RATELIMIT (!src,"No SGI source for INTID %d\n" , + irq->intid)) +return 0 ; + + val |= (src -1 ) <<GICH_LR_PHYSID_CPUID_SHIFT ; +if (irq->source & ~BIT (src -1 )) + val |=ICH_LR_EOI ; + } + } + +if (irq->group) + val |=ICH_LR_GROUP ; + + val |= (u64 )irq->priority <<ICH_LR_PRIORITY_SHIFT ; + +return val; +} + +void vgic_v3_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr) +{ +u32 model = vcpu->kvm->arch.vgic.vgic_model; +u64 val =vgic_v3_compute_lr (vcpu, irq); + + vcpu->arch.vgic_cpu.vgic_v3.vgic_lr[lr] = val; + +if (val &ICH_LR_PENDING_BIT ) {if (irq->config ==VGIC_CONFIG_EDGE ) irq->pending_latch = false;
@@ -161,16 +190,9 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) model == KVM_DEV_TYPE_ARM_VGIC_V2) {u32 src =ffs (irq->source); -if (WARN_RATELIMIT (!src,"No SGI source for INTID %d\n" , - irq->intid)) -return ; - - val |= (src -1 ) <<GICH_LR_PHYSID_CPUID_SHIFT ; - irq->source &= ~(1 << (src -1 )); -if (irq->source) { + irq->source &= ~BIT (src -1 ); +if (irq->source) irq->pending_latch = true; - val |=ICH_LR_EOI ; - } } }
@@ -183,13 +205,6 @@ void vgic_v3_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)if (vgic_irq_is_mapped_level (irq) && (val &ICH_LR_PENDING_BIT )) irq->line_level = false; -if (irq->group) - val |=ICH_LR_GROUP ; - - val |= (u64 )irq->priority <<ICH_LR_PRIORITY_SHIFT ; - - vcpu->arch.vgic_cpu.vgic_v3.vgic_lr[lr] = val; - irq->on_lr = true; }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 14/33 KVM: arm64: GICv2: Preserve EOIcount on exit
In reply to Marc ZyngierEOIcount is how the virtual CPU interface signals that the guest is deactivating interrupts outside of the LRs when EOImode==0.
We therefore need to preserve that information so that we can find out what actually needs deactivating, just like we already do on GICv3.
--- arch/arm64/kvm/vgic/vgic-v2.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/kvm/vgic/vgic-v2.c +6 -0
@@ -437,6 +437,12 @@ void vgic_v2_save_state(struct kvm_vcpu *vcpu)return ;if (used_lrs) { +if (vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr &GICH_HCR_LRENPIE ) { +u32 val =readl_relaxed (base +GICH_HCR ); + + vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr &= ~GICH_HCR_EOICOUNT ; + vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr |= val &GICH_HCR_EOICOUNT ; + }save_lrs (vcpu, base);writel_relaxed (0 , base +GICH_HCR ); }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 15/33 KVM: arm64: GICv2: Decouple GICH_HCR programming from LRs being loaded
In reply to Marc ZyngierNot programming GICH_HCR while no LRs are populated is a bit of an issue, as we otherwise don't see any maintenance interrupt when the guest interacts with the LRs.
Decouple the two and always program the control register, even when we don't have to touch the LRs.
This is very similar to what we are already doing for GICv3.
--- arch/arm64/kvm/vgic/vgic-v2.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-)
arch/arm64/kvm/vgic/vgic-v2.c +14 -14
@@ -430,22 +430,25 @@ static void save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)void vgic_v2_save_state (struct kvm_vcpu *vcpu) { +struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;void __iomem *base = kvm_vgic_global_state.vctrl_base;u64 used_lrs = vcpu->arch.vgic_cpu.vgic_v2.used_lrs;if (!base)return ; -if (used_lrs) { -if (vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr &GICH_HCR_LRENPIE ) { -u32 val =readl_relaxed (base +GICH_HCR ); - vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr &= ~GICH_HCR_EOICOUNT ; - vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr |= val &GICH_HCR_EOICOUNT ; - } +if (used_lrs)save_lrs (vcpu, base); -writel_relaxed (0 , base +GICH_HCR ); + +if (cpu_if->vgic_hcr &GICH_HCR_LRENPIE ) { +u32 val =readl_relaxed (base +GICH_HCR ); + + cpu_if->vgic_hcr &= ~GICH_HCR_EOICOUNT ; + cpu_if->vgic_hcr |= val &GICH_HCR_EOICOUNT ; } + +writel_relaxed (0 , base +GICH_HCR ); }void vgic_v2_restore_state (struct kvm_vcpu *vcpu)
@@ -458,13 +461,10 @@ void vgic_v2_restore_state(struct kvm_vcpu *vcpu)if (!base)return ; -if (used_lrs) { -writel_relaxed (cpu_if->vgic_hcr, base +GICH_HCR ); -for (i =0 ; i < used_lrs; i++) { -writel_relaxed (cpu_if->vgic_lr[i], - base +GICH_LR0 + (i *4 )); - } - } +writel_relaxed (cpu_if->vgic_hcr, base +GICH_HCR ); + +for (i =0 ; i < used_lrs; i++) +writel_relaxed (cpu_if->vgic_lr[i], base +GICH_LR0 + (i *4 )); }void vgic_v2_load (struct kvm_vcpu *vcpu)
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 16/33 KVM: arm64: GICv2: Extract LR folding primitive
In reply to Marc ZyngierAs we are going to need to handle deactivation for interrupts that are not in the LRs, split vgic_v2_fold_lr_state() into a helper that deals with a single interrupt, and the function that loops over the used LRs.
--- arch/arm64/kvm/vgic/vgic-v2.c | 67 +++++++++++++++++------------------ 1 file changed, 32 insertions(+), 35 deletions(-)
arch/arm64/kvm/vgic/vgic-v2.c +32 -35
@@ -39,43 +39,23 @@ static bool lr_signals_eoi_mi(u32 lr_val) !(lr_val &GICH_LR_HW ); } -/* -* transfer the content of the LRs back into the corresponding ap_list: -* - active bit is transferred as is -* - pending bit is -* - transferred as is in case of edge sensitive IRQs -* - set to the line-level (resample time) for level sensitive IRQs -*/ -void vgic_v2_fold_lr_state (struct kvm_vcpu *vcpu) +static void vgic_v2_fold_lr (struct kvm_vcpu *vcpu,u32 val) { -struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; -struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2; -int lr; - -DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ()); - - cpuif->vgic_hcr &= ~GICH_HCR_UIE ; +u32 cpuid, intid = val &GICH_LR_VIRTUALID ; +struct vgic_irq *irq; +bool deactivated; -for (lr =0 ; lr < vgic_cpu->vgic_v2.used_lrs; lr++) { -u32 val = cpuif->vgic_lr[lr]; -u32 cpuid, intid = val &GICH_LR_VIRTUALID ; -struct vgic_irq *irq; -bool deactivated; +/* Extract the source vCPU id from the LR */ + cpuid =FIELD_GET (GICH_LR_PHYSID_CPUID , val) &7 ; -/* Extract the source vCPU id from the LR */ - cpuid = val &GICH_LR_PHYSID_CPUID ; - cpuid >>=GICH_LR_PHYSID_CPUID_SHIFT ; - cpuid &=7 ; +/* Notify fds when the guest EOI'ed a level-triggered SPI */ +if (lr_signals_eoi_mi (val) &&vgic_valid_spi (vcpu->kvm, intid)) +kvm_notify_acked_irq (vcpu->kvm,0 , + intid -VGIC_NR_PRIVATE_IRQS ); -/* Notify fds when the guest EOI'ed a level-triggered SPI */ -if (lr_signals_eoi_mi (val) &&vgic_valid_spi (vcpu->kvm, intid)) -kvm_notify_acked_irq (vcpu->kvm,0 , - intid -VGIC_NR_PRIVATE_IRQS ); - - irq =vgic_get_vcpu_irq (vcpu, intid); - -raw_spin_lock (&irq->irq_lock); + irq =vgic_get_vcpu_irq (vcpu, intid); + scoped_guard(raw_spinlock, &irq->irq_lock) {/* Always preserve the active bit, note deactivation */ deactivated = irq->active && !(val &GICH_LR_ACTIVE_BIT ); irq->active = !!(val &GICH_LR_ACTIVE_BIT );
@@ -102,11 +82,28 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu)vgic_irq_handle_resampling (irq, deactivated, val &GICH_LR_PENDING_BIT ); irq->on_lr = false; - -raw_spin_unlock (&irq->irq_lock); -vgic_put_irq (vcpu->kvm, irq); } +vgic_put_irq (vcpu->kvm, irq); +} + +/* +* transfer the content of the LRs back into the corresponding ap_list: +* - active bit is transferred as is +* - pending bit is +* - transferred as is in case of edge sensitive IRQs +* - set to the line-level (resample time) for level sensitive IRQs +*/ +void vgic_v2_fold_lr_state (struct kvm_vcpu *vcpu) +{ +struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; +struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2; + +DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ()); + +for (int lr =0 ; lr < vgic_cpu->vgic_v2.used_lrs; lr++) +vgic_v2_fold_lr (vcpu, cpuif->vgic_lr[lr]); + cpuif->used_lrs =0 ; }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 17/33 KVM: arm64: GICv2: Extract LR computing primitive
In reply to Marc ZyngierSplit vgic_v2_populate_lr() into two helpers, so that we have another primitive that computes the LR from a vgic_irq, but doesn't update anything in the shadow structure.
--- arch/arm64/kvm/vgic/vgic-v2.c | 61 ++++++++++++++++++++++------------- 1 file changed, 39 insertions(+), 22 deletions(-)
arch/arm64/kvm/vgic/vgic-v2.c +39 -22
@@ -107,18 +107,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) cpuif->used_lrs =0 ; } -/* -* Populates the particular LR with the state of a given IRQ: -* - for an edge sensitive IRQ the pending state is cleared in struct vgic_irq -* - for a level sensitive IRQ the pending state value is unchanged; -* it is dictated directly by the input level -* -* If @irq describes an SGI with multiple sources, we choose the -* lowest-numbered source VCPU and clear that bit in the source bitmap. -* -* The irq_lock must be held by the caller. -*/ -void vgic_v2_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr) +static u32 vgic_v2_compute_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq) {u32 val = irq->intid;bool allow_pending = true;
@@ -164,22 +153,52 @@ void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr)if (allow_pending &&irq_is_pending (irq)) { val |=GICH_LR_PENDING_BIT ; -if (irq->config ==VGIC_CONFIG_EDGE ) - irq->pending_latch = false; -if (vgic_irq_is_sgi (irq->intid)) {u32 src =ffs (irq->source);if (WARN_RATELIMIT (!src,"No SGI source for INTID %d\n" , irq->intid)) -return ; +return 0 ; val |= (src -1 ) <<GICH_LR_PHYSID_CPUID_SHIFT ; - irq->source &= ~(1 << (src -1 )); -if (irq->source) { - irq->pending_latch = true; +if (irq->source & ~BIT (src -1 )) val |=GICH_LR_EOI ; - } + } + } + +/* The GICv2 LR only holds five bits of priority. */ + val |= (irq->priority >>3 ) <<GICH_LR_PRIORITY_SHIFT ; + +return val; +} + +/* +* Populates the particular LR with the state of a given IRQ: +* - for an edge sensitive IRQ the pending state is cleared in struct vgic_irq +* - for a level sensitive IRQ the pending state value is unchanged; +* it is dictated directly by the input level +* +* If @irq describes an SGI with multiple sources, we choose the +* lowest-numbered source VCPU and clear that bit in the source bitmap. +* +* The irq_lock must be held by the caller. +*/ +void vgic_v2_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr) +{ +u32 val =vgic_v2_compute_lr (vcpu, irq); + + vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val; + +if (val &GICH_LR_PENDING_BIT ) { +if (irq->config ==VGIC_CONFIG_EDGE ) + irq->pending_latch = false; + +if (vgic_irq_is_sgi (irq->intid)) { +u32 src =ffs (irq->source); + + irq->source &= ~BIT (src -1 ); +if (irq->source) + irq->pending_latch = true; } }
@@ -196,8 +215,6 @@ void vgic_v2_populate_lr(struct kvm_vcpu *vcpu, struct vgic_irq *irq, int lr) val |= (irq->priority >>3 ) <<GICH_LR_PRIORITY_SHIFT ; irq->on_lr = true; - - vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val; }void vgic_v2_clear_lr (struct kvm_vcpu *vcpu,int lr)
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 18/33 KVM: arm64: Compute vgic state irrespective of the number of interrupts
In reply to Marc ZyngierAs we are going to rely on the [G]ICH_HCR{,_EL2} register to be programmed with MI information at all times, slightly de-optimise the flush/sync code to always be called. This is rather lightweight when no interrupts are in flight.
--- arch/arm64/kvm/vgic/vgic.c | 35 ++--------------------------------- 1 file changed, 2 insertions(+), 33 deletions(-)
arch/arm64/kvm/vgic/vgic.c +2 -33
@@ -985,8 +985,6 @@ static inline void vgic_save_state(struct kvm_vcpu *vcpu)/* Sync back the hardware VGIC state into our emulation after a guest's run. */ void kvm_vgic_sync_hwstate (struct kvm_vcpu *vcpu) { -int used_lrs; -/* If nesting, emulate the HW effect from L0 to L1 */ if (vgic_state_is_nested (vcpu)) {vgic_v3_sync_nested (vcpu);
@@ -996,20 +994,10 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)if (vcpu_has_nv (vcpu))vgic_v3_nested_update_mi (vcpu); -/* An empty ap_list_head implies used_lrs == 0 */ -if (list_empty (&vcpu->arch.vgic_cpu.ap_list_head)) -return ; -if (can_access_vgic_from_kernel ())vgic_save_state (vcpu); -if (!static_branch_unlikely (&kvm_vgic_global_state.gicv3_cpuif)) - used_lrs = vcpu->arch.vgic_cpu.vgic_v2.used_lrs; -else - used_lrs = vcpu->arch.vgic_cpu.vgic_v3.used_lrs; - -if (used_lrs) -vgic_fold_lr_state (vcpu); +vgic_fold_lr_state (vcpu);vgic_prune_ap_list (vcpu); }
@@ -1053,29 +1041,10 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu)if (vcpu_has_nv (vcpu))vgic_v3_nested_update_mi (vcpu); -/* -* If there are no virtual interrupts active or pending for this -* VCPU, then there is no work to do and we can bail out without -* taking any lock. There is a potential race with someone injecting -* interrupts to the VCPU, but it is a benign race as the VCPU will -* either observe the new interrupt before or after doing this check, -* and introducing additional synchronization mechanism doesn't change -* this. -* -* Note that we still need to go through the whole thing if anything -* can be directly injected (GICv4). -*/ -if (list_empty (&vcpu->arch.vgic_cpu.ap_list_head) && - !vgic_supports_direct_irqs (vcpu->kvm)) -return ; -DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ()); -if (!list_empty (&vcpu->arch.vgic_cpu.ap_list_head)) { -raw_spin_lock (&vcpu->arch.vgic_cpu.ap_list_lock); + scoped_guard(raw_spinlock, &vcpu->arch.vgic_cpu.ap_list_lock)vgic_flush_lr_state (vcpu); -raw_spin_unlock (&vcpu->arch.vgic_cpu.ap_list_lock); - }if (can_access_vgic_from_kernel ())vgic_restore_state (vcpu);
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 19/33 KVM: arm64: Eagerly save VMCR on exit
In reply to Marc ZyngierWe currently save/restore the VMCR register in a pretty lazy way (on load/put, consistently with what we do with the APRs).
However, we are going to need the group-enable bits that are backed by VMCR on each entry (so that we can avoid injecting interrupts for disabled groups).
Move the synchronisation from put to sync, which results in some minor churn in the nVHE hypercalls to simplify things.
--- arch/arm64/include/asm/kvm_asm.h | 2 +- arch/arm64/include/asm/kvm_hyp.h | 2 +- arch/arm64/kvm/arm.c | 3 +-- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 7 ++++--- arch/arm64/kvm/hyp/vgic-v3-sr.c | 15 +++------------ arch/arm64/kvm/vgic/vgic-v2.c | 2 +- arch/arm64/kvm/vgic/vgic-v3-nested.c | 2 +- arch/arm64/kvm/vgic/vgic-v3.c | 2 +- 8 files changed, 13 insertions(+), 22 deletions(-)
arch/arm64/include/asm/kvm_asm.h +1 -1
@@ -79,7 +79,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range, __KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context, __KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff, - __KVM_HOST_SMCCC_FUNC___vgic_v3_save_vmcr_aprs, + __KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs, __KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs, __KVM_HOST_SMCCC_FUNC___pkvm_reserve_vm, __KVM_HOST_SMCCC_FUNC___pkvm_unreserve_vm,
arch/arm64/include/asm/kvm_hyp.h +1 -1
@@ -82,7 +82,7 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_restore_state (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_activate_traps (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_deactivate_traps (struct vgic_v3_cpu_if *cpu_if); -void __vgic_v3_save_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_save_aprs (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_restore_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if);int __vgic_v3_perform_cpuif_access (struct kvm_vcpu *vcpu);
arch/arm64/kvm/arm.c +1 -2
@@ -659,8 +659,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)void kvm_arch_vcpu_put (struct kvm_vcpu *vcpu) {if (is_protected_kvm_enabled ()) { -kvm_call_hyp (__vgic_v3_save_vmcr_aprs, - &vcpu->arch.vgic_cpu.vgic_v3); +kvm_call_hyp (__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3);kvm_call_hyp_nvhe (__pkvm_vcpu_put); }
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4 -3
@@ -157,6 +157,7 @@ static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags; host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr; + host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr;for (i =0 ; i < hyp_cpu_if->used_lrs; ++i) host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i]; }
@@ -464,11 +465,11 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)__vgic_v3_init_lrs (); } -static void handle___vgic_v3_save_vmcr_aprs (struct kvm_cpu_context *host_ctxt) +static void handle___vgic_v3_save_aprs (struct kvm_cpu_context *host_ctxt) { DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1); -__vgic_v3_save_vmcr_aprs (kern_hyp_va (cpu_if)); +__vgic_v3_save_aprs (kern_hyp_va (cpu_if)); }static void handle___vgic_v3_restore_vmcr_aprs (struct kvm_cpu_context *host_ctxt)
@@ -616,7 +617,7 @@ static const hcall_t host_hcall[] = {HANDLE_FUNC (__kvm_tlb_flush_vmid_range),HANDLE_FUNC (__kvm_flush_cpu_context),HANDLE_FUNC (__kvm_timer_set_cntvoff), -HANDLE_FUNC (__vgic_v3_save_vmcr_aprs), +HANDLE_FUNC (__vgic_v3_save_aprs),HANDLE_FUNC (__vgic_v3_restore_vmcr_aprs),HANDLE_FUNC (__pkvm_reserve_vm), HANDLE_FUNC(__pkvm_unreserve_vm),
arch/arm64/kvm/hyp/vgic-v3-sr.c +3 -12
@@ -235,6 +235,8 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) } } + cpu_if->vgic_vmcr =read_gicreg (ICH_VMCR_EL2 ); +if (cpu_if->vgic_hcr &ICH_HCR_EL2_LRENPIE ) {u64 val =read_gicreg (ICH_HCR_EL2 ); cpu_if->vgic_hcr &= ~ICH_HCR_EL2_EOIcount;
@@ -332,10 +334,6 @@ void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if) {u64 val; -if (!cpu_if->vgic_sre) { - cpu_if->vgic_vmcr =read_gicreg (ICH_VMCR_EL2 ); - } -/* Only restore SRE if the host implements the GICv2 interface */ if (static_branch_unlikely (&vgic_v3_has_v2_compat)) { val =read_gicreg (ICC_SRE_EL2 );
@@ -357,7 +355,7 @@ void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if)write_gicreg (0 ,ICH_HCR_EL2 ); } -static void __vgic_v3_save_aprs (struct vgic_v3_cpu_if *cpu_if) +void __vgic_v3_save_aprs (struct vgic_v3_cpu_if *cpu_if) {u64 val;u32 nr_pre_bits;
@@ -518,13 +516,6 @@ static void __vgic_v3_write_vmcr(u32 vmcr)write_gicreg (vmcr,ICH_VMCR_EL2 ); } -void __vgic_v3_save_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if) -{ -__vgic_v3_save_aprs (cpu_if); -if (cpu_if->vgic_sre) - cpu_if->vgic_vmcr =__vgic_v3_read_vmcr (); -} -void __vgic_v3_restore_vmcr_aprs (struct vgic_v3_cpu_if *cpu_if) {__vgic_v3_compat_mode_enable ();
arch/arm64/kvm/vgic/vgic-v2.c +1 -1
@@ -451,6 +451,7 @@ void vgic_v2_save_state(struct kvm_vcpu *vcpu)if (!base)return ; + cpu_if->vgic_vmcr =readl_relaxed (kvm_vgic_global_state.vctrl_base +GICH_VMCR );if (used_lrs)save_lrs (vcpu, base);
@@ -495,6 +496,5 @@ void vgic_v2_put(struct kvm_vcpu *vcpu) {struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; - cpu_if->vgic_vmcr =readl_relaxed (kvm_vgic_global_state.vctrl_base +GICH_VMCR ); cpu_if->vgic_apr =readl_relaxed (kvm_vgic_global_state.vctrl_base +GICH_APR ); }
arch/arm64/kvm/vgic/vgic-v3-nested.c +1 -1
@@ -341,7 +341,7 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu)u64 val;int i; -__vgic_v3_save_vmcr_aprs (s_cpu_if); +__vgic_v3_save_aprs (s_cpu_if);__vgic_v3_deactivate_traps (s_cpu_if);__vgic_v3_save_state (s_cpu_if);
arch/arm64/kvm/vgic/vgic-v3.c +1 -1
@@ -815,7 +815,7 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) }if (likely (!is_protected_kvm_enabled ())) -kvm_call_hyp (__vgic_v3_save_vmcr_aprs, cpu_if); +kvm_call_hyp (__vgic_v3_save_aprs, cpu_if);WARN_ON (vgic_v4_put (vcpu));if (has_vhe ())
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 20/33 KVM: arm64: Revamp vgic maintenance interrupt configuration
In reply to Marc ZyngierWe currently don't use the maintenance interrupt very much, apart from EOI on level interrupts, and for LR underflow in limited cases.
However, as we are moving toward a setup where active interrupts can live outside of the LRs, we need to use the MIs in a more diverse set of cases.
Add a new helper that produces a digest of the ap_list, and use that summary to set the various control bits as required.
This slightly changes the way v2 SGIs are handled, as they used to count for more than one interrupt, but not anymore.
--- arch/arm64/kvm/vgic/vgic-v2.c | 12 ++++- arch/arm64/kvm/vgic/vgic-v3.c | 15 +++++- arch/arm64/kvm/vgic/vgic.c | 91 ++++++++++++----------------------- arch/arm64/kvm/vgic/vgic.h | 19 +++++++- 4 files changed, 71 insertions(+), 66 deletions(-)
arch/arm64/kvm/vgic/vgic-v2.c +10 -2
@@ -26,11 +26,19 @@ void vgic_v2_init_lrs(void)vgic_v2_write_lr (i,0 ); } -void vgic_v2_set_underflow (struct kvm_vcpu *vcpu) +void vgic_v2_configure_hcr (struct kvm_vcpu *vcpu, +struct ap_list_summary *als) {struct vgic_v2_cpu_if *cpuif = &vcpu->arch.vgic_cpu.vgic_v2; - cpuif->vgic_hcr |=GICH_HCR_UIE ; + cpuif->vgic_hcr =GICH_HCR_EN ; + +if (irqs_pending_outside_lrs (als)) + cpuif->vgic_hcr |=GICH_HCR_NPIE ; +if (irqs_active_outside_lrs (als)) + cpuif->vgic_hcr |=GICH_HCR_LRENPIE ; +if (irqs_outside_lrs (als)) + cpuif->vgic_hcr |=GICH_HCR_UIE ; }static bool lr_signals_eoi_mi (u32 lr_val)
arch/arm64/kvm/vgic/vgic-v3.c +13 -2
@@ -20,11 +20,22 @@ static bool common_trap;static bool dir_trap;static bool gicv4_enable; -void vgic_v3_set_underflow (struct kvm_vcpu *vcpu) +void vgic_v3_configure_hcr (struct kvm_vcpu *vcpu, +struct ap_list_summary *als) {struct vgic_v3_cpu_if *cpuif = &vcpu->arch.vgic_cpu.vgic_v3; - cpuif->vgic_hcr |=ICH_HCR_EL2_UIE ; + cpuif->vgic_hcr = ICH_HCR_EL2_En; + +if (irqs_pending_outside_lrs (als)) + cpuif->vgic_hcr |=ICH_HCR_EL2_NPIE ; +if (irqs_active_outside_lrs (als)) + cpuif->vgic_hcr |=ICH_HCR_EL2_LRENPIE ; +if (irqs_outside_lrs (als)) + cpuif->vgic_hcr |=ICH_HCR_EL2_UIE ; + +if (!als->nr_sgi) + cpuif->vgic_hcr |= ICH_HCR_EL2_vSGIEOICount; }static bool lr_signals_eoi_mi (u64 lr_val)
arch/arm64/kvm/vgic/vgic.c +31 -60
129 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.h +17 -2
@@ -220,6 +220,21 @@ struct its_ite {u32 event_id; }; +struct ap_list_summary { +unsigned int nr_pend;/* purely pending, not active */ +unsigned int nr_act;/* active, or active+pending */ +unsigned int nr_sgi;/* any SGI */ +}; + +#define irqs_outside_lrs (s) \ + (((s)->nr_pend + (s)->nr_act) > kvm_vgic_global_state.nr_lr) + +#define irqs_pending_outside_lrs (s) \ + ((s)->nr_pend > kvm_vgic_global_state.nr_lr) + +#define irqs_active_outside_lrs (s) \ + ((s)->nr_act && irqs_outside_lrs(s)) +int vgic_v3_parse_attr (struct kvm_device *dev,struct kvm_device_attr *attr,struct vgic_reg_attr *reg_attr); int vgic_v2_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr,
@@ -246,7 +261,7 @@ int vgic_check_iorange(struct kvm *kvm, phys_addr_t ioaddr,void vgic_v2_fold_lr_state (struct kvm_vcpu *vcpu);void vgic_v2_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr);void vgic_v2_clear_lr (struct kvm_vcpu *vcpu,int lr); -void vgic_v2_set_underflow (struct kvm_vcpu *vcpu); +void vgic_v2_configure_hcr (struct kvm_vcpu *vcpu,struct ap_list_summary *als);int vgic_v2_has_attr_regs (struct kvm_device *dev,struct kvm_device_attr *attr);int vgic_v2_dist_uaccess (struct kvm_vcpu *vcpu,bool is_write,int offset,u32 *val);
@@ -286,7 +301,7 @@ static inline void vgic_get_irq_ref(struct vgic_irq *irq)void vgic_v3_fold_lr_state (struct kvm_vcpu *vcpu);void vgic_v3_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr);void vgic_v3_clear_lr (struct kvm_vcpu *vcpu,int lr); -void vgic_v3_set_underflow (struct kvm_vcpu *vcpu); +void vgic_v3_configure_hcr (struct kvm_vcpu *vcpu,struct ap_list_summary *als);void vgic_v3_set_vmcr (struct kvm_vcpu *vcpu,struct vgic_vmcr *vmcr);void vgic_v3_get_vmcr (struct kvm_vcpu *vcpu,struct vgic_vmcr *vmcr);void vgic_v3_enable (struct kvm_vcpu *vcpu);
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 21/33 KVM: arm64: Make vgic_target_oracle() globally available
In reply to Marc ZyngierMake the internal crystal ball global, so that implementation-specific code can use it.
--- arch/arm64/kvm/vgic/vgic.c | 2 +- arch/arm64/kvm/vgic/vgic.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic.c +1 -1
@@ -237,7 +237,7 @@ void vgic_irq_set_phys_active(struct vgic_irq *irq, bool active)* * Requires the IRQ lock to be held. */ -static struct kvm_vcpu *vgic_target_oracle (struct vgic_irq *irq) +struct kvm_vcpu *vgic_target_oracle (struct vgic_irq *irq) {lockdep_assert_held (&irq->irq_lock);
arch/arm64/kvm/vgic/vgic.h +1 -0
@@ -245,6 +245,7 @@ vgic_get_mmio_region(struct kvm_vcpu *vcpu, struct vgic_io_device *iodev,struct vgic_irq *vgic_get_irq (struct kvm *kvm,u32 intid);struct vgic_irq *vgic_get_vcpu_irq (struct kvm_vcpu *vcpu,u32 intid);void vgic_put_irq (struct kvm *kvm,struct vgic_irq *irq); +struct kvm_vcpu *vgic_target_oracle (struct vgic_irq *irq);bool vgic_get_phys_line_level (struct vgic_irq *irq);void vgic_irq_set_phys_pending (struct vgic_irq *irq,bool pending);void vgic_irq_set_phys_active (struct vgic_irq *irq,bool active);
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 22/33 KVM: arm64: Invert ap_list sorting to push active interrupts out
In reply to Marc ZyngierHaving established that pending interrupts should have priority to be moved into the LRs over the active interrupts, implement this in the ap_list sorting.
--- arch/arm64/kvm/vgic/vgic.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-)
arch/arm64/kvm/vgic/vgic.c +12 -15
@@ -270,10 +270,7 @@ struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq)* well, the first items in the list being the first things populated in the * LRs. * -* A hard rule is that active interrupts can never be pushed out of the LRs -* (and therefore take priority) since we cannot reliably trap on deactivation -* of IRQs and therefore they have to be present in the LRs. -* +* Pending, non-active interrupts must be placed at the head of the list. * Otherwise things should be sorted by the priority field and the GIC * hardware support will take care of preemption of priority groups etc. *
@@ -298,21 +295,21 @@ static int vgic_irq_cmp(void *priv, const struct list_head *a,raw_spin_lock (&irqa->irq_lock);raw_spin_lock_nested (&irqb->irq_lock,SINGLE_DEPTH_NESTING ); -if (irqa->active || irqb->active) { - ret = (int )irqb->active - (int )irqa->active; - goto out; - } + penda = irqa->enabled &&irq_is_pending (irqa) && !irqa->active; + pendb = irqb->enabled &&irq_is_pending (irqb) && !irqb->active; - penda = irqa->enabled &&irq_is_pending (irqa); - pendb = irqb->enabled &&irq_is_pending (irqb); + ret = (int )pendb - (int )penda; +if (ret) + goto out; -if (!penda || !pendb) { - ret = (int )pendb - (int )penda; +/* Both pending and enabled, sort by priority (lower number first) */ + ret = (int )irqa->priority - (int )irqb->priority; +if (ret) goto out; - } -/* Both pending and enabled, sort by priority */ - ret = irqa->priority - irqb->priority; +/* Finally, HW bit active interrupts have priority over non-HW ones */ + ret = (int )irqb->hw - (int )irqa->hw; + out:raw_spin_unlock (&irqb->irq_lock);raw_spin_unlock (&irqa->irq_lock);
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 23/33 KVM: arm64: Move undeliverable interrupts to the end of ap_list
In reply to Marc ZyngierInterrupts in the ap_list that cannot be acted upon because they are not enabled, or that their group is not enabled, shouldn't make it into the LRs if we are space-constrained.
--- arch/arm64/kvm/vgic/vgic.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic.c +22 -1
@@ -265,6 +265,11 @@ struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq)return NULL ; } +struct vgic_sort_info { +struct kvm_vcpu *vcpu; +struct vgic_vmcr vmcr; +}; +/* * The order of items in the ap_lists defines how we'll pack things in LRs as * well, the first items in the list being the first things populated in the
@@ -273,6 +278,7 @@ struct kvm_vcpu *vgic_target_oracle(struct vgic_irq *irq)* Pending, non-active interrupts must be placed at the head of the list. * Otherwise things should be sorted by the priority field and the GIC * hardware support will take care of preemption of priority groups etc. +* Interrupts that are not deliverable should be at the end of the list. * * Return negative if "a" sorts before "b", 0 to preserve order, and positive * to sort "b" before "a".
@@ -282,6 +288,8 @@ static int vgic_irq_cmp(void *priv, const struct list_head *a, { struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list); struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list); +struct vgic_sort_info *info = priv; +struct kvm_vcpu *vcpu = info->vcpu;bool penda, pendb;int ret;
@@ -295,6 +303,17 @@ static int vgic_irq_cmp(void *priv, const struct list_head *a,raw_spin_lock (&irqa->irq_lock);raw_spin_lock_nested (&irqb->irq_lock,SINGLE_DEPTH_NESTING ); +/* Undeliverable interrupts should be last */ + ret = (int )(vgic_target_oracle (irqb) == vcpu) - (int )(vgic_target_oracle (irqa) == vcpu); +if (ret) + goto out; + +/* Same thing for interrupts targeting a disabled group */ + ret = (int )(irqb->group ? info->vmcr.grpen1 : info->vmcr.grpen0); + ret -= (int )(irqa->group ? info->vmcr.grpen1 : info->vmcr.grpen0); +if (ret) + goto out; + penda = irqa->enabled &&irq_is_pending (irqa) && !irqa->active; pendb = irqb->enabled &&irq_is_pending (irqb) && !irqb->active;
@@ -320,10 +339,12 @@ static int vgic_irq_cmp(void *priv, const struct list_head *a,static void vgic_sort_ap_list (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; +struct vgic_sort_info info = { .vcpu = vcpu, };lockdep_assert_held (&vgic_cpu->ap_list_lock); -list_sort (NULL , &vgic_cpu->ap_list_head, vgic_irq_cmp); +vgic_get_vmcr (vcpu, &info.vmcr); +list_sort (&info, &vgic_cpu->ap_list_head, vgic_irq_cmp); }/*
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 24/33 KVM: arm64: Use MI to detect groups being enabled/disabled
In reply to Marc ZyngierAdd the maintenance interrupt to force an exit when the guest enables/disables individual groups, so that we can resort the ap_list accordingly.
--- arch/arm64/kvm/vgic/vgic-v2.c | 5 +++++ arch/arm64/kvm/vgic/vgic-v3.c | 5 +++++ 2 files changed, 10 insertions(+)
arch/arm64/kvm/vgic/vgic-v2.c +5 -0
@@ -39,6 +39,11 @@ void vgic_v2_configure_hcr(struct kvm_vcpu *vcpu, cpuif->vgic_hcr |=GICH_HCR_LRENPIE ;if (irqs_outside_lrs (als)) cpuif->vgic_hcr |=GICH_HCR_UIE ; + + cpuif->vgic_hcr |= (cpuif->vgic_vmcr &GICH_VMCR_ENABLE_GRP0_MASK ) ? + GICH_HCR_VGrp0DIE : GICH_HCR_VGrp0EIE; + cpuif->vgic_hcr |= (cpuif->vgic_vmcr &GICH_VMCR_ENABLE_GRP1_MASK ) ? + GICH_HCR_VGrp1DIE : GICH_HCR_VGrp1EIE; }static bool lr_signals_eoi_mi (u32 lr_val)
arch/arm64/kvm/vgic/vgic-v3.c +5 -0
@@ -36,6 +36,11 @@ void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu,if (!als->nr_sgi) cpuif->vgic_hcr |= ICH_HCR_EL2_vSGIEOICount; + + cpuif->vgic_hcr |= (cpuif->vgic_vmcr &ICH_VMCR_ENG0_MASK ) ? + ICH_HCR_EL2_VGrp0DIE : ICH_HCR_EL2_VGrp0EIE; + cpuif->vgic_hcr |= (cpuif->vgic_vmcr &ICH_VMCR_ENG1_MASK ) ? + ICH_HCR_EL2_VGrp1DIE : ICH_HCR_EL2_VGrp1EIE; }static bool lr_signals_eoi_mi (u64 lr_val)
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 25/33 KVM: arm64: Add AP-list overflow split/splice
In reply to Marc ZyngierWhen EOImode==0, the deactivation of an interrupt outside of the LRs results in an increment of EOIcount, but not much else.
In order to figure out what interrupts did not make it into the LRs, split the ap_list to make it easy to find the interrupts EOIcount applies to. Also provide a bit of documentation for how things are expected to work.
--- arch/arm64/kvm/vgic/vgic-init.c | 2 ++ arch/arm64/kvm/vgic/vgic.c | 48 ++++++++++++++++++++++++++++++++- include/kvm/arm_vgic.h | 6 +++++ 3 files changed, 55 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-init.c +2 -0
@@ -331,6 +331,7 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) vgic_cpu->rd_iodev.base_addr =VGIC_ADDR_UNDEF ;INIT_LIST_HEAD (&vgic_cpu->ap_list_head); +INIT_LIST_HEAD (&vgic_cpu->overflow_ap_list_head);raw_spin_lock_init (&vgic_cpu->ap_list_lock);atomic_set (&vgic_cpu->vgic_v3.its_vpe.vlpi_count,0 );
@@ -455,6 +456,7 @@ static void __kvm_vgic_vcpu_destroy(struct kvm_vcpu *vcpu)vgic_flush_pending_lpis (vcpu);INIT_LIST_HEAD (&vgic_cpu->ap_list_head); +INIT_LIST_HEAD (&vgic_cpu->overflow_ap_list_head);kfree (vgic_cpu->private_irqs); vgic_cpu->private_irqs =NULL ;
arch/arm64/kvm/vgic/vgic.c +47 -1
77 lines not on this page · raw message
include/kvm/arm_vgic.h +6 -0
@@ -359,6 +359,12 @@ struct vgic_cpu {*/ struct list_head ap_list_head; +/* +* List of IRQs that have not made it onto an LR, but still +* notionally par of the AP list +*/ +struct list_head overflow_ap_list_head; +/* * Members below are used with GICv3 emulation only and represent * parts of the redistributor.
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 26/33 KVM: arm64: GICv3: Handle LR overflow when EOImode==0
In reply to Marc ZyngierNow that we can identify interrupts that have not made it into the LRs, it becomes relatively easy to use EOIcount to walk the overflow list.
What is a bit odd is that we compute a fake LR for the original state of the interrupt, clear the active bit, and feed into the existing logic for processing. In a way, this is what would have happened if the interrupt was in an LR.
--- arch/arm64/kvm/vgic/vgic-v3.c | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
arch/arm64/kvm/vgic/vgic-v3.c +45 -0
@@ -109,16 +109,61 @@ static void vgic_v3_fold_lr(struct kvm_vcpu *vcpu, u64 val)vgic_put_irq (vcpu->kvm, irq); } +static u64 vgic_v3_compute_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq); + +static void vgic_v3_deactivate_phys (u32 intid) +{ +if (cpus_have_final_cap (ARM64_HAS_GICV5_LEGACY )) +gic_insn (intid |FIELD_PREP (GICV5_GIC_CDDI_TYPE_MASK ,1 ),CDDI ); +else +gic_write_dir (intid); +} +void vgic_v3_fold_lr_state (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; +u32 eoicount =FIELD_GET (ICH_HCR_EL2_EOIcount, cpuif->vgic_hcr); +struct vgic_irq *irq;DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ());for (int lr =0 ; lr < cpuif->used_lrs; lr++)vgic_v3_fold_lr (vcpu, cpuif->vgic_lr[lr]); +/* +* EOIMode=0: use EOIcount to emulate deactivation. We are +* guaranteed to deactivate in reverse order of the activation, so +* just pick one interrupt after the other in the overflow list, and +* replay the deactivation as if the CPU was doing it. We also rely +* on priority drop to have taken place. +*/ +list_for_each_entry (irq, &vgic_cpu->overflow_ap_list_head, ap_list) { +u64 lr; + +/* +* I would have loved to write this using a scoped_guard(), +* but using 'continue' here is a total train wreck. +*/ +if (!eoicount) { +break ; + }else { +guard (raw_spinlock)(&irq->irq_lock); + +if (!(likely (vgic_target_oracle (irq) == vcpu) && + irq->active)) +continue ; + + lr =vgic_v3_compute_lr (vcpu, irq) & ~ICH_LR_ACTIVE_BIT ; + } + +if (lr &ICH_LR_HW ) +vgic_v3_deactivate_phys (FIELD_GET (ICH_LR_PHYS_ID_MASK , lr)); + +vgic_v3_fold_lr (vcpu, lr); + eoicount--; + } + cpuif->used_lrs =0 ; }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 27/33 KVM: arm64: GICv3: Handle deactivation via ICV_DIR_EL1 traps
In reply to Marc ZyngierDeactivation via ICV_DIR_EL1 is both relatively straightforward (we have the interrupt that needs deactivation) and really awkward.
The main issue is that the interrupt may either be in an LR on another CPU, or ourside of any LR.
In the former case, we process the deactivation is if ot was a write to GICD_CACTIVERn, which is already implemented as a big hammer IPI'ing all vcpus. In the latter case, we just perform a normal deactivation, similar to what we do for EOImode==0.
Another annoying aspect is that we need to tell the CPU owning the interrupt that its ap_list needs laudering. We use a brand new vcpu request to that effect.
Note that this doesn't address deactivation via the GICV MMIO view, which will be taken care of in a later change.
--- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/arm.c | 4 ++ arch/arm64/kvm/hyp/vgic-v3-sr.c | 3 + arch/arm64/kvm/sys_regs.c | 19 ++++++- arch/arm64/kvm/vgic/vgic-v3.c | 93 +++++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic.c | 11 ++++ arch/arm64/kvm/vgic/vgic.h | 1 + include/kvm/arm_vgic.h | 1 + 8 files changed, 131 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -54,6 +54,7 @@#define KVM_REQ_NESTED_S2_UNMAP KVM_ARCH_REQ(8)#define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(9)#define KVM_REQ_MAP_L1_VNCR_EL2 KVM_ARCH_REQ(10) +#define KVM_REQ_VGIC_PROCESS_UPDATE KVM_ARCH_REQ(11)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/kvm/arm.c +4 -0
@@ -1041,6 +1041,10 @@ static int check_vcpu_requests(struct kvm_vcpu *vcpu)*/ kvm_check_request (KVM_REQ_IRQ_PENDING , vcpu); +/* Process interrupts deactivated through a trap */ +if (kvm_check_request (KVM_REQ_VGIC_PROCESS_UPDATE , vcpu)) +kvm_vgic_process_async_update (vcpu); +if (kvm_check_request (KVM_REQ_RECORD_STEAL , vcpu))kvm_update_stolen_time (vcpu);
arch/arm64/kvm/hyp/vgic-v3-sr.c +3 -0
@@ -1247,6 +1247,9 @@ int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu)case SYS_ICC_DIR_EL1 :if (unlikely (is_read))return 0 ; +/* Full exit if required to handle overflow deactivation... */ +if (vcpu->arch.vgic_cpu.vgic_v3.vgic_hcr &ICH_HCR_EL2_TDIR ) +return 0 ; fn = __vgic_v3_write_dir;break ;case SYS_ICC_RPR_EL1 :
arch/arm64/kvm/sys_regs.c +17 -2
@@ -666,6 +666,21 @@ static bool access_gic_sre(struct kvm_vcpu *vcpu,return true; } +static bool access_gic_dir (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (!kvm_has_gicv3 (vcpu->kvm)) +return undef_access (vcpu, p, r); + +if (!p->is_write) +return undef_access (vcpu, p, r); + +vgic_v3_deactivate (vcpu, p->regval); + +return true; +} +static bool trap_raz_wi (struct kvm_vcpu *vcpu,struct sys_reg_params *p,const struct sys_reg_desc *r)
@@ -3370,7 +3385,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_ICC_AP1R1_EL1), undef_access }, { SYS_DESC(SYS_ICC_AP1R2_EL1), undef_access }, { SYS_DESC(SYS_ICC_AP1R3_EL1), undef_access }, - { SYS_DESC(SYS_ICC_DIR_EL1), undef_access }, + { SYS_DESC(SYS_ICC_DIR_EL1), access_gic_dir }, { SYS_DESC(SYS_ICC_RPR_EL1), undef_access }, { SYS_DESC(SYS_ICC_SGI1R_EL1), access_gic_sgi }, { SYS_DESC(SYS_ICC_ASGI1R_EL1), access_gic_sgi },
@@ -4495,7 +4510,7 @@ static const struct sys_reg_desc cp15_regs[] = { { CP15_SYS_DESC(SYS_ICC_AP1R1_EL1), undef_access }, { CP15_SYS_DESC(SYS_ICC_AP1R2_EL1), undef_access }, { CP15_SYS_DESC(SYS_ICC_AP1R3_EL1), undef_access }, - { CP15_SYS_DESC(SYS_ICC_DIR_EL1), undef_access }, + { CP15_SYS_DESC(SYS_ICC_DIR_EL1), access_gic_dir }, { CP15_SYS_DESC(SYS_ICC_RPR_EL1), undef_access }, { CP15_SYS_DESC(SYS_ICC_IAR1_EL1), undef_access }, { CP15_SYS_DESC(SYS_ICC_EOIR1_EL1), undef_access },
arch/arm64/kvm/vgic/vgic-v3.c +93 -0
111 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.c +11 -0
@@ -1036,6 +1036,17 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu)vgic_prune_ap_list (vcpu); } +/* Sync interrupts that were deactivated through a DIR trap */ +void kvm_vgic_process_async_update (struct kvm_vcpu *vcpu) +{ +unsigned long flags; + +/* Make sure we're in the same context as LR handling */ +local_irq_save (flags); +vgic_prune_ap_list (vcpu); +local_irq_restore (flags); +} +static inline void vgic_restore_state (struct kvm_vcpu *vcpu) {if (!static_branch_unlikely (&kvm_vgic_global_state.gicv3_cpuif))
arch/arm64/kvm/vgic/vgic.h +1 -0
@@ -302,6 +302,7 @@ static inline void vgic_get_irq_ref(struct vgic_irq *irq)void vgic_v3_fold_lr_state (struct kvm_vcpu *vcpu);void vgic_v3_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr);void vgic_v3_clear_lr (struct kvm_vcpu *vcpu,int lr); +void vgic_v3_deactivate (struct kvm_vcpu *vcpu,u64 val);void vgic_v3_configure_hcr (struct kvm_vcpu *vcpu,struct ap_list_summary *als);void vgic_v3_set_vmcr (struct kvm_vcpu *vcpu,struct vgic_vmcr *vmcr);void vgic_v3_get_vmcr (struct kvm_vcpu *vcpu,struct vgic_vmcr *vmcr);
include/kvm/arm_vgic.h +1 -0
@@ -427,6 +427,7 @@ bool kvm_vcpu_has_pending_irqs(struct kvm_vcpu *vcpu);void kvm_vgic_sync_hwstate (struct kvm_vcpu *vcpu);void kvm_vgic_flush_hwstate (struct kvm_vcpu *vcpu);void kvm_vgic_reset_mapped_irq (struct kvm_vcpu *vcpu,u32 vintid); +void kvm_vgic_process_async_update (struct kvm_vcpu *vcpu);void vgic_v3_dispatch_sgi (struct kvm_vcpu *vcpu,u64 reg,bool allow_group1);
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 28/33 KVM: arm64: GICv3: Add GICv2 SGI handling to deactivation primitive
In reply to Marc ZyngierThe GICv2 SGIs require additional handling for deactivation, as they are effectively multiple interrrupts muxed into one. Make sure we check for the source CPU when deactivating.
--- arch/arm64/kvm/vgic/vgic-v3.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
arch/arm64/kvm/vgic/vgic-v3.c +19 -1
@@ -177,11 +177,20 @@ void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;struct vgic_v3_cpu_if *cpuif = &vgic_cpu->vgic_v3; +u32 model = vcpu->kvm->arch.vgic.vgic_model;struct kvm_vcpu *target_vcpu =NULL ; +bool mmio = false, is_v2_sgi;struct vgic_irq *irq;unsigned long flags; -bool mmio = false;u64 lr =0 ; +u8 cpuid; + +/* Snapshot CPUID, and remove it from the INTID */ + cpuid =FIELD_GET (GENMASK_ULL (12 ,10 ), val); + val &= ~GENMASK_ULL (12 ,10 ); + + is_v2_sgi = (model ==KVM_DEV_TYPE_ARM_VGIC_V2 && + val <VGIC_NR_SGIS );/* * We only deal with DIR when EOIMode==1, and only for SGI,
@@ -214,6 +223,9 @@ void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val)* and queue a request to prune the resulting ap_list, * * - Or the irq is not active, and there is nothing to do. +* +* Special care must be taken to match the source CPUID when +* deactivating a GICv2 SGI. */ scoped_guard(raw_spinlock, &irq->irq_lock) { target_vcpu =vgic_target_oracle (irq);
@@ -237,6 +249,12 @@ void vgic_v3_deactivate(struct kvm_vcpu *vcpu, u64 val) goto put; } +/* GICv2 SGI: check that the cpuid matches */ +if (is_v2_sgi && irq->active_source != cpuid) { + target_vcpu =NULL ; + goto put; + } +/* (with a Dalek voice) DEACTIVATE!!!! */ lr =vgic_v3_compute_lr (vcpu, irq) & ~ICH_LR_ACTIVE_BIT ; }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 29/33 KVM: arm64: GICv3: Set ICH_HCR_EL2.TDIR when interrupts overflow LR capacity
In reply to Marc ZyngierNow that we are ready to handle deactivation through ICV_DIR_EL1, set the trap bit if we have active interrupts outside of the LRs.
--- arch/arm64/kvm/vgic/vgic-v3.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/arm64/kvm/vgic/vgic-v3.c +7 -0
@@ -42,6 +42,13 @@ void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu, ICH_HCR_EL2_VGrp0DIE : ICH_HCR_EL2_VGrp0EIE; cpuif->vgic_hcr |= (cpuif->vgic_vmcr &ICH_VMCR_ENG1_MASK ) ? ICH_HCR_EL2_VGrp1DIE : ICH_HCR_EL2_VGrp1EIE; + +/* +* Note that we set the trap irrespective of EOIMode, as that +* can change behind our back without any warning... +*/ +if (irqs_active_outside_lrs (als)) + cpuif->vgic_hcr |=ICH_HCR_EL2_TDIR ; }static bool lr_signals_eoi_mi (u64 lr_val)
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 30/33 KVM: arm64: GICv2: Handle LR overflow when EOImode==0
In reply to Marc ZyngierSimilarly to the GICv3 version, handle the EOIcount-driven deactivation by walking the overflow list.
--- arch/arm64/kvm/vgic/vgic-v2.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
arch/arm64/kvm/vgic/vgic-v2.c +27 -0
@@ -100,6 +100,8 @@ static void vgic_v2_fold_lr(struct kvm_vcpu *vcpu, u32 val)vgic_put_irq (vcpu->kvm, irq); } +static u32 vgic_v2_compute_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq); +/* * transfer the content of the LRs back into the corresponding ap_list: * - active bit is transferred as is
@@ -111,12 +113,37 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2; +u32 eoicount =FIELD_GET (GICH_HCR_EOICOUNT , cpuif->vgic_hcr); +struct vgic_irq *irq;DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ());for (int lr =0 ; lr < vgic_cpu->vgic_v2.used_lrs; lr++)vgic_v2_fold_lr (vcpu, cpuif->vgic_lr[lr]); +/* See the GICv3 equivalent for the EOIcount handling rationale */ +list_for_each_entry (irq, &vgic_cpu->overflow_ap_list_head, ap_list) { +u32 lr; + +if (!eoicount) { +break ; + }else { +guard (raw_spinlock)(&irq->irq_lock); + +if (!(likely (vgic_target_oracle (irq) == vcpu) && + irq->active)) +continue ; + + lr =vgic_v2_compute_lr (vcpu, irq) & ~GICH_LR_ACTIVE_BIT ; + } + +if (lr &GICH_LR_HW ) +writel_relaxed (FIELD_GET (GICH_LR_PHYSID_CPUID , lr), + kvm_vgic_global_state.gicc_base +GIC_CPU_DEACTIVATE ); +vgic_v2_fold_lr (vcpu, lr); + eoicount--; + } + cpuif->used_lrs =0 ; }
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 31/33 KVM: arm64: GICv2: Handle deactivation via GICV_DIR traps
In reply to Marc ZyngierAdd the plumbing of GICv2 interrupt deactivation via GICV_DIR. This requires adding a new device so that we can easily decode the DIR address.
The deactivation itself is very similar to the GICv3 version.
--- arch/arm64/kvm/vgic/vgic-mmio-v2.c | 24 +++++++ arch/arm64/kvm/vgic/vgic-mmio.h | 1 + arch/arm64/kvm/vgic/vgic-v2.c | 110 +++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic.h | 1 + include/kvm/arm_vgic.h | 1 + 5 files changed, 137 insertions(+)
arch/arm64/kvm/vgic/vgic-mmio-v2.c +24 -0
@@ -359,6 +359,16 @@ static void vgic_mmio_write_vcpuif(struct kvm_vcpu *vcpu,vgic_set_vmcr (vcpu, &vmcr); } +static void vgic_mmio_write_dir (struct kvm_vcpu *vcpu, +gpa_t addr,unsigned int len, +unsigned long val) +{ +if (kvm_vgic_global_state.type ==VGIC_V2 ) +vgic_v2_deactivate (vcpu, val); +else +vgic_v3_deactivate (vcpu, val); +} +static unsigned long vgic_mmio_read_apr (struct kvm_vcpu *vcpu,gpa_t addr,unsigned int len) {
@@ -482,6 +492,10 @@ static const struct vgic_register_region vgic_v2_cpu_registers[] = {REGISTER_DESC_WITH_LENGTH (GIC_CPU_IDENT , vgic_mmio_read_vcpuif, vgic_mmio_write_vcpuif,4 , VGIC_ACCESS_32bit), +REGISTER_DESC_WITH_LENGTH_UACCESS (GIC_CPU_DEACTIVATE , + vgic_mmio_read_raz, vgic_mmio_write_dir, + vgic_mmio_read_raz, vgic_mmio_uaccess_write_wi, + 4, VGIC_ACCESS_32bit), };unsigned int vgic_v2_init_dist_iodev (struct vgic_io_device *dev)
@@ -494,6 +508,16 @@ unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev)return SZ_4K ; } +unsigned int vgic_v2_init_cpuif_iodev (struct vgic_io_device *dev) +{ + dev->regions = vgic_v2_cpu_registers; + dev->nr_regions =ARRAY_SIZE (vgic_v2_cpu_registers); + +kvm_iodevice_init (&dev->dev, &kvm_io_gic_ops); + +return KVM_VGIC_V2_CPU_SIZE ; +} +int vgic_v2_has_attr_regs (struct kvm_device *dev,struct kvm_device_attr *attr) {const struct vgic_register_region *region;
arch/arm64/kvm/vgic/vgic-mmio.h +1 -0
@@ -213,6 +213,7 @@ void vgic_write_irq_line_level_info(struct kvm_vcpu *vcpu, u32 intid, const u32 val);unsigned int vgic_v2_init_dist_iodev (struct vgic_io_device *dev); +unsigned int vgic_v2_init_cpuif_iodev (struct vgic_io_device *dev);unsigned int vgic_v3_init_dist_iodev (struct vgic_io_device *dev);
arch/arm64/kvm/vgic/vgic-v2.c +110 -0
134 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic.h +1 -0
@@ -261,6 +261,7 @@ int vgic_check_iorange(struct kvm *kvm, phys_addr_t ioaddr,void vgic_v2_fold_lr_state (struct kvm_vcpu *vcpu);void vgic_v2_populate_lr (struct kvm_vcpu *vcpu,struct vgic_irq *irq,int lr); +void vgic_v2_deactivate (struct kvm_vcpu *vcpu,u32 val);void vgic_v2_clear_lr (struct kvm_vcpu *vcpu,int lr);void vgic_v2_configure_hcr (struct kvm_vcpu *vcpu,struct ap_list_summary *als);int vgic_v2_has_attr_regs (struct kvm_device *dev,struct kvm_device_attr *attr);
include/kvm/arm_vgic.h +1 -0
@@ -284,6 +284,7 @@ struct vgic_dist {struct vgic_irq *spis;struct vgic_io_device dist_iodev; +struct vgic_io_device cpuif_iodev;bool has_its;bool table_write_in_progress;
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 32/33 KVM: arm64: GICv2: Always trap GICV_DIR register
In reply to Marc ZyngierSince we can't decide to trap the DIR register on a per-vcpu basis, always trap the second page of the GIC CPU interface. Yes, this is costly. On the bright side, no sane SW should use EOImode==1 on GICv2...
--- arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c | 4 ++++ arch/arm64/kvm/vgic/vgic-v2.c | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c +4 -0
@@ -63,6 +63,10 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)return -1 ; } +/* Handle deactivation as a normal exit */ +if ((fault_ipa - vgic->vgic_cpu_base) >=GIC_CPU_DEACTIVATE ) +return 0 ; + rd =kvm_vcpu_dabt_get_rd (vcpu); addr = kvm_vgic_global_state.vcpu_hyp_va; addr += fault_ipa - vgic->vgic_cpu_base;
arch/arm64/kvm/vgic/vgic-v2.c +1 -1
@@ -485,7 +485,7 @@ int vgic_v2_map_resources(struct kvm *kvm)if (!static_branch_unlikely (&vgic_v2_cpuif_trap)) { ret =kvm_phys_addr_ioremap (kvm, dist->vgic_cpu_base, kvm_vgic_global_state.vcpu_base, -KVM_VGIC_V2_CPU_SIZE , true); +KVM_VGIC_V2_CPU_SIZE -SZ_4K , true);if (ret) {kvm_err ("Unable to remap VGIC CPU to VCPU\n" );return ret;
Signature · 1 line
2.47.3
Marc Zyngier
maintainerPATCH 33/33 KVM: arm64: GICv3: Add SPI tracking to handle asymmetric deactivation
In reply to Marc ZyngierSPIs are specially annpying, as they can be activated on a CPU and deactivated on another. WHich means that when an SPI is in flight anywhere, all CPUs need to have their TDIR trap bit set.
This translates into broadcasting an IPI across all CPUs to make sure they set their trap bit, The number of in-flight SPIs is kept in an atomic variable so that CPUs can turn the trap bit off as soon as possible.
--- arch/arm64/kvm/vgic/vgic-init.c | 1 + arch/arm64/kvm/vgic/vgic-v3.c | 21 +++++++++++++++------ arch/arm64/kvm/vgic/vgic.c | 14 ++++++++++++-- arch/arm64/kvm/vgic/vgic.h | 6 ++++++ include/kvm/arm_vgic.h | 3 +++ 5 files changed, 37 insertions(+), 8 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +1 -0
@@ -188,6 +188,7 @@ static int kvm_vgic_dist_init(struct kvm *kvm, unsigned int nr_spis)struct kvm_vcpu *vcpu0 =kvm_get_vcpu (kvm,0 );int i; + dist->active_spis = (atomic_t )ATOMIC_INIT (0 ); dist->spis =kcalloc (nr_spis,sizeof (struct vgic_irq ),GFP_KERNEL_ACCOUNT );if (!dist->spis)return -ENOMEM ;
arch/arm64/kvm/vgic/vgic-v3.c +15 -6
@@ -44,10 +44,17 @@ void vgic_v3_configure_hcr(struct kvm_vcpu *vcpu, ICH_HCR_EL2_VGrp1DIE : ICH_HCR_EL2_VGrp1EIE;/* +* Dealing with EOImode=1 is a massive source of headache. Not +* only do we need to track that we have active interrupts +* outside of the LRs and force DIR to be trapped, we also +* need to deal with SPIs that can be deactivated on another +* CPU. +* * Note that we set the trap irrespective of EOIMode, as that * can change behind our back without any warning... */ -if (irqs_active_outside_lrs (als)) +if (irqs_active_outside_lrs (als) || +atomic_read (&vcpu->kvm->arch.vgic.active_spis)) cpuif->vgic_hcr |=ICH_HCR_EL2_TDIR ; }
@@ -75,11 +82,6 @@ static void vgic_v3_fold_lr(struct kvm_vcpu *vcpu, u64 val)if (!irq)/* An LPI could have been unmapped. */ return ; -/* Notify fds when the guest EOI'ed a level-triggered IRQ */ -if (lr_signals_eoi_mi (val) &&vgic_valid_spi (vcpu->kvm, intid)) -kvm_notify_acked_irq (vcpu->kvm,0 , - intid -VGIC_NR_PRIVATE_IRQS ); - scoped_guard(raw_spinlock, &irq->irq_lock) {/* Always preserve the active bit for !LPIs, note deactivation */ if (irq->intid >=VGIC_MIN_LPI )
@@ -114,6 +116,13 @@ static void vgic_v3_fold_lr(struct kvm_vcpu *vcpu, u64 val) irq->on_lr = false; } +/* Notify fds when the guest EOI'ed a level-triggered SPI, and drop the refcount */ +if (deactivated &&lr_signals_eoi_mi (val) &&vgic_valid_spi (vcpu->kvm, intid)) { +kvm_notify_acked_irq (vcpu->kvm,0 , + intid -VGIC_NR_PRIVATE_IRQS ); +atomic_dec_if_positive (&vcpu->kvm->arch.vgic.active_spis); + } +vgic_put_irq (vcpu->kvm, irq); }
arch/arm64/kvm/vgic/vgic.c +12 -2
@@ -379,6 +379,7 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq, unsigned long flags) __releases(&irq->irq_lock) {struct kvm_vcpu *vcpu; +bool bcast;lockdep_assert_held (&irq->irq_lock);
@@ -453,11 +454,20 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,list_add_tail (&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head); irq->vcpu = vcpu; +/* A new SPI may result in deactivation trapping on all vcpus */ + bcast = (vgic_model_needs_bcst_kick (vcpu->kvm) && +vgic_valid_spi (vcpu->kvm, irq->intid) && +atomic_fetch_inc (&vcpu->kvm->arch.vgic.active_spis) ==0 ); +raw_spin_unlock (&irq->irq_lock);raw_spin_unlock_irqrestore (&vcpu->arch.vgic_cpu.ap_list_lock, flags); -kvm_make_request (KVM_REQ_IRQ_PENDING , vcpu); -kvm_vcpu_kick (vcpu); +if (!bcast) { +kvm_make_request (KVM_REQ_IRQ_PENDING , vcpu); +kvm_vcpu_kick (vcpu); + }else { +kvm_make_all_cpus_request (vcpu->kvm,KVM_REQ_IRQ_PENDING ); + }return true; }
arch/arm64/kvm/vgic/vgic.h +6 -0
@@ -164,6 +164,12 @@ static inline int vgic_write_guest_lock(struct kvm *kvm, gpa_t gpa,return ret; } +static inline bool vgic_model_needs_bcst_kick (struct kvm *kvm) +{ +return (cpus_have_final_cap (ARM64_HAS_ICH_HCR_EL2_TDS ) && + kvm->arch.vgic.vgic_model ==KVM_DEV_TYPE_ARM_VGIC_V3 ); +} +/* * This struct provides an intermediate representation of the fields contained * in the GICH_VMCR and ICH_VMCR registers, such that code exporting the GIC
include/kvm/arm_vgic.h +3 -0
@@ -263,6 +263,9 @@ struct vgic_dist {/* The GIC maintenance IRQ for nested hypervisors. */ u32 mi_intid; +/* Track the number of in-flight active SPIs */ +atomic_t active_spis; +/* base addresses in guest physical address space: */ gpa_t vgic_dist_base;/* distributor */ union {
Signature · 1 line
2.47.3
Re: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping
In reply to Marc ZyngierQuoted 49 lines · Marc Zyngier
A long time ago, an unsuspecting architect forgot to add a trap bit for
ICV_DIR_EL1inICH_HCR_EL2. Which was unfortunate, but what's a bit of spec between friends? Thankfully, this was fixed in a later revision, and ARM "deprecates" the lack of trapping ability.Unfortuantely, a few (billion) CPUs went out with that defect, anything ARMv8.0 from ARM, give or take. And on these CPUs, you can't trap DIR on its own, full stop.
As the next best thing, we can trap everything in the common group, which is a tad expensive, but hey ho, that's what you get. You can otherwise recycle the HW in the neaby bin.
Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/virt.h | 7 ++++++- arch/arm64/kernel/cpufeature.c | 34 ++++++++++++++++++++++++++++++++++ arch/arm64/kernel/hyp-stub.S | 5 +++++ arch/arm64/kvm/vgic/vgic-v3.c | 3 +++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 49 insertions(+), 1 deletion(-)arch/arm64/include/asm/virt.h +6
-1@@ -40,8 +40,13 @@*/ #define HVC_FINALISE_EL2 3 +/* +* HVC_GET_ICH_VTR_EL2 - Retrieve the ICH_VTR_EL2 value +*/ +#define HVC_GET_ICH_VTR_EL2 4 +/* Max number of HYP stub hypercalls */ -#define HVC_STUB_HCALL_NR 4 +#define HVC_STUB_HCALL_NR 5/* Error returned when an invalid stub number is passed into x0 */ #define HVC_STUB_ERR 0xbadca11arch/arm64/kernel/cpufeature.c +0
-0@@ -2303,6 +2303,31 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry }#endif
arch/arm64/kernel/cpufeature.c static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry · line 2314
Quoted 10 lines
arch/arm64/kernel/cpufeature.c +9
-0+static bool can_trap_icv_dir_el1 (const struct arm64_cpu_capabilities *entry, +int scope) +{ +struct arm_smccc_res res = {}; + +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV3_CPUIF ); +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV5_LEGACY ); +if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ) || + !cpus_have_cap (ARM64_HAS_GICV3_CPUIF ))
Duplicated checking ?
Quoted 80 lines
arch/arm64/kernel/cpufeature.c +25
-0+return false; + +if (!is_hyp_mode_available ()) +return false; + +if (is_kernel_in_hyp_mode ()) + res.a1 =read_sysreg_s (SYS_ICH_VTR_EL2 ); +else +arm_smccc_1_1_hvc (HVC_GET_ICH_VTR_EL2 , &res); + +if (res.a0 ==HVC_STUB_ERR ) +return false; + +return res.a1 &ICH_VTR_EL2_TDS ; +} +#ifdef CONFIG_ARM64_BTI static void bti_enable (const struct arm64_cpu_capabilities *__unused) {@@ -2814,6 +2839,15 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_gic_prio_relaxed_sync, },#endif + { + /* + * Depends on having GICv3 + */ + .desc = "ICV_DIR_EL1 trapping", + .capability = ARM64_HAS_ICH_HCR_EL2_TDS, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = can_trap_icv_dir_el1, + }, #ifdef CONFIG_ARM64_E0PD { .desc = "E0PD",arch/arm64/kernel/hyp-stub.S +5
-0@@ -54,6 +54,11 @@ SYM_CODE_START_LOCAL(elx_sync) 1: cmp x0, #HVC_FINALISE_EL2 b.eq __finalise_el2 + cmp x0, #HVC_GET_ICH_VTR_EL2 + b.ne 2f + mrs x1, ich_vtr_el2 + b 9f + 2: cmp x0, #HVC_SOFT_RESTART b.ne 3f mov x0, x2arch/arm64/kvm/vgic/vgic-v3.c +3
-0@@ -648,6 +648,9 @@ void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt, dir_trap = true; } +if (!cpus_have_cap (ARM64_HAS_ICH_HCR_EL2_TDS )) + common_trap = true; +if (group0_trap) hcr |=ICH_HCR_EL2_TALL0 ;if (group1_trap)arch/arm64/tools/cpucaps +1
-1@@ -40,6 +40,7 @@ HAS_GICV5_CPUIF HAS_GICV5_LEGACY HAS_GIC_PRIO_MASKING HAS_GIC_PRIO_RELAXED_SYNC +HAS_ICH_HCR_EL2_TDS HAS_HCR_NV1 HAS_HCX HAS_LDAPR --2.47.3
Marc Zyngier
maintainerRe: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping
Quoted 52 lines · Yao Yuan
Quoted 50 lines · Marc Zyngier
A long time ago, an unsuspecting architect forgot to add a trap bit for
ICV_DIR_EL1inICH_HCR_EL2. Which was unfortunate, but what's a bit of spec between friends? Thankfully, this was fixed in a later revision, and ARM "deprecates" the lack of trapping ability.Unfortuantely, a few (billion) CPUs went out with that defect, anything ARMv8.0 from ARM, give or take. And on these CPUs, you can't trap DIR on its own, full stop.
As the next best thing, we can trap everything in the common group, which is a tad expensive, but hey ho, that's what you get. You can otherwise recycle the HW in the neaby bin.
Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/virt.h | 7 ++++++- arch/arm64/kernel/cpufeature.c | 34 ++++++++++++++++++++++++++++++++++ arch/arm64/kernel/hyp-stub.S | 5 +++++ arch/arm64/kvm/vgic/vgic-v3.c | 3 +++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 49 insertions(+), 1 deletion(-)arch/arm64/include/asm/virt.h +6
-1@@ -40,8 +40,13 @@*/ #define HVC_FINALISE_EL2 3 +/* +* HVC_GET_ICH_VTR_EL2 - Retrieve the ICH_VTR_EL2 value +*/ +#define HVC_GET_ICH_VTR_EL2 4 +/* Max number of HYP stub hypercalls */ -#define HVC_STUB_HCALL_NR 4 +#define HVC_STUB_HCALL_NR 5/* Error returned when an invalid stub number is passed into x0 */ #define HVC_STUB_ERR 0xbadca11arch/arm64/kernel/cpufeature.c +0
-0@@ -2303,6 +2303,31 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry }#endif
Quoted 11 lines
arch/arm64/kernel/cpufeature.c static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry · line 2314
Quoted 9 lines
arch/arm64/kernel/cpufeature.c +9
-0+static bool can_trap_icv_dir_el1 (const struct arm64_cpu_capabilities *entry, +int scope) +{ +struct arm_smccc_res res = {}; + +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV3_CPUIF ); +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV5_LEGACY ); +if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ) || + !cpus_have_cap (ARM64_HAS_GICV3_CPUIF ))Duplicated checking ?
Yup, cut'n'paste, and lack of GICv5 testing... This should really look like the hack below, since GICv5 legacy feature is guaranteed to have TDIR:
arch/arm64/kernel/cpufeature.c +4 -2
@@ -2310,13 +2310,15 @@ static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV3_CPUIF );BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV5_LEGACY ); -if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ) || - !cpus_have_cap (ARM64_HAS_GICV3_CPUIF )) +if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ))return false;if (!is_hyp_mode_available ())return false; +if (cpus_have_cap (ARM64_HAS_GICV5_LEGACY )) +return true; +if (is_kernel_in_hyp_mode ()) res.a1 =read_sysreg_s (SYS_ICH_VTR_EL2 );else
Thanks for the heads-up!
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping
Quoted 66 lines · Marc Zyngier
Quoted 52 lines · Yao Yuan
Quoted 50 lines · Marc Zyngier
A long time ago, an unsuspecting architect forgot to add a trap bit for
ICV_DIR_EL1inICH_HCR_EL2. Which was unfortunate, but what's a bit of spec between friends? Thankfully, this was fixed in a later revision, and ARM "deprecates" the lack of trapping ability.Unfortuantely, a few (billion) CPUs went out with that defect, anything ARMv8.0 from ARM, give or take. And on these CPUs, you can't trap DIR on its own, full stop.
As the next best thing, we can trap everything in the common group, which is a tad expensive, but hey ho, that's what you get. You can otherwise recycle the HW in the neaby bin.
Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/virt.h | 7 ++++++- arch/arm64/kernel/cpufeature.c | 34 ++++++++++++++++++++++++++++++++++ arch/arm64/kernel/hyp-stub.S | 5 +++++ arch/arm64/kvm/vgic/vgic-v3.c | 3 +++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 49 insertions(+), 1 deletion(-)arch/arm64/include/asm/virt.h +6
-1@@ -40,8 +40,13 @@*/ #define HVC_FINALISE_EL2 3 +/* +* HVC_GET_ICH_VTR_EL2 - Retrieve the ICH_VTR_EL2 value +*/ +#define HVC_GET_ICH_VTR_EL2 4 +/* Max number of HYP stub hypercalls */ -#define HVC_STUB_HCALL_NR 4 +#define HVC_STUB_HCALL_NR 5/* Error returned when an invalid stub number is passed into x0 */ #define HVC_STUB_ERR 0xbadca11arch/arm64/kernel/cpufeature.c +0
-0@@ -2303,6 +2303,31 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry }#endif Quoted 11 lines
arch/arm64/kernel/cpufeature.c static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry · line 2314
Quoted 9 lines
arch/arm64/kernel/cpufeature.c +9
-0+static bool can_trap_icv_dir_el1 (const struct arm64_cpu_capabilities *entry, +int scope) +{ +struct arm_smccc_res res = {}; + +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV3_CPUIF ); +BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV5_LEGACY ); +if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ) || + !cpus_have_cap (ARM64_HAS_GICV3_CPUIF ))Duplicated checking ?
Quoted 1 line
Yup, cut'n'paste, and lack of GICv5 testing... This should really look
Thanks for the quick reply!
Oh, not awared that checking GICv5 is necessary here. I looked v3/v4 spec before but stop before v5. Ok it's time to have a look on v5.
Quoted 33 lines
like the hack below, since GICv5 legacy feature is guaranteed to have TDIR:
arch/arm64/kernel/cpufeature.c +4
-2@@ -2310,13 +2310,15 @@ static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV3_CPUIF );BUILD_BUG_ON (ARM64_HAS_ICH_HCR_EL2_TDS <=ARM64_HAS_GICV5_LEGACY ); -if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ) || - !cpus_have_cap (ARM64_HAS_GICV3_CPUIF )) +if (!cpus_have_cap (ARM64_HAS_GICV3_CPUIF ))return false;if (!is_hyp_mode_available ())return false; +if (cpus_have_cap (ARM64_HAS_GICV5_LEGACY )) +return true; +if (is_kernel_in_hyp_mode ()) res.a1 =read_sysreg_s (SYS_ICH_VTR_EL2 );else Thanks for the heads-up!
M.--
Without deviation from the norm, progress is not possible.
Zenghui Yu
reviewerRe: [PATCH 03/33] irqchip/apple-aic: Spit out ICH_MIDR_EL2 value on spurious vGIC MI
In reply to Marc ZyngierSubject: s/ICH_MIDR_EL2/ICH_MISR_EL2/
Thanks,
Zenghui
kernel test robot
build botRe: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping
In reply to Marc Zyngierkernel test robot noticed the following build errors:
Hi Marc,
kernel test robot noticed the following build errors:
[auto build test ERROR on kvmarm/next]
[also build test ERROR on linus/master v6.18-rc4 next-20251104] [cannot apply to arm64/for-next/core tip/irq/core]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Marc-Zyngier/irqchip-gic-Add-missing-GICH_HCR-control-bits/20251104-011133 base: https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next patch link: https://lore.kernel.org/r/20251103165517.2960148-6-maz%40kernel.org patch subject: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping config: arm64-randconfig-003-20251105 (https://download.01.org/0day-ci/archive/20251105/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 8.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251105/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
arch/arm64/kernel/hyp-stub.S: Assembler messages:
>> arch/arm64/kernel/hyp-stub.S:59: Error: unknown or missing system register name at operand 2 -- `mrs x1,ich_vtr_el2'
vim +59 arch/arm64/kernel/hyp-stub.S
45
46 .align 11
47
48 SYM_CODE_START_LOCAL(elx_sync)
49 cmp x0, #HVC_SET_VECTORS
50 b.ne 1f
51 msr vbar_el2, x1
52 b 9f
53
54 1: cmp x0, #HVC_FINALISE_EL2
55 b.eq __finalise_el2
56
57 cmp x0, #HVC_GET_ICH_VTR_EL2
58 b.ne 2f
> 59 mrs x1, ich_vtr_el2
60 b 9f
61 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Marc Zyngier
maintainerRe: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping
Quoted 33 lines · kernel test robot
Hi Marc,
kernel test robot noticed the following build errors:
[auto build test ERROR on kvmarm/next]
[also build test ERROR on linus/master v6.18-rc4 next-20251104] [cannot apply to arm64/for-next/core tip/irq/core]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Marc-Zyngier/irqchip-gic-Add-missing-GICH_HCR-control-bits/20251104-011133 base: https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next patch link: https://lore.kernel.org/r/20251103165517.2960148-6-maz%40kernel.org patch subject: [PATCH 05/33] KVM: arm64: GICv3: Detect and work around the lack of ICV_DIR_EL1 trapping config: arm64-randconfig-003-20251105 (https://download.01.org/0day-ci/archive/20251105/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 8.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251105/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
arch/arm64/kernel/hyp-stub.S: Assembler messages:Quoted 1 line
Quoted 1 line
arch/arm64/kernel/hyp-stub.S:59: Error: unknown or missing system register name at operand 2 -- `mrs x1,ich_vtr_el2'vim +59
arch/arm64/kernel/hyp-stub.S
Quoted 15 lines
45 46 .align 11 47 48 SYM_CODE_START_LOCAL(elx_sync) 49 cmp x0, #HVC_SET_VECTORS 50 b.ne 1f 51 msr vbar_el2, x1 52 b 9f 53 54 1: cmp x0, #HVC_FINALISE_EL2 55 b.eq __finalise_el2 56 57 cmp x0, #HVC_GET_ICH_VTR_EL2 58 b.ne 2fQuoted 1 line
59 mrs x1,
ich_vtr_el2
Ah, ancient assembler that doesn't understand the GICv3 registers... Oh well, I'll give it the generated encoding instead.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.