KVM: arm64: nv: Add support for address translation instructions
Marc Zyngier
maintainerv4 cover KVM: arm64: nv: Add support for address translation instructions
This is the fourth revision of the address translation emulation for NV support on arm64 previously posted at [1].
Thanks again to Alex for his continuous (contiguous? ;-) scrutiny on this series.
* From v3:
- Fix out of range conditions for TxSZ when LVA is implemented
- Fix implementation of R_VPBBF to deliver an Address Size Fault
- Don't grant PX if UW is set
- Various cleanups
- Collected Alex's RBs, with thanks.
I've added the usual reviewers on Cc, plus people who explicitly asked to be on it, and people who seem to be super keen on NV.
Patches on top of 6.11-rc1, tested on my usual M2 (so VHE only). FWIW, I plan to take this into 6.12.
[1] https://lore.kernel.org/r/[email protected]
Joey Gouly (1):
KVM: arm64: Make kvm_at() take an OP_AT_*
Marc Zyngier (17):
arm64: Add missing APTable and TCR_ELx.HPD masks
arm64: Add PAR_EL1 field description
arm64: Add system register encoding for PSTATE.PAN
arm64: Add ESR_ELx_FSC_ADDRSZ_L() helper
KVM: arm64: nv: Enforce S2 alignment when contiguous bit is set
KVM: arm64: nv: Turn upper_attr for S2 walk into the full descriptor
KVM: arm64: nv: Honor absence of FEAT_PAN2
KVM: arm64: nv: Add basic emulation of AT S1E{0,1}{R,W}
KVM: arm64: nv: Add basic emulation of AT S1E1{R,W}P
KVM: arm64: nv: Add basic emulation of AT S1E2{R,W}
KVM: arm64: nv: Add emulation of AT S12E{0,1}{R,W}
KVM: arm64: nv: Make ps_to_output_size() generally available
KVM: arm64: nv: Add SW walker for AT S1 emulation
KVM: arm64: nv: Sanitise SCTLR_EL1.EPAN according to VM configuration
KVM: arm64: nv: Make AT+PAN instructions aware of FEAT_PAN3
KVM: arm64: nv: Plumb handling of AT S1* traps from EL2
KVM: arm64: nv: Add support for FEAT_ATS1A
arch/arm64/include/asm/esr.h | 5 +- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_asm.h | 6 +- arch/arm64/include/asm/kvm_nested.h | 40 +- arch/arm64/include/asm/pgtable-hwdef.h | 9 + arch/arm64/include/asm/sysreg.h | 22 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/at.c | 1101 ++++++++++++++++++++++++ arch/arm64/kvm/emulate-nested.c | 2 + arch/arm64/kvm/hyp/include/hyp/fault.h | 2 +- arch/arm64/kvm/nested.c | 41 +- arch/arm64/kvm/sys_regs.c | 60 ++ 12 files changed, 1259 insertions(+), 32 deletions(-) create mode 100644 arch/arm64/kvm/at.c
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 1/18 arm64: Add missing APTable and TCR_ELx.HPD masks
Although Linux doesn't make use of hierarchical permissions (TFFT!), KVM needs to know where the various bits related to this feature live in the TCR_ELx registers as well as in the page tables.
Add the missing bits.
--- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/pgtable-hwdef.h | 9 +++++++++ 2 files changed, 10 insertions(+)
arch/arm64/include/asm/kvm_arm.h +1 -0
@@ -107,6 +107,7 @@/* TCR_EL2 Registers bits */ #define TCR_EL2_DS (1UL << 32)#define TCR_EL2_RES1 ((1U << 31) | (1 << 23)) +#define TCR_EL2_HPD (1 << 24)#define TCR_EL2_TBI (1 << 20)#define TCR_EL2_PS_SHIFT 16#define TCR_EL2_PS_MASK (7 << TCR_EL2_PS_SHIFT)
arch/arm64/include/asm/pgtable-hwdef.h +9 -0
@@ -204,6 +204,11 @@*/ #define PTE_S2_MEMATTR (t) (_AT(pteval_t, (t)) << 2) +/* +* Hierarchical permission for Stage-1 tables +*/ +#define S1_TABLE_AP (_AT(pmdval_t, 3) << 61) +/* * Highest possible physical address supported. */
@@ -298,6 +303,10 @@#define TCR_TBI1 (UL(1) << 38)#define TCR_HA (UL(1) << 39)#define TCR_HD (UL(1) << 40) +#define TCR_HPD0_SHIFT 41 +#define TCR_HPD0 (UL(1) << TCR_HPD0_SHIFT) +#define TCR_HPD1_SHIFT 42 +#define TCR_HPD1 (UL(1) << TCR_HPD1_SHIFT)#define TCR_TBID0 (UL(1) << 51)#define TCR_TBID1 (UL(1) << 52)#define TCR_NFD0 (UL(1) << 53)
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 2/18 arm64: Add PAR_EL1 field description
In reply to Marc ZyngierAs KVM is about to grow a full emulation for the AT instructions, add the layout of the PAR_EL1 register in its non-D128 configuration.
Note that the constants are a bit ugly, as the register has two layouts, based on the state of the F bit.
--- arch/arm64/include/asm/sysreg.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
arch/arm64/include/asm/sysreg.h +18 -0
@@ -325,7 +325,25 @@#define SYS_PAR_EL1 sys_reg(3, 0, 7, 4, 0)#define SYS_PAR_EL1_F BIT(0) +/* When PAR_EL1.F == 1 */ #define SYS_PAR_EL1_FST GENMASK(6, 1) +#define SYS_PAR_EL1_PTW BIT(8) +#define SYS_PAR_EL1_S BIT(9) +#define SYS_PAR_EL1_AssuredOnly BIT(12) +#define SYS_PAR_EL1_TopLevel BIT(13) +#define SYS_PAR_EL1_Overlay BIT(14) +#define SYS_PAR_EL1_DirtyBit BIT(15) +#define SYS_PAR_EL1_F1_IMPDEF GENMASK_ULL(63, 48) +#define SYS_PAR_EL1_F1_RES0 (BIT(7) | BIT(10) | GENMASK_ULL(47, 16)) +#define SYS_PAR_EL1_RES1 BIT(11) +/* When PAR_EL1.F == 0 */ +#define SYS_PAR_EL1_SH GENMASK_ULL(8, 7) +#define SYS_PAR_EL1_NS BIT(9) +#define SYS_PAR_EL1_F0_IMPDEF BIT(10) +#define SYS_PAR_EL1_NSE BIT(11) +#define SYS_PAR_EL1_PA GENMASK_ULL(51, 12) +#define SYS_PAR_EL1_ATTR GENMASK_ULL(63, 56) +#define SYS_PAR_EL1_F0_RES0 (GENMASK_ULL(6, 1) | GENMASK_ULL(55, 52))/*** Statistical Profiling Extension ***/ #define PMSEVFR_EL1_RES0_IMP \
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 3/18 arm64: Add system register encoding for PSTATE.PAN
In reply to Marc ZyngierAlthough we already have the primitives to set PSTATE.PAN with an immediate, we don't have a way to read the current state nor set it ot an arbitrary value (i.e. we can generally save/restore it).
Thankfully, all that is missing for this is the definition for the PAN pseudo system register, here named SYS_PSTATE_PAN.
--- arch/arm64/include/asm/sysreg.h | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/include/asm/sysreg.h +3 -0
@@ -109,6 +109,9 @@#define set_pstate_ssbs (x) asm volatile(SET_PSTATE_SSBS(x))#define set_pstate_dit (x) asm volatile(SET_PSTATE_DIT(x)) +/* Register-based PAN access, for save/restore purposes */ +#define SYS_PSTATE_PAN sys_reg(3, 0, 4, 2, 3) +#define __SYS_BARRIER_INSN (CRm, op2, Rt) \ __emit_inst(0xd5000000 | sys_insn(0, 3, 3, (CRm), (op2)) | ((Rt) & 0x1f))
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 4/18 arm64: Add ESR_ELx_FSC_ADDRSZ_L() helper
In reply to Marc ZyngierAlthough we have helpers that encode the level of a given fault type, the Address Size fault type is missing it.
While we're at it, fix the bracketting for ESR_ELx_FSC_ACCESS_L() and ESR_ELx_FSC_PERM_L().
--- arch/arm64/include/asm/esr.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/arm64/include/asm/esr.h +3 -2
@@ -122,8 +122,8 @@#define ESR_ELx_FSC_SECC_TTW (n) (0x1c + (n))/* Status codes for individual page table levels */ -#define ESR_ELx_FSC_ACCESS_L (n) (ESR_ELx_FSC_ACCESS + n) -#define ESR_ELx_FSC_PERM_L (n) (ESR_ELx_FSC_PERM + n) +#define ESR_ELx_FSC_ACCESS_L (n) (ESR_ELx_FSC_ACCESS + (n)) +#define ESR_ELx_FSC_PERM_L (n) (ESR_ELx_FSC_PERM + (n))#define ESR_ELx_FSC_FAULT_nL (0x2C)#define ESR_ELx_FSC_FAULT_L (n) (((n) < 0 ? ESR_ELx_FSC_FAULT_nL : \
@@ -161,6 +161,7 @@/* ISS field definitions for exceptions taken in to Hyp */ #define ESR_ELx_FSC_ADDRSZ (0x00) +#define ESR_ELx_FSC_ADDRSZ_L (n) (ESR_ELx_FSC_ADDRSZ + (n))#define ESR_ELx_CV (UL(1) << 24)#define ESR_ELx_COND_SHIFT (20)#define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT)
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 5/18 KVM: arm64: Make kvm_at() take an OP_AT_*
In reply to Marc ZyngierFrom: Joey Gouly <[email protected]>
To allow using newer instructions that current assemblers don't know about, replace the at instruction with the underlying SYS instruction.
--- arch/arm64/include/asm/kvm_asm.h | 3 ++- arch/arm64/kvm/hyp/include/hyp/fault.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_asm.h +2 -1
@@ -10,6 +10,7 @@#include <asm/hyp_image.h> #include <asm/insn.h> #include <asm/virt.h> +#include <asm/sysreg.h> #define ARM_EXIT_WITH_SERROR_BIT 31#define ARM_EXCEPTION_CODE (x) ((x) & ~(1U << ARM_EXIT_WITH_SERROR_BIT))
@@ -259,7 +260,7 @@ extern u64 __kvm_get_mdcr_el2(void); asmvolatile ( \" mrs %1, spsr_el2\n" \" mrs %2, elr_el2\n" \ -"1: at " at_op", %3\n" \ + "1: " __msr_s(at_op, "%3") "\n" \" isb\n" \" b 9f\n" \"2: msr spsr_el2, %1\n" \
arch/arm64/kvm/hyp/include/hyp/fault.h +1 -1
@@ -27,7 +27,7 @@ static inline bool __translate_far_to_hpfar(u64 far, u64 *hpfar)* saved the guest context yet, and we may return early... */ par =read_sysreg_par (); -if (!__kvm_at ("s1e1r" , far)) +if (!__kvm_at (OP_AT_S1E1R , far)) tmp =read_sysreg_par ();else tmp =SYS_PAR_EL1_F ;/* back to the guest */
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 6/18 KVM: arm64: nv: Enforce S2 alignment when contiguous bit is set
In reply to Marc ZyngierDespite KVM not using the contiguous bit for anything related to TLBs, the spec does require that the alignment defined by the contiguous bit for the page size and the level is enforced.
Add the required checks to offset the point where PA and VA merge.
--- arch/arm64/include/asm/kvm_nested.h | 22 ++++++++++++++++++++++ arch/arm64/kvm/nested.c | 7 ++----- 2 files changed, 24 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_nested.h +22 -0
@@ -205,4 +205,26 @@ static inline u64 kvm_encode_nested_level(struct kvm_s2_trans *trans) return FIELD_PREP(KVM_NV_GUEST_MAP_SZ, trans->level); } +/* Adjust alignment for the contiguous bit as per StageOA() */ +#define contiguous_bit_shift (d, wi, l) \ + ({ \ + u8 shift = 0; \ + \ + if ((d) & PTE_CONT) { \ + switch (BIT((wi)->pgshift)) { \ + case SZ_4K: \ + shift = 4; \ + break; \ + case SZ_16K: \ + shift = (l) == 2 ? 5 : 7; \ + break; \ + case SZ_64K: \ + shift = 5; \ + break; \ + } \ + } \ + \ + shift; \ + }) +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +2 -5
@@ -282,11 +282,6 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,return 1 ; } -/* -* We don't use the contiguous bit in the stage-2 ptes, so skip check -* for misprogramming of the contiguous bit. -*/ -if (check_output_size (wi, desc)) { out->esr =compute_fsc (level, ESR_ELx_FSC_ADDRSZ); out->upper_attr = desc;
@@ -299,6 +294,8 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,return 1 ; } + addr_bottom +=contiguous_bit_shift (desc, wi, level); +/* Calculate and return the result */ paddr = (desc &GENMASK_ULL (47 , addr_bottom)) | (ipa &GENMASK_ULL (addr_bottom -1 ,0 ));
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 7/18 KVM: arm64: nv: Turn upper_attr for S2 walk into the full descriptor
In reply to Marc ZyngierThe upper_attr attribute has been badly named, as it most of the time carries the full "last walked descriptor".
Rename it to "desc" and make ti contain the full 64bit descriptor. This will be used by the S1 PTW.
--- arch/arm64/include/asm/kvm_nested.h | 4 ++-- arch/arm64/kvm/nested.c | 12 ++++++------ 2 files changed, 8 insertions(+), 8 deletions(-)
arch/arm64/include/asm/kvm_nested.h +2 -2
@@ -85,7 +85,7 @@ struct kvm_s2_trans {bool readable;int level;u32 esr; -u64 upper_attr; +u64 desc; };static inline phys_addr_t kvm_s2_trans_output (struct kvm_s2_trans *trans)
@@ -115,7 +115,7 @@ static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans)static inline bool kvm_s2_trans_executable (struct kvm_s2_trans *trans) { -return !(trans->upper_attr &BIT (54 )); +return !(trans->desc &BIT (54 )); } extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
arch/arm64/kvm/nested.c +6 -6
@@ -256,7 +256,7 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,/* Check for valid descriptor at this point */ if (!(desc &1 ) || ((desc &3 ) ==1 && level ==3 )) { out->esr =compute_fsc (level, ESR_ELx_FSC_FAULT); - out->upper_attr = desc; + out->desc = desc;return 1 ; }
@@ -266,7 +266,7 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,if (check_output_size (wi, desc)) { out->esr =compute_fsc (level, ESR_ELx_FSC_ADDRSZ); - out->upper_attr = desc; + out->desc = desc;return 1 ; }
@@ -278,19 +278,19 @@ static int walk_nested_s2_pgd(phys_addr_t ipa,if (level < first_block_level) { out->esr =compute_fsc (level, ESR_ELx_FSC_FAULT); - out->upper_attr = desc; + out->desc = desc;return 1 ; }if (check_output_size (wi, desc)) { out->esr =compute_fsc (level, ESR_ELx_FSC_ADDRSZ); - out->upper_attr = desc; + out->desc = desc;return 1 ; }if (!(desc &BIT (10 ))) { out->esr =compute_fsc (level, ESR_ELx_FSC_ACCESS); - out->upper_attr = desc; + out->desc = desc;return 1 ; }
@@ -304,7 +304,7 @@ static int walk_nested_s2_pgd(phys_addr_t ipa, out->readable = desc & (0b01 <<6 ); out->writable = desc & (0b10 <<6 ); out->level = level; - out->upper_attr = desc &GENMASK_ULL (63 ,52 ); + out->desc = desc;return 0 ; }
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 8/18 KVM: arm64: nv: Honor absence of FEAT_PAN2
In reply to Marc ZyngierIf our guest has been configured without PAN2, make sure that AT S1E1{R,W}P will generate an UNDEF.
--- arch/arm64/kvm/sys_regs.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/kvm/sys_regs.c +4 -0
@@ -4600,6 +4600,10 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu)HFGITR_EL2_TLBIRVAAE1OS | HFGITR_EL2_TLBIRVAE1OS); +if (!kvm_has_feat (kvm,ID_AA64MMFR1_EL1 ,PAN ,PAN2 )) + kvm->arch.fgu[HFGITR_GROUP ] |= (HFGITR_EL2_ATS1E1RP | +HFGITR_EL2_ATS1E1WP ); +if (!kvm_has_feat (kvm,ID_AA64MMFR3_EL1 ,S1PIE ,IMP )) kvm->arch.fgu[HFGxTR_GROUP] |= (HFGxTR_EL2_nPIRE0_EL1 | HFGxTR_EL2_nPIR_EL1);
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 9/18 KVM: arm64: nv: Add basic emulation of AT S1E{0,1}{R,W}
In reply to Marc ZyngierEmulating AT instructions is one the tasks devolved to the host hypervisor when NV is on.
Here, we take the basic approach of emulating AT S1E{0,1}{R,W} using the AT instructions themselves. While this mostly work, it doesn't *always* work:
- S1 page tables can be swapped out
- shadow S2 can be incomplete and not contain mappings for the S1 page tables
We are not trying to handle these case here, and defer it to a later patch. Suitable comments indicate where we are in dire need of better handling.
--- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/at.c | 140 +++++++++++++++++++++++++++++++ 3 files changed, 142 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/at.c
arch/arm64/include/asm/kvm_asm.h +1 -0
@@ -236,6 +236,7 @@ extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);extern int __kvm_tlbi_s1e2 (struct kvm_s2_mmu *mmu,u64 va,u64 sys_encoding);extern void __kvm_timer_set_cntvoff (u64 cntvoff); +extern void __kvm_at_s1e01 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr);extern int __kvm_vcpu_run (struct kvm_vcpu *vcpu);
arch/arm64/kvm/Makefile +1 -1
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ inject_fault.o va_layout.o handle_exit.o \ guest.o debug.o reset.o sys_regs.o stacktrace.o \ vgic-sys-reg-v3.o fpsimd.o pkvm.o \ - arch_timer.o trng.o vmid.o emulate-nested.o nested.o \ + arch_timer.o trng.o vmid.o emulate-nested.o nested.o at.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/at.c new file +140 -0
140 lines not on this page · raw message
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 10/18 KVM: arm64: nv: Add basic emulation of AT S1E1{R,W}P
In reply to Marc ZyngierBuilding on top of our primitive AT S1E{0,1}{R,W} emulation, add minimal support for the FEAT_PAN2 instructions, momentary context-switching PSTATE.PAN so that it takes effect in the context of the guest.
--- arch/arm64/kvm/at.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
arch/arm64/kvm/at.c +26 -0
@@ -49,6 +49,28 @@ static void __mmu_config_restore(struct mmu_config *config)write_sysreg (config->vtcr, vtcr_el2); } +static bool at_s1e1p_fast (struct kvm_vcpu *vcpu,u32 op,u64 vaddr) +{ +u64 host_pan; +bool fail; + + host_pan =read_sysreg_s (SYS_PSTATE_PAN ); +write_sysreg_s (*vcpu_cpsr (vcpu) &PSTATE_PAN ,SYS_PSTATE_PAN ); + +switch (op) { +case OP_AT_S1E1RP : + fail =__kvm_at (OP_AT_S1E1RP , vaddr); +break ; +case OP_AT_S1E1WP : + fail =__kvm_at (OP_AT_S1E1WP , vaddr); +break ; + } + +write_sysreg_s (host_pan,SYS_PSTATE_PAN ); + +return fail; +} +/* * Return the PAR_EL1 value as the result of a valid translation. *
@@ -105,6 +127,10 @@ static u64 __kvm_at_s1e01_fast(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)isb ();switch (op) { +case OP_AT_S1E1RP : +case OP_AT_S1E1WP : + fail =at_s1e1p_fast (vcpu, op, vaddr); +break ;case OP_AT_S1E1R : fail =__kvm_at (OP_AT_S1E1R , vaddr);break ;
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 11/18 KVM: arm64: nv: Add basic emulation of AT S1E2{R,W}
In reply to Marc ZyngierSimilar to our AT S1E{0,1} emulation, we implement the AT S1E2 handling.
This emulation of course suffers from the same problems, but is somehow simpler due to the lack of PAN2 and the fact that we are guaranteed to execute it from the correct context.
--- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/at.c | 51 ++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)
arch/arm64/include/asm/kvm_asm.h +1 -0
@@ -237,6 +237,7 @@ extern int __kvm_tlbi_s1e2(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding);extern void __kvm_timer_set_cntvoff (u64 cntvoff);extern void __kvm_at_s1e01 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr); +extern void __kvm_at_s1e2 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr);extern int __kvm_vcpu_run (struct kvm_vcpu *vcpu);
arch/arm64/kvm/at.c +51 -0
54 lines not on this page · raw message
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 12/18 KVM: arm64: nv: Add emulation of AT S12E{0,1}{R,W}
In reply to Marc ZyngierOn the face of it, AT S12E{0,1}{R,W} is pretty simple. It is the combination of AT S1E{0,1}{R,W}, followed by an extra S2 walk.
However, there is a great deal of complexity coming from combining the S1 and S2 attributes to report something consistent in PAR_EL1.
This is an absolute mine field, and I have a splitting headache.
--- arch/arm64/include/asm/kvm_asm.h | 1 + arch/arm64/kvm/at.c | 253 +++++++++++++++++++++++++++++++ 2 files changed, 254 insertions(+)
arch/arm64/include/asm/kvm_asm.h +1 -0
@@ -238,6 +238,7 @@ extern int __kvm_tlbi_s1e2(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding);extern void __kvm_timer_set_cntvoff (u64 cntvoff);extern void __kvm_at_s1e01 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr);extern void __kvm_at_s1e2 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr); +extern void __kvm_at_s12 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr);extern int __kvm_vcpu_run (struct kvm_vcpu *vcpu);
arch/arm64/kvm/at.c +253 -0
262 lines not on this page · raw message
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 13/18 KVM: arm64: nv: Make ps_to_output_size() generally available
In reply to Marc ZyngierMake this helper visible to at.c, we are going to need it.
--- arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++ arch/arm64/kvm/nested.c | 14 -------------- 2 files changed, 14 insertions(+), 14 deletions(-)
arch/arm64/include/asm/kvm_nested.h +14 -0
@@ -227,4 +227,18 @@ static inline u64 kvm_encode_nested_level(struct kvm_s2_trans *trans) shift; \ }) +static inline unsigned int ps_to_output_size (unsigned int ps) +{ +switch (ps) { +case 0 :return 32 ; +case 1 :return 36 ; +case 2 :return 40 ; +case 3 :return 42 ; +case 4 :return 44 ; +case 5 : +default : +return 48 ; + } +} +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +0 -14
@@ -103,20 +103,6 @@ struct s2_walk_info {bool be; }; -static unsigned int ps_to_output_size (unsigned int ps) -{ -switch (ps) { -case 0 :return 32 ; -case 1 :return 36 ; -case 2 :return 40 ; -case 3 :return 42 ; -case 4 :return 44 ; -case 5 : -default : -return 48 ; - } -} -static u32 compute_fsc (int level,u32 fsc) {return fsc | (level &0x3 );
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 14/18 KVM: arm64: nv: Add SW walker for AT S1 emulation
In reply to Marc ZyngierIn order to plug the brokenness of our current AT implementation, we need a SW walker that is going to... err.. walk the S1 tables and tell us what it finds.
Of course, it builds on top of our S2 walker, and share similar concepts. The beauty of it is that since it uses kvm_read_guest(), it is able to bring back pages that have been otherwise evicted.
This is then plugged in the two AT S1 emulation functions as a "slow path" fallback. I'm not sure it is that slow, but hey.
--- arch/arm64/kvm/at.c | 610 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 608 insertions(+), 2 deletions(-)
arch/arm64/kvm/at.c +608 -2
@@ -4,9 +4,408 @@* Author: Jintack Lim <[email protected]> */ +#include <linux/kvm_host.h> + +#include <asm/esr.h> #include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> +enum trans_regime { +TR_EL10 , +TR_EL20 , +TR_EL2 , +}; + +struct s1_walk_info { +u64 baddr; +enum trans_regime regime; +unsigned int max_oa_bits; +unsigned int pgshift; +unsigned int txsz; +int sl; +bool hpd; +bool be; +bool s2; +}; + +struct s1_walk_result { +union { +struct { +u64 desc; +u64 pa; +s8 level; +u8 APTable; +bool UXNTable; +bool PXNTable; + }; +struct { +u8 fst; +bool ptw; +bool s2; + }; + }; +bool failed; +}; + +static void fail_s1_walk (struct s1_walk_result *wr,u8 fst,bool ptw,bool s2) +{ + wr->fst = fst; + wr->ptw = ptw; + wr->s2 = s2; + wr->failed = true; +} + +#define S1_MMU_DISABLED (-127) + +static int get_ia_size (struct s1_walk_info *wi) +{ +return 64 - wi->txsz; +} + +/* Return true if the IPA is out of the OA range */ +static bool check_output_size (u64 ipa,struct s1_walk_info *wi) +{ +return wi->max_oa_bits <48 && (ipa &GENMASK_ULL (47 , wi->max_oa_bits)); +} + +/* Return the translation regime that applies to an AT instruction */ +static enum trans_regime compute_translation_regime (struct kvm_vcpu *vcpu,u32 op) +{ +/* +* We only get here from guest EL2, so the translation +* regime AT applies to is solely defined by {E2H,TGE}. +*/ +switch (op) { +case OP_AT_S1E2R : +case OP_AT_S1E2W : +return vcpu_el2_e2h_is_set (vcpu) ?TR_EL20 :TR_EL2 ; +break ; +default : +return (vcpu_el2_e2h_is_set (vcpu) && +vcpu_el2_tge_is_set (vcpu)) ?TR_EL20 :TR_EL10 ; + } +} + +static int setup_s1_walk (struct kvm_vcpu *vcpu,u32 op,struct s1_walk_info *wi, +struct s1_walk_result *wr,u64 va) +{ +u64 hcr, sctlr, tcr, tg, ps, ia_bits, ttbr; +unsigned int stride, x; +bool va55, tbi, lva, as_el0; + + hcr =__vcpu_sys_reg (vcpu,HCR_EL2 ); + + wi->regime =compute_translation_regime (vcpu, op); + as_el0 = (op ==OP_AT_S1E0R || op ==OP_AT_S1E0W ); + + va55 = va &BIT (55 ); + +if (wi->regime ==TR_EL2 && va55) + goto addrsz; + + wi->s2 = wi->regime ==TR_EL10 && (hcr & (HCR_VM |HCR_DC )); + +switch (wi->regime) { +case TR_EL10 : + sctlr =vcpu_read_sys_reg (vcpu,SCTLR_EL1 ); + tcr =vcpu_read_sys_reg (vcpu,TCR_EL1 ); + ttbr = (va55 ? +vcpu_read_sys_reg (vcpu,TTBR1_EL1 ) : +vcpu_read_sys_reg (vcpu,TTBR0_EL1 )); +break ; +case TR_EL2 : +case TR_EL20 : + sctlr =vcpu_read_sys_reg (vcpu,SCTLR_EL2 ); + tcr =vcpu_read_sys_reg (vcpu,TCR_EL2 ); + ttbr = (va55 ? +vcpu_read_sys_reg (vcpu,TTBR1_EL2 ) : +vcpu_read_sys_reg (vcpu,TTBR0_EL2 )); +break ; +default : +BUG (); + } + + tbi = (wi->regime ==TR_EL2 ? +FIELD_GET (TCR_EL2_TBI , tcr) : + (va55 ? +FIELD_GET (TCR_TBI1 , tcr) : +FIELD_GET (TCR_TBI0 , tcr))); + +if (!tbi && (u64 )sign_extend64 (va,55 ) != va) + goto addrsz; + + va = (u64 )sign_extend64 (va,55 ); + +/* Let's put the MMU disabled case aside immediately */ +switch (wi->regime) { +case TR_EL10 : +/* +* If dealing with the EL1&0 translation regime, 3 things +* can disable the S1 translation: +* +* - HCR_EL2.DC = 1 +* - HCR_EL2.{E2H,TGE} = {0,1} +* - SCTLR_EL1.M = 0 +* +* The TGE part is interesting. If we have decided that this +* is EL1&0, then it means that either {E2H,TGE} == {1,0} or +* {0,x}, and we only need to test for TGE == 1. +*/ +if (hcr & (HCR_DC |HCR_TGE )) { + wr->level =S1_MMU_DISABLED ; +break ; + } + fallthrough; +case TR_EL2 : +case TR_EL20 : +if (!(sctlr & SCTLR_ELx_M)) + wr->level =S1_MMU_DISABLED ; +break ; + } + +if (wr->level ==S1_MMU_DISABLED ) { +if (va >=BIT (kvm_get_pa_bits (vcpu->kvm))) + goto addrsz; + + wr->pa = va; +return 0 ; + } + + wi->be = sctlr & SCTLR_ELx_EE; + + wi->hpd =kvm_has_feat (vcpu->kvm,ID_AA64MMFR1_EL1 ,HPDS ,IMP ); + wi->hpd &= (wi->regime ==TR_EL2 ? +FIELD_GET (TCR_EL2_HPD , tcr) : + (va55 ? +FIELD_GET (TCR_HPD1 , tcr) : +FIELD_GET (TCR_HPD0 , tcr))); + +/* Someone was silly enough to encode TG0/TG1 differently */ +if (va55) { + wi->txsz =FIELD_GET (TCR_T1SZ_MASK , tcr); + tg =FIELD_GET (TCR_TG1_MASK , tcr); + +switch (tg <<TCR_TG1_SHIFT ) { +case TCR_TG1_4K : + wi->pgshift =12 ;break ; +case TCR_TG1_16K : + wi->pgshift =14 ;break ; +case TCR_TG1_64K : +default :/* IMPDEF: treat any other value as 64k */ + wi->pgshift =16 ;break ; + } + }else { + wi->txsz =FIELD_GET (TCR_T0SZ_MASK , tcr); + tg =FIELD_GET (TCR_TG0_MASK , tcr); + +switch (tg <<TCR_TG0_SHIFT ) { +case TCR_TG0_4K : + wi->pgshift =12 ;break ; +case TCR_TG0_16K : + wi->pgshift =14 ;break ; +case TCR_TG0_64K : +default :/* IMPDEF: treat any other value as 64k */ + wi->pgshift =16 ;break ; + } + } + +/* R_PLCGL, R_YXNYW */ + if (!kvm_has_feat_enum(vcpu->kvm, ID_AA64MMFR2_EL1, ST, 48_47)) { +if (wi->txsz >39 ) + goto transfault_l0; + }else { +if (wi->txsz >48 || (BIT (wi->pgshift) ==SZ_64K && wi->txsz >47 )) + goto transfault_l0; + } + +/* R_GTJBY, R_SXWGM */ +switch (BIT (wi->pgshift)) { +case SZ_4K : + lva = kvm_has_feat(vcpu->kvm, ID_AA64MMFR0_EL1, TGRAN4, 52_BIT); + lva &= tcr & (wi->regime ==TR_EL2 ?TCR_EL2_DS :TCR_DS ); +break ; +case SZ_16K : + lva = kvm_has_feat(vcpu->kvm, ID_AA64MMFR0_EL1, TGRAN16, 52_BIT); + lva &= tcr & (wi->regime ==TR_EL2 ?TCR_EL2_DS :TCR_DS ); +break ; +case SZ_64K : + lva =kvm_has_feat (vcpu->kvm,ID_AA64MMFR2_EL1 , VARange,52 ); +break ; + } + +if ((lva && wi->txsz <12 ) || (!lva && wi->txsz <16 )) + goto transfault_l0; + + ia_bits =get_ia_size (wi); + +/* R_YYVYV, I_THCZK */ +if ((!va55 && va >GENMASK (ia_bits -1 ,0 )) || + (va55 && va <GENMASK (63 , ia_bits))) + goto transfault_l0; + +/* I_ZFSYQ */ +if (wi->regime !=TR_EL2 && + (tcr & (va55 ?TCR_EPD1_MASK :TCR_EPD0_MASK ))) + goto transfault_l0; + +/* R_BNDVG and following statements */ +if (kvm_has_feat (vcpu->kvm,ID_AA64MMFR2_EL1 ,E0PD ,IMP ) && + as_el0 && (tcr & (va55 ?TCR_E0PD1 :TCR_E0PD0 ))) + goto transfault_l0; + +/* AArch64.S1StartLevel() */ + stride = wi->pgshift -3 ; + wi->sl =3 - (((ia_bits -1 ) - wi->pgshift) / stride); + + ps = (wi->regime ==TR_EL2 ? +FIELD_GET (TCR_EL2_PS_MASK , tcr) :FIELD_GET (TCR_IPS_MASK , tcr)); + + wi->max_oa_bits =min (get_kvm_ipa_limit (),ps_to_output_size (ps)); + +/* Compute minimal alignment */ + x =3 + ia_bits - ((3 - wi->sl) * stride + wi->pgshift); + + wi->baddr = ttbr & TTBRx_EL1_BADDR; + +/* R_VPBBF */ +if (check_output_size (wi->baddr, wi)) + goto addrsz; + + wi->baddr &=GENMASK_ULL (wi->max_oa_bits -1 , x); + +return 0 ; + +addrsz:/* Address Size Fault level 0 */ +fail_s1_walk (wr,ESR_ELx_FSC_ADDRSZ_L (0 ), false, false); +return -EFAULT ; + +transfault_l0:/* Translation Fault level 0 */ +fail_s1_walk (wr,ESR_ELx_FSC_FAULT_L (0 ), false, false); +return -EFAULT ; +} + +static int walk_s1 (struct kvm_vcpu *vcpu,struct s1_walk_info *wi, +struct s1_walk_result *wr,u64 va) +{ +u64 va_top, va_bottom, baddr, desc; +int level, stride, ret; + + level = wi->sl; + stride = wi->pgshift -3 ; + baddr = wi->baddr; + + va_top =get_ia_size (wi) -1 ; + +while (1 ) { +u64 index, ipa; + + va_bottom = (3 - level) * stride + wi->pgshift; + index = (va &GENMASK_ULL (va_top, va_bottom)) >> (va_bottom -3 ); + + ipa = baddr | index; + +if (wi->s2) { +struct kvm_s2_trans s2_trans = {}; + + ret =kvm_walk_nested_s2 (vcpu, ipa, &s2_trans); +if (ret) { +fail_s1_walk (wr, + (s2_trans.esr & ~ESR_ELx_FSC_LEVEL) | level, + true, true); +return ret; + } + +if (!kvm_s2_trans_readable (&s2_trans)) { +fail_s1_walk (wr,ESR_ELx_FSC_PERM_L (level), + true, true); + +return -EPERM ; + } + + ipa =kvm_s2_trans_output (&s2_trans); + } + + ret =kvm_read_guest (vcpu->kvm, ipa, &desc,sizeof (desc)); +if (ret) { +fail_s1_walk (wr,ESR_ELx_FSC_SEA_TTW (level), + true, false); +return ret; + } + +if (wi->be) + desc =be64_to_cpu ((__force __be64 )desc); +else + desc =le64_to_cpu ((__force __le64 )desc); + +/* Invalid descriptor */ +if (!(desc &BIT (0 ))) + goto transfault; + +/* Block mapping, check validity down the line */ +if (!(desc &BIT (1 ))) +break ; + +/* Page mapping */ +if (level ==3 ) +break ; + +/* Table handling */ +if (!wi->hpd) { + wr->APTable |=FIELD_GET (S1_TABLE_AP , desc); + wr->UXNTable |=FIELD_GET (PMD_TABLE_UXN , desc); + wr->PXNTable |=FIELD_GET (PMD_TABLE_PXN , desc); + } + + baddr = desc &GENMASK_ULL (47 , wi->pgshift); + +/* Check for out-of-range OA */ +if (check_output_size (baddr, wi)) + goto addrsz; + +/* Prepare for next round */ + va_top = va_bottom -1 ; + level++; + } + +/* Block mapping, check the validity of the level */ +if (!(desc &BIT (1 ))) { +bool valid_block = false; + +switch (BIT (wi->pgshift)) { +case SZ_4K : + valid_block = level ==1 || level ==2 ; +break ; +case SZ_16K : +case SZ_64K : + valid_block = level ==2 ; +break ; + } + +if (!valid_block) + goto transfault; + } + +if (check_output_size (desc &GENMASK (47 , va_bottom), wi)) + goto addrsz; + + va_bottom +=contiguous_bit_shift (desc, wi, level); + + wr->failed = false; + wr->level = level; + wr->desc = desc; + wr->pa = desc &GENMASK (47 , va_bottom); + wr->pa |= va &GENMASK_ULL (va_bottom -1 ,0 ); + +return 0 ; + +addrsz: +fail_s1_walk (wr,ESR_ELx_FSC_ADDRSZ_L (level), true, false); +return -EINVAL ; +transfault: +fail_s1_walk (wr,ESR_ELx_FSC_FAULT_L (level), true, false); +return -ENOENT ; +} +struct mmu_config {u64 ttbr0;u64 ttbr1;
@@ -188,6 +587,16 @@ static u8 compute_sh(u8 attr, u64 desc)return sh; } +static u8 combine_sh (u8 s1_sh,u8 s2_sh) +{ +if (s1_sh ==ATTR_OSH || s2_sh ==ATTR_OSH ) +return ATTR_OSH ; +if (s1_sh ==ATTR_ISH || s2_sh ==ATTR_ISH ) +return ATTR_ISH ; + +return ATTR_NSH ; +} +static u64 compute_par_s12 (struct kvm_vcpu *vcpu,u64 s1_par,struct kvm_s2_trans *tr) {
@@ -260,11 +669,185 @@ static u64 compute_par_s12(struct kvm_vcpu *vcpu, u64 s1_par, par =FIELD_PREP (SYS_PAR_EL1_ATTR , final_attr); par |= tr->output &GENMASK (47 ,12 ); par |=FIELD_PREP (SYS_PAR_EL1_SH , -compute_sh (final_attr, tr->desc)); +combine_sh (FIELD_GET (SYS_PAR_EL1_SH , s1_par), +compute_sh (final_attr, tr->desc))); + +return par; +} + +static u64 compute_par_s1 (struct kvm_vcpu *vcpu,struct s1_walk_result *wr, +enum trans_regime regime) +{ +u64 par; + +if (wr->failed) { + par =SYS_PAR_EL1_RES1 ; + par |=SYS_PAR_EL1_F ; + par |=FIELD_PREP (SYS_PAR_EL1_FST , wr->fst); + par |= wr->ptw ?SYS_PAR_EL1_PTW :0 ; + par |= wr->s2 ?SYS_PAR_EL1_S :0 ; + }else if (wr->level ==S1_MMU_DISABLED ) { +/* MMU off or HCR_EL2.DC == 1 */ + par =SYS_PAR_EL1_NSE ; + par |= wr->pa &GENMASK_ULL (47 ,12 ); + +if (regime ==TR_EL10 && + (__vcpu_sys_reg (vcpu,HCR_EL2 ) &HCR_DC )) { + par |=FIELD_PREP (SYS_PAR_EL1_ATTR , +MEMATTR (WbRaWa, WbRaWa)); + par |=FIELD_PREP (SYS_PAR_EL1_SH ,ATTR_NSH ); + }else { + par |=FIELD_PREP (SYS_PAR_EL1_ATTR ,0 );/* nGnRnE */ + par |=FIELD_PREP (SYS_PAR_EL1_SH ,ATTR_OSH ); + } + }else { +u64 mair, sctlr; +u8 sh; + + par =SYS_PAR_EL1_NSE ; + + mair = (regime ==TR_EL10 ? +vcpu_read_sys_reg (vcpu,MAIR_EL1 ) : +vcpu_read_sys_reg (vcpu,MAIR_EL2 )); + + mair >>=FIELD_GET (PTE_ATTRINDX_MASK , wr->desc) *8 ; + mair &=0xff ; + + sctlr = (regime ==TR_EL10 ? +vcpu_read_sys_reg (vcpu,SCTLR_EL1 ) : +vcpu_read_sys_reg (vcpu,SCTLR_EL2 )); + +/* Force NC for memory if SCTLR_ELx.C is clear */ +if (!(sctlr &SCTLR_EL1_C ) && !MEMATTR_IS_DEVICE (mair)) + mair =MEMATTR (NC ,NC ); + + par |=FIELD_PREP (SYS_PAR_EL1_ATTR , mair); + par |= wr->pa &GENMASK_ULL (47 ,12 ); + + sh =compute_sh (mair, wr->desc); + par |=FIELD_PREP (SYS_PAR_EL1_SH , sh); + }return par; } +static u64 handle_at_slow (struct kvm_vcpu *vcpu,u32 op,u64 vaddr) +{ +bool perm_fail, ur, uw, ux, pr, pw, px; +struct s1_walk_result wr = {}; +struct s1_walk_info wi = {}; +int ret, idx; + + ret =setup_s1_walk (vcpu, op, &wi, &wr, vaddr); +if (ret) + goto compute_par; + +if (wr.level ==S1_MMU_DISABLED ) + goto compute_par; + + idx =srcu_read_lock (&vcpu->kvm->srcu); + + ret =walk_s1 (vcpu, &wi, &wr, vaddr); + +srcu_read_unlock (&vcpu->kvm->srcu, idx); + +if (ret) + goto compute_par; + +/* FIXME: revisit when adding indirect permission support */ +/* AArch64.S1DirectBasePermissions() */ +if (wi.regime !=TR_EL2 ) { +switch (FIELD_GET (PTE_USER |PTE_RDONLY , wr.desc)) { +case 0b00 : + pr = pw = true; + ur = uw = false; +break ; +case 0b01 : + pr = pw = ur = uw = true; +break ; +case 0b10 : + pr = true; + pw = ur = uw = false; +break ; +case 0b11 : + pr = ur = true; + pw = uw = false; +break ; + } + +switch (wr.APTable) { +case 0b00 : +break ; +case 0b01 : + ur = uw = false; +break ; +case 0b10 : + pw = uw = false; +break ; +case 0b11 : + pw = ur = uw = false; +break ; + } + +/* We don't use px for anything yet, but hey... */ + px = !((wr.desc &PTE_PXN ) || wr.PXNTable || uw); + ux = !((wr.desc &PTE_UXN ) || wr.UXNTable); + +if (op ==OP_AT_S1E1RP || op ==OP_AT_S1E1WP ) { +bool pan; + + pan = *vcpu_cpsr (vcpu) &PSR_PAN_BIT ; + pan &= ur || uw; + pw &= !pan; + pr &= !pan; + } + }else { + ur = uw = ux = false; + +if (!(wr.desc &PTE_RDONLY )) { + pr = pw = true; + }else { + pr = true; + pw = false; + } + +if (wr.APTable &BIT (1 )) + pw = false; + +/* XN maps to UXN */ + px = !((wr.desc &PTE_UXN ) || wr.UXNTable); + } + + perm_fail = false; + +switch (op) { +case OP_AT_S1E1RP : +case OP_AT_S1E1R : +case OP_AT_S1E2R : + perm_fail = !pr; +break ; +case OP_AT_S1E1WP : +case OP_AT_S1E1W : +case OP_AT_S1E2W : + perm_fail = !pw; +break ; +case OP_AT_S1E0R : + perm_fail = !ur; +break ; +case OP_AT_S1E0W : + perm_fail = !uw; +break ; +default : +BUG (); + } + +if (perm_fail) +fail_s1_walk (&wr,ESR_ELx_FSC_PERM_L (wr.level), false, false); + +compute_par: +return compute_par_s1 (vcpu, &wr, wi.regime); +} +/* * Return the PAR_EL1 value as the result of a valid translation. *
@@ -352,10 +935,29 @@ static u64 __kvm_at_s1e01_fast(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)return par; } +static bool par_check_s1_perm_fault (u64 par) +{ +u8 fst =FIELD_GET (SYS_PAR_EL1_FST , par); + +return ((fst & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_PERM && + !(par &SYS_PAR_EL1_S )); +} +void __kvm_at_s1e01 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr) {u64 par =__kvm_at_s1e01_fast (vcpu, op, vaddr); +/* +* If PAR_EL1 reports that AT failed on a S1 permission fault, we +* know for sure that the PTW was able to walk the S1 tables and +* there's nothing else to do. +* +* If AT failed for any other reason, then we must walk the guest S1 +* to emulate the instruction. +*/ +if ((par &SYS_PAR_EL1_F ) && !par_check_s1_perm_fault (par)) + par =handle_at_slow (vcpu, op, vaddr); +vcpu_write_sys_reg (vcpu, par,PAR_EL1 ); }
@@ -407,6 +1009,10 @@ void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)isb (); } +/* We failed the translation, let's replay it in slow motion */ +if ((par &SYS_PAR_EL1_F ) && !par_check_s1_perm_fault (par)) + par =handle_at_slow (vcpu, op, vaddr); +vcpu_write_sys_reg (vcpu, par,PAR_EL1 ); }
@@ -463,7 +1069,7 @@ void __kvm_at_s12(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)/* Check the access permission */ if (!out.esr && ((!write && !out.readable) || (write && !out.writable))) - out.esr = ESR_ELx_FSC_PERM | (out.level &0x3 ); + out.esr =ESR_ELx_FSC_PERM_L (out.level &0x3 ); par =compute_par_s12 (vcpu, par, &out);vcpu_write_sys_reg (vcpu, par,PAR_EL1 );
Signature · 1 line
2.39.2
kernel test robot
build botRe: [PATCH v4 14/18] KVM: arm64: nv: Add SW walker for AT S1 emulation
kernel test robot noticed the following build warnings:
Hi Marc,
kernel test robot noticed the following build warnings:
[auto build test WARNING on kvmarm/next]
[also build test WARNING on arm64/for-next/core soc/for-next linus/master v6.11-rc4 next-20240821] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Marc-Zyngier/arm64-Add-missing-APTable-and-TCR_ELx-HPD-masks/20240820-184117 base: https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next patch link: https://lore.kernel.org/r/20240820103756.3545976-15-maz%40kernel.org patch subject: [PATCH v4 14/18] KVM: arm64: nv: Add SW walker for AT S1 emulation config: arm64-defconfig (https://download.01.org/0day-ci/archive/20240822/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 14.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240822/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
arch/arm64/kvm/at.c: In function 'handle_at_slow':
>> arch/arm64/kvm/at.c:736:45: warning: variable 'px' set but not used [-Wunused-but-set-variable]
736 | bool perm_fail, ur, uw, ux, pr, pw, px;
| ^~
arch/arm64/kvm/at.c: In function '__kvm_at_s1e2':
arch/arm64/kvm/at.c:973:36: warning: variable 'mmu' set but not used [-Wunused-but-set-variable]
973 | struct kvm_s2_mmu *mmu;
| ^~~vim +/px +736 arch/arm64/kvm/at.c
733
734 static u64 handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)
735 {
> 736 bool perm_fail, ur, uw, ux, pr, pw, px;
737 struct s1_walk_result wr = {};
738 struct s1_walk_info wi = {};
739 int ret, idx;
740
741 ret = setup_s1_walk(vcpu, op, &wi, &wr, vaddr);
742 if (ret)
743 goto compute_par;
744
745 if (wr.level == S1_MMU_DISABLED)
746 goto compute_par;
747
748 idx = srcu_read_lock(&vcpu->kvm->srcu);
749
750 ret = walk_s1(vcpu, &wi, &wr, vaddr);
751
752 srcu_read_unlock(&vcpu->kvm->srcu, idx);
753
754 if (ret)
755 goto compute_par;
756
757 /* FIXME: revisit when adding indirect permission support */
758 /* AArch64.S1DirectBasePermissions() */
759 if (wi.regime != TR_EL2) {
760 switch (FIELD_GET(PTE_USER | PTE_RDONLY, wr.desc)) {
761 case 0b00:
762 pr = pw = true;
763 ur = uw = false;
764 break;
765 case 0b01:
766 pr = pw = ur = uw = true;
767 break;
768 case 0b10:
769 pr = true;
770 pw = ur = uw = false;
771 break;
772 case 0b11:
773 pr = ur = true;
774 pw = uw = false;
775 break;
776 }
777
778 switch (wr.APTable) {
779 case 0b00:
780 break;
781 case 0b01:
782 ur = uw = false;
783 break;
784 case 0b10:
785 pw = uw = false;
786 break;
787 case 0b11:
788 pw = ur = uw = false;
789 break;
790 }
791
792 /* We don't use px for anything yet, but hey... */
793 px = !((wr.desc & PTE_PXN) || wr.PXNTable || uw);
794 ux = !((wr.desc & PTE_UXN) || wr.UXNTable); 795
796 if (op == OP_AT_S1E1RP || op == OP_AT_S1E1WP) {
797 bool pan;
798
799 pan = *vcpu_cpsr(vcpu) & PSR_PAN_BIT;
800 pan &= ur || uw;
801 pw &= !pan;
802 pr &= !pan;
803 }
804 } else {
805 ur = uw = ux = false;
806
807 if (!(wr.desc & PTE_RDONLY)) {
808 pr = pw = true;
809 } else {
810 pr = true;
811 pw = false;
812 }
813
814 if (wr.APTable & BIT(1))
815 pw = false;
816
817 /* XN maps to UXN */
818 px = !((wr.desc & PTE_UXN) || wr.UXNTable);
819 }
820
821 perm_fail = false;
822
823 switch (op) {
824 case OP_AT_S1E1RP:
825 case OP_AT_S1E1R:
826 case OP_AT_S1E2R:
827 perm_fail = !pr;
828 break;
829 case OP_AT_S1E1WP:
830 case OP_AT_S1E1W:
831 case OP_AT_S1E2W:
832 perm_fail = !pw;
833 break;
834 case OP_AT_S1E0R:
835 perm_fail = !ur;
836 break;
837 case OP_AT_S1E0W:
838 perm_fail = !uw;
839 break;
840 default:
841 BUG();
842 }
843
844 if (perm_fail)
845 fail_s1_walk(&wr, ESR_ELx_FSC_PERM_L(wr.level), false, false);
846
847 compute_par:
848 return compute_par_s1(vcpu, &wr, wi.regime);
849 }
850
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Marc Zyngier
maintainerPATCH v4 15/18 KVM: arm64: nv: Sanitise SCTLR_EL1.EPAN according to VM configuration
In reply to Marc ZyngierEnsure that SCTLR_EL1.EPAN is RES0 when FEAT_PAN3 isn't supported.
--- arch/arm64/kvm/nested.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/arm64/kvm/nested.c +8 -0
@@ -1178,6 +1178,14 @@ int kvm_init_nv_sysregs(struct kvm *kvm)if (!kvm_has_feat (kvm,ID_AA64PFR0_EL1 ,AMU ,V1P1 )) res0 |= ~(res0 | res1);set_sysreg_masks (kvm,HAFGRTR_EL2 , res0, res1); + +/* SCTLR_EL1 */ + res0 =SCTLR_EL1_RES0 ; + res1 =SCTLR_EL1_RES1 ; +if (!kvm_has_feat (kvm,ID_AA64MMFR1_EL1 ,PAN ,PAN3 )) + res0 |=SCTLR_EL1_EPAN ; +set_sysreg_masks (kvm,SCTLR_EL1 , res0, res1); + out:mutex_unlock (&kvm->arch.config_lock);
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 16/18 KVM: arm64: nv: Make AT+PAN instructions aware of FEAT_PAN3
In reply to Marc ZyngierFEAT_PAN3 added a check for executable permissions to FEAT_PAN2. Add the required SCTLR_ELx.EPAN and descriptor checks to handle this correctly.
--- arch/arm64/kvm/at.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
arch/arm64/kvm/at.c +16 -1
@@ -731,6 +731,21 @@ static u64 compute_par_s1(struct kvm_vcpu *vcpu, struct s1_walk_result *wr,return par; } +static bool pan3_enabled (struct kvm_vcpu *vcpu,enum trans_regime regime) +{ +u64 sctlr; + +if (!kvm_has_feat (vcpu->kvm,ID_AA64MMFR1_EL1 ,PAN ,PAN3 )) +return false; + +if (regime ==TR_EL10 ) + sctlr =vcpu_read_sys_reg (vcpu,SCTLR_EL1 ); +else + sctlr =vcpu_read_sys_reg (vcpu,SCTLR_EL2 ); + +return sctlr &SCTLR_EL1_EPAN ; +} +static u64 handle_at_slow (struct kvm_vcpu *vcpu,u32 op,u64 vaddr) {bool perm_fail, ur, uw, ux, pr, pw, px;
@@ -797,7 +812,7 @@ static u64 handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)bool pan; pan = *vcpu_cpsr (vcpu) &PSR_PAN_BIT ; - pan &= ur || uw; + pan &= ur || uw || (pan3_enabled (vcpu, wi.regime) && ux); pw &= !pan; pr &= !pan; }
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 17/18 KVM: arm64: nv: Plumb handling of AT S1* traps from EL2
In reply to Marc ZyngierHooray, we're done. Plug the AT traps into the system instruction table, and let it rip.
--- arch/arm64/kvm/sys_regs.c | 45 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+)
arch/arm64/kvm/sys_regs.c +45 -0
@@ -2803,6 +2803,36 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG(SP_EL2, NULL, reset_unknown, 0), }; +static bool handle_at_s1e01 (struct kvm_vcpu *vcpu,struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +u32 op =sys_insn (p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + +__kvm_at_s1e01 (vcpu, op, p->regval); + +return true; +} + +static bool handle_at_s1e2 (struct kvm_vcpu *vcpu,struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +u32 op =sys_insn (p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + +__kvm_at_s1e2 (vcpu, op, p->regval); + +return true; +} + +static bool handle_at_s12 (struct kvm_vcpu *vcpu,struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +u32 op =sys_insn (p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + +__kvm_at_s12 (vcpu, op, p->regval); + +return true; +} +static bool kvm_supported_tlbi_s12_op (struct kvm_vcpu *vpcu,u32 instr) {struct kvm *kvm = vpcu->kvm;
@@ -3065,6 +3095,14 @@ static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_ISW), access_dcsw }, { SYS_DESC(SYS_DC_IGSW), access_dcgsw }, { SYS_DESC(SYS_DC_IGDSW), access_dcgsw }, + +SYS_INSN (AT_S1E1R , handle_at_s1e01), +SYS_INSN (AT_S1E1W , handle_at_s1e01), +SYS_INSN (AT_S1E0R , handle_at_s1e01), +SYS_INSN (AT_S1E0W , handle_at_s1e01), +SYS_INSN (AT_S1E1RP , handle_at_s1e01), +SYS_INSN (AT_S1E1WP , handle_at_s1e01), + { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CGSW), access_dcgsw }, { SYS_DESC(SYS_DC_CGDSW), access_dcgsw },
@@ -3144,6 +3182,13 @@ static struct sys_reg_desc sys_insn_descs[] = {SYS_INSN (TLBI_VALE1NXS , handle_tlbi_el1),SYS_INSN (TLBI_VAALE1NXS , handle_tlbi_el1), +SYS_INSN (AT_S1E2R , handle_at_s1e2), +SYS_INSN (AT_S1E2W , handle_at_s1e2), +SYS_INSN (AT_S12E1R , handle_at_s12), +SYS_INSN (AT_S12E1W , handle_at_s12), +SYS_INSN (AT_S12E0R , handle_at_s12), +SYS_INSN (AT_S12E0W , handle_at_s12), +SYS_INSN (TLBI_IPAS2E1IS , handle_ipas2e1is),SYS_INSN (TLBI_RIPAS2E1IS , handle_ripas2e1is), SYS_INSN(TLBI_IPAS2LE1IS, handle_ipas2e1is),
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 18/18 KVM: arm64: nv: Add support for FEAT_ATS1A
In reply to Marc ZyngierHandling FEAT_ATS1A (which provides the AT S1E{1,2}A instructions) is pretty easy, as it is just the usual AT without the permission check.
This basically amounts to plumbing the instructions in the various dispatch tables, and handling FEAT_ATS1A being disabled in the ID registers.
--- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/at.c | 10 ++++++++++ arch/arm64/kvm/emulate-nested.c | 2 ++ arch/arm64/kvm/sys_regs.c | 11 +++++++++++ 4 files changed, 24 insertions(+)
arch/arm64/include/asm/sysreg.h +1 -0
@@ -673,6 +673,7 @@#define OP_AT_S12E1W sys_insn(AT_Op0, 4, AT_CRn, 8, 5)#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6)#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) +#define OP_AT_S1E2A sys_insn(AT_Op0, 4, AT_CRn, 9, 2)/* TLBI instructions */ #define TLBI_Op0 1
arch/arm64/kvm/at.c +10 -0
@@ -78,6 +78,7 @@ static enum trans_regime compute_translation_regime(struct kvm_vcpu *vcpu, u32 oswitch (op) {case OP_AT_S1E2R :case OP_AT_S1E2W : +case OP_AT_S1E2A :return vcpu_el2_e2h_is_set (vcpu) ?TR_EL20 :TR_EL2 ;break ;default :
@@ -852,6 +853,9 @@ static u64 handle_at_slow(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)case OP_AT_S1E0W : perm_fail = !uw;break ; +case OP_AT_S1E1A : +case OP_AT_S1E2A : +break ;default :BUG (); }
@@ -935,6 +939,9 @@ static u64 __kvm_at_s1e01_fast(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)case OP_AT_S1E0W : fail =__kvm_at (OP_AT_S1E0W , vaddr);break ; +case OP_AT_S1E1A : + fail =__kvm_at (OP_AT_S1E1A , vaddr); +break ;default :WARN_ON_ONCE (1 ); fail = true;
@@ -1010,6 +1017,9 @@ void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr)case OP_AT_S1E2W : fail =__kvm_at (OP_AT_S1E1W , vaddr);break ; +case OP_AT_S1E2A : + fail =__kvm_at (OP_AT_S1E1A , vaddr); +break ;default :WARN_ON_ONCE (1 ); fail = true;
arch/arm64/kvm/emulate-nested.c +2 -0
@@ -786,6 +786,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {SR_TRAP (OP_AT_S12E1W ,CGT_HCR_NV ),SR_TRAP (OP_AT_S12E0R ,CGT_HCR_NV ),SR_TRAP (OP_AT_S12E0W ,CGT_HCR_NV ), +SR_TRAP (OP_AT_S1E2A ,CGT_HCR_NV ),SR_TRAP (OP_TLBI_IPAS2E1 ,CGT_HCR_NV ),SR_TRAP (OP_TLBI_RIPAS2E1 ,CGT_HCR_NV ), SR_TRAP(OP_TLBI_IPAS2LE1, CGT_HCR_NV),
@@ -867,6 +868,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {SR_TRAP (OP_AT_S1E0W ,CGT_HCR_AT ),SR_TRAP (OP_AT_S1E1RP ,CGT_HCR_AT ),SR_TRAP (OP_AT_S1E1WP ,CGT_HCR_AT ), +SR_TRAP (OP_AT_S1E1A ,CGT_HCR_AT ),SR_TRAP (SYS_ERXPFGF_EL1 , CGT_HCR_nFIEN),SR_TRAP (SYS_ERXPFGCTL_EL1 , CGT_HCR_nFIEN), SR_TRAP(SYS_ERXPFGCDN_EL1, CGT_HCR_nFIEN),
arch/arm64/kvm/sys_regs.c +11 -0
@@ -2818,6 +2818,13 @@ static bool handle_at_s1e2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, {u32 op =sys_insn (p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); +/* There is no FGT associated with AT S1E2A :-( */ +if (op ==OP_AT_S1E2A && + !kvm_has_feat (vcpu->kvm,ID_AA64ISAR2_EL1 ,ATS1A ,IMP )) { +kvm_inject_undefined (vcpu); +return false; + } +__kvm_at_s1e2 (vcpu, op, p->regval);return true;
@@ -3188,6 +3195,7 @@ static struct sys_reg_desc sys_insn_descs[] = {SYS_INSN (AT_S12E1W , handle_at_s12),SYS_INSN (AT_S12E0R , handle_at_s12),SYS_INSN (AT_S12E0W , handle_at_s12), +SYS_INSN (AT_S1E2A , handle_at_s1e2),SYS_INSN (TLBI_IPAS2E1IS , handle_ipas2e1is), SYS_INSN(TLBI_RIPAS2E1IS, handle_ripas2e1is),
@@ -4645,6 +4653,9 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu)HFGITR_EL2_TLBIRVAAE1OS | HFGITR_EL2_TLBIRVAE1OS); +if (!kvm_has_feat (kvm,ID_AA64ISAR2_EL1 ,ATS1A ,IMP )) + kvm->arch.fgu[HFGITR_GROUP ] |=HFGITR_EL2_ATS1E1A ; +if (!kvm_has_feat (kvm,ID_AA64MMFR1_EL1 ,PAN ,PAN2 )) kvm->arch.fgu[HFGITR_GROUP ] |= (HFGITR_EL2_ATS1E1RP |HFGITR_EL2_ATS1E1WP );
Signature · 1 line
2.39.2
Hi Marc,
Quoted 21 lines · Marc Zyngier
This is the fourth revision of the address translation emulation for NV support on arm64 previously posted at [1].
Thanks again to Alex for his continuous (contiguous? ;-) scrutiny on this series.
* From v3:
- Fix out of range conditions for TxSZ when LVA is implemented
- Fix implementation of
R_VPBBFto deliver an Address Size Fault- Don't grant PX if UW is set
- Various cleanups
- Collected Alex's RBs, with thanks.
I've added the usual reviewers on Cc, plus people who explicitly asked to be on it, and people who seem to be super keen on NV.
Quoted 8 lines
Patches on top of 6.11-rc1, tested on my usual M2 (so VHE only). FWIW, I plan to take this into 6.12.
[1] https://lore.kernel.org/r/[email protected]
Joey Gouly (1):
KVM: arm64: Makekvm_at()take an OP_AT_*
Have you tested/tried NV with host/L0 booted with GICv4.x enabled? We do see L2 boot hang and I don't have much debug info at the moment.
Quoted 34 lines
Marc Zyngier (17):
arm64: Add missing APTable and TCR_ELx.HPD masks
arm64: AddPAR_EL1field description
arm64: Add system register encoding for PSTATE.PAN
arm64: AddESR_ELx_FSC_ADDRSZ_L()helper
KVM: arm64: nv: Enforce S2 alignment when contiguous bit is set
KVM: arm64: nv: Turnupper_attrfor S2 walk into the full descriptor
KVM: arm64: nv: Honor absence ofFEAT_PAN2
KVM: arm64: nv: Add basic emulation of AT S1E{0,1}{R,W}
KVM: arm64: nv: Add basic emulation of AT S1E1{R,W}P
KVM: arm64: nv: Add basic emulation of AT S1E2{R,W}
KVM: arm64: nv: Add emulation of AT S12E{0,1}{R,W}
KVM: arm64: nv: Makeps_to_output_size()generally available
KVM: arm64: nv: Add SW walker for AT S1 emulation
KVM: arm64: nv: Sanitise SCTLR_EL1.EPAN according to VM configuration
KVM: arm64: nv: Make AT+PAN instructions aware ofFEAT_PAN3
KVM: arm64: nv: Plumb handling of AT S1* traps from EL2
KVM: arm64: nv: Add support forFEAT_ATS1Aarch/arm64/include/asm/esr.h | 5 +- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_asm.h | 6 +- arch/arm64/include/asm/kvm_nested.h | 40 +- arch/arm64/include/asm/pgtable-hwdef.h | 9 + arch/arm64/include/asm/sysreg.h | 22 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/at.c | 1101 ++++++++++++++++++++++++ arch/arm64/kvm/emulate-nested.c | 2 + arch/arm64/kvm/hyp/include/hyp/fault.h | 2 +- arch/arm64/kvm/nested.c | 41 +- arch/arm64/kvm/sys_regs.c | 60 ++ 12 files changed, 1259 insertions(+), 32 deletions(-)create mode 100644
arch/arm64/kvm/at.c
Signature · 2 lines
Thanks, Ganapat/GK
Hi Ganapat,
Quoted 2 lines · Ganapatrao Kulkarni
Have you tested/tried NV with host/L0 booted with GICv4.x enabled? We do see L2 boot hang and I don't have much debug info at the moment.
Sorry, I've been sitting on a fix for this that I've been meaning to send out.
The issue has to do with the fact that the vpe is marked as runnable (its_vpe::pending_last = true) when descheduled w/o requesting a doorbell IRQ. Once KVM completes the nested ERET, it believes an IRQ is pending for L1 (kvm_vgic_vcpu_pending_irq() returns true), and injects the nested exception.
This can be papered over by requesting the doorbell IRQ, which we need anyway to kick us out of the L2 when an IRQ becomes pending for L1.
Could you take this diff for a spin?
arch/arm64/include/asm/kvm_host.h +12 -4
@@ -613,6 +613,12 @@ struct cpu_sve_state {* field. */ struct kvm_host_data { +/* SVE enabled for EL0 */ +#define HOST_SVE_ENABLED 0 +/* SME enabled for EL0 */ +#define HOST_SME_ENABLED 1 +unsigned long flags; +struct kvm_cpu_context host_ctxt;/*
@@ -908,10 +914,8 @@ struct kvm_vcpu_arch {/* Save TRBE context if active */ #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6)) -/* SVE enabled for host EL0 */ -#define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0)) -/* SME enabled for EL0 */ -#define HOST_SME_ENABLED __vcpu_single_flag(sflags, BIT(1)) +/* KVM is currently emulating a nested ERET */ +#define IN_NESTED_ERET __vcpu_single_flag(sflags, BIT(0))/* Physical CPU not in supported_cpus */ #define ON_UNSUPPORTED_CPU __vcpu_single_flag(sflags, BIT(2))/* WFIT instruction trapped */
@@ -1294,6 +1298,10 @@ DECLARE_KVM_HYP_PER_CPU(struct kvm_host_data, kvm_host_data); &this_cpu_ptr_hyp_sym(kvm_host_data)->f)#endif +#define host_data_set_flag (nr) set_bit(nr, host_data_ptr(flags)) +#define host_data_test_flag (nr) test_bit(nr, host_data_ptr(flags)) +#define host_data_clear_flag (nr) clear_bit(nr, host_data_ptr(flags)) +/* Check whether the FP regs are owned by the guest */ static inline bool guest_owns_fp_regs (void ) {
arch/arm64/kvm/emulate-nested.c +2 -0
@@ -2310,6 +2310,7 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) }preempt_disable (); +vcpu_set_flag (vcpu,IN_NESTED_ERET );kvm_arch_vcpu_put (vcpu);if (!esr_iss_is_eretax (esr))
@@ -2321,6 +2322,7 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) *vcpu_cpsr (vcpu) = spsr;kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +vcpu_clear_flag (vcpu,IN_NESTED_ERET );preempt_enable (); }
arch/arm64/kvm/fpsimd.c +6 -6
@@ -64,14 +64,14 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; *host_data_ptr (fpsimd_state) =kern_hyp_va (¤t->thread.uw.fpsimd_state); -vcpu_clear_flag (vcpu,HOST_SVE_ENABLED ); +host_data_clear_flag (HOST_SVE_ENABLED );if (read_sysreg (cpacr_el1) &CPACR_EL1_ZEN_EL0EN ) -vcpu_set_flag (vcpu,HOST_SVE_ENABLED ); +host_data_set_flag (HOST_SVE_ENABLED );if (system_supports_sme ()) { -vcpu_clear_flag (vcpu,HOST_SME_ENABLED ); +host_data_clear_flag (HOST_SME_ENABLED );if (read_sysreg (cpacr_el1) &CPACR_EL1_SMEN_EL0EN ) -vcpu_set_flag (vcpu,HOST_SME_ENABLED ); +host_data_set_flag (HOST_SME_ENABLED );/* * If PSTATE.SM is enabled then save any pending FP
@@ -167,7 +167,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)*/ if (has_vhe () &&system_supports_sme ()) {/* Also restore EL0 state seen on entry */ -if (vcpu_get_flag (vcpu,HOST_SME_ENABLED )) +if (host_data_test_flag (HOST_SME_ENABLED ))sysreg_clear_set (CPACR_EL1 ,0 , CPACR_ELx_SMEN);else sysreg_clear_set(CPACR_EL1,
@@ -226,7 +226,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)* for EL0. To avoid spurious traps, restore the trap state * seen by kvm_arch_vcpu_load_fp(): */ -if (vcpu_get_flag (vcpu,HOST_SVE_ENABLED )) +if (host_data_test_flag (HOST_SVE_ENABLED ))sysreg_clear_set (CPACR_EL1 ,0 ,CPACR_EL1_ZEN_EL0EN );else sysreg_clear_set (CPACR_EL1 ,CPACR_EL1_ZEN_EL0EN ,0 );
arch/arm64/kvm/vgic/vgic-v4.c +17 -1
@@ -336,6 +336,22 @@ void vgic_v4_teardown(struct kvm *kvm) its_vm->vpes =NULL ; } +static inline bool vgic_v4_want_doorbell (struct kvm_vcpu *vcpu) +{ +if (vcpu_get_flag (vcpu,IN_WFI )) +return true; + +if (likely (!vcpu_has_nv (vcpu))) +return false; + +/* +* GICv4 hardware is only ever used for the L1. Mark the vPE (i.e. the +* L1 context) nonresident and request a doorbell to kick us out of the +* L2 when an IRQ becomes pending. +*/ +return vcpu_get_flag (vcpu,IN_NESTED_ERET ); +} +int vgic_v4_put (struct kvm_vcpu *vcpu) {struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
@@ -343,7 +359,7 @@ int vgic_v4_put(struct kvm_vcpu *vcpu)if (!vgic_supports_direct_msis (vcpu->kvm) || !vpe->resident)return 0 ; -return its_make_vpe_non_resident (vpe, !!vcpu_get_flag (vcpu,IN_WFI )); +return its_make_vpe_non_resident (vpe,vgic_v4_want_doorbell (vcpu)); }int vgic_v4_load (struct kvm_vcpu *vcpu)
Signature · 2 lines
Thanks, Oliver
Quoted 6 lines · Oliver Upton
Hi Ganapat,
Quoted 2 lines · Ganapatrao Kulkarni
Have you tested/tried NV with host/L0 booted with GICv4.x enabled? We do see L2 boot hang and I don't have much debug info at the moment.
Quoted 8 lines
Sorry, I've been sitting on a fix for this that I've been meaning to send out.
The issue has to do with the fact that the vpe is marked as runnable (
its_vpe::pending_last= true) when descheduled w/o requesting a doorbell IRQ. Once KVM completes the nested ERET, it believes an IRQ is pending for L1 (kvm_vgic_vcpu_pending_irq()returns true), and injects the nested exception.
Ah OK, I could see it was returning back to L1 after ERET in ftrace and this was getting in loop and L2 was never getting a chance to run.
Quoted 5 lines
This can be papered over by requesting the doorbell IRQ, which we need anyway to kick us out of the L2 when an IRQ becomes pending for L1.
Could you take this diff for a spin?
Thanks Oliver for this fix!.
I could boot L1 and then L2 with this diff.
Quoted 140 lines
arch/arm64/include/asm/kvm_host.h +12
-4@@ -613,6 +613,12 @@ struct cpu_sve_state {* field. */ struct kvm_host_data { +/* SVE enabled for EL0 */ +#define HOST_SVE_ENABLED 0 +/* SME enabled for EL0 */ +#define HOST_SME_ENABLED 1 +unsigned long flags; +struct kvm_cpu_context host_ctxt;/* @@ -908,10 +914,8 @@ struct kvm_vcpu_arch {/* Save TRBE context if active */ #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6)) -/* SVE enabled for host EL0 */ -#define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0)) -/* SME enabled for EL0 */ -#define HOST_SME_ENABLED __vcpu_single_flag(sflags, BIT(1)) +/* KVM is currently emulating a nested ERET */ +#define IN_NESTED_ERET __vcpu_single_flag(sflags, BIT(0))/* Physical CPU not in supported_cpus */ #define ON_UNSUPPORTED_CPU __vcpu_single_flag(sflags, BIT(2))/* WFIT instruction trapped */ @@ -1294,6 +1298,10 @@ DECLARE_KVM_HYP_PER_CPU(struct kvm_host_data, kvm_host_data); &this_cpu_ptr_hyp_sym(kvm_host_data)->f)#endif +#define host_data_set_flag (nr) set_bit(nr, host_data_ptr(flags)) +#define host_data_test_flag (nr) test_bit(nr, host_data_ptr(flags)) +#define host_data_clear_flag (nr) clear_bit(nr, host_data_ptr(flags)) +/* Check whether the FP regs are owned by the guest */ static inline bool guest_owns_fp_regs (void ) {arch/arm64/kvm/emulate-nested.c +2
-0@@ -2310,6 +2310,7 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) }preempt_disable (); +vcpu_set_flag (vcpu,IN_NESTED_ERET );kvm_arch_vcpu_put (vcpu);if (!esr_iss_is_eretax (esr))@@ -2321,6 +2322,7 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) *vcpu_cpsr (vcpu) = spsr;kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +vcpu_clear_flag (vcpu,IN_NESTED_ERET );preempt_enable (); }arch/arm64/kvm/fpsimd.c +6
-6@@ -64,14 +64,14 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; *host_data_ptr (fpsimd_state) =kern_hyp_va (¤t->thread.uw.fpsimd_state); -vcpu_clear_flag (vcpu,HOST_SVE_ENABLED ); +host_data_clear_flag (HOST_SVE_ENABLED );if (read_sysreg (cpacr_el1) &CPACR_EL1_ZEN_EL0EN ) -vcpu_set_flag (vcpu,HOST_SVE_ENABLED ); +host_data_set_flag (HOST_SVE_ENABLED );if (system_supports_sme ()) { -vcpu_clear_flag (vcpu,HOST_SME_ENABLED ); +host_data_clear_flag (HOST_SME_ENABLED );if (read_sysreg (cpacr_el1) &CPACR_EL1_SMEN_EL0EN ) -vcpu_set_flag (vcpu,HOST_SME_ENABLED ); +host_data_set_flag (HOST_SME_ENABLED );/* * If PSTATE.SM is enabled then save any pending FP @@ -167,7 +167,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)*/ if (has_vhe () &&system_supports_sme ()) {/* Also restore EL0 state seen on entry */ -if (vcpu_get_flag (vcpu,HOST_SME_ENABLED )) +if (host_data_test_flag (HOST_SME_ENABLED ))sysreg_clear_set (CPACR_EL1 ,0 , CPACR_ELx_SMEN);else sysreg_clear_set(CPACR_EL1,@@ -226,7 +226,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)* for EL0. To avoid spurious traps, restore the trap state * seen by kvm_arch_vcpu_load_fp(): */ -if (vcpu_get_flag (vcpu,HOST_SVE_ENABLED )) +if (host_data_test_flag (HOST_SVE_ENABLED ))sysreg_clear_set (CPACR_EL1 ,0 ,CPACR_EL1_ZEN_EL0EN );else sysreg_clear_set (CPACR_EL1 ,CPACR_EL1_ZEN_EL0EN ,0 );arch/arm64/kvm/vgic/vgic-v4.c +17
-1@@ -336,6 +336,22 @@ void vgic_v4_teardown(struct kvm *kvm) its_vm->vpes =NULL ; } +static inline bool vgic_v4_want_doorbell (struct kvm_vcpu *vcpu) +{ +if (vcpu_get_flag (vcpu,IN_WFI )) +return true; + +if (likely (!vcpu_has_nv (vcpu))) +return false; + +/* +* GICv4 hardware is only ever used for the L1. Mark the vPE (i.e. the +* L1 context) nonresident and request a doorbell to kick us out of the +* L2 when an IRQ becomes pending. +*/ +return vcpu_get_flag (vcpu,IN_NESTED_ERET ); +} +int vgic_v4_put (struct kvm_vcpu *vcpu) {struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;@@ -343,7 +359,7 @@ int vgic_v4_put(struct kvm_vcpu *vcpu)if (!vgic_supports_direct_msis (vcpu->kvm) || !vpe->resident)return 0 ; -return its_make_vpe_non_resident (vpe, !!vcpu_get_flag (vcpu,IN_WFI )); +return its_make_vpe_non_resident (vpe,vgic_v4_want_doorbell (vcpu)); }int vgic_v4_load (struct kvm_vcpu *vcpu)
Signature · 2 lines
Thanks, Ganapat/GK
Quoted 9 lines · Marc Zyngier
This is the fourth revision of the address translation emulation for NV support on arm64 previously posted at [1].
Thanks again to Alex for his continuous (contiguous? ;-) scrutiny on this series.
* From v3:
[...]
Applied to kvm-arm64/s2-ptdump, thanks!
[01/18] arm64: Add missing APTable and TCR_ELx.HPD masks
commit: 4abc783e4741cd33216e7796e9b2f4973b4bca61
[02/18] arm64: Add PAR_EL1 field description
commit: 6dcd2ac7ea7c5b20b416ee09d8d5d2ec89866ef8
[03/18] arm64: Add system register encoding for PSTATE.PAN
commit: b229b46b0bf7828bef5f88c91708776869b751ac
[04/18] arm64: Add ESR_ELx_FSC_ADDRSZ_L() helper
commit: 5fddf9abc31a57e2cc35287998994cf4a684fada
[05/18] KVM: arm64: Make kvm_at() take an OP_AT_*
commit: 69231a6fcb638b7929e9fc88c4fa73a04e6d4e0c
[06/18] KVM: arm64: nv: Enforce S2 alignment when contiguous bit is set
commit: 4155539bc5baab514ac71285a1a13fcf148f9cf1
[07/18] KVM: arm64: nv: Turn upper_attr for S2 walk into the full descriptor
commit: 0a0f25b71ca544388717f8bf4a54ba324e234e7a
[08/18] KVM: arm64: nv: Honor absence of FEAT_PAN2
commit: 90659853febcf63ceb71529b247d518df3c2a76c
[09/18] KVM: arm64: nv: Add basic emulation of AT S1E{0,1}{R,W}
commit: 477e89cabb1428d5989430d57828347f5de2be9c
[10/18] KVM: arm64: nv: Add basic emulation of AT S1E1{R,W}P
commit: be0135bde1df5e80cffacd2ed6f952e6d38d6f71
[11/18] KVM: arm64: nv: Add basic emulation of AT S1E2{R,W}
commit: e794049b9acbd6500b77b9ce92a95101091b52d3
[12/18] KVM: arm64: nv: Add emulation of AT S12E{0,1}{R,W}
commit: be04cebf3e78874627dc1042991d5d504464a5cc
[13/18] KVM: arm64: nv: Make ps_to_output_size() generally available
commit: 97634dac1974d28e5ffc067d257f0b0f79b5ed2e
[14/18] KVM: arm64: nv: Add SW walker for AT S1 emulation
commit: d6a01a2dc760c8350fa182a6afd69fabab131f73
[15/18] KVM: arm64: nv: Sanitise SCTLR_EL1.EPAN according to VM configuration
commit: 2441418f3aadb3f9232431aeb10d89e48a934d94
[16/18] KVM: arm64: nv: Make AT+PAN instructions aware of FEAT_PAN3
commit: d95bb9ef164edb33565cb73e3f0b0a581b3e4fbb
[17/18] KVM: arm64: nv: Plumb handling of AT S1* traps from EL2
commit: 8df747f4f3a5c680e3c0e68af3487b97343ca80a
[18/18] KVM: arm64: nv: Add support for FEAT_ATS1A
commit: ff987ffc0c18c98f05ddc7696d56bb493b994450Cheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Marc Zyngier
maintainerQuoted 5 lines · Marc Zyngier
Quoted 3 lines · Marc Zyngier
This is the fourth revision of the address translation emulation for NV support on arm64 previously posted at [1].
Quoted 8 lines
Quoted 6 lines
Thanks again to Alex for his continuous (contiguous? ;-) scrutiny on this series.
* From v3:
[...]
Applied to kvm-arm64/s2-ptdump, thanks!
Note to self: fix the bloody script so that it reports -next and whatever branch I'm currently working on.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.