KVM: arm64: ARMv8.3/8.4 Nested Virtualization support
Subsystems (4)
Marc Zyngier
maintainerv3 cover KVM: arm64: ARMv8.3/8.4 Nested Virtualization support
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11
From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
And to be clear: although Jintack and Christoffer have written tons of the stuff originaly, I'm the one responsible for breaking it!
[1] https://lore.kernel.org/r/[email protected]
[2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/nv-5.11.-WIP
[3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5
Andre Przywara (1):
KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ
Christoffer Dall (15):
KVM: arm64: nv: Introduce nested virtualization VCPU feature
KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
KVM: arm64: nv: Handle trapped ERET from virtual EL2
KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
changes
KVM: arm64: nv: Implement nested Stage-2 page table walk logic
KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
KVM: arm64: nv: arch_timer: Support hyp timer emulation
KVM: arm64: nv: vgic: Emulate the HW bit in software
KVM: arm64: nv: Add nested GICv3 tracepoints
KVM: arm64: nv: Sync nested timer state with ARMv8.4
Jintack Lim (19):
arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
KVM: arm64: nv: Handle HCR_EL2.NV system register traps
KVM: arm64: nv: Support virtual EL2 exceptions
KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
KVM: arm64: nv: Handle PSCI call via smc from the guest
KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
KVM: arm64: nv: Set a handler for the system instruction traps
KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
KVM: arm64: nv: Nested GICv3 Support
Marc Zyngier (31):
KVM: arm64: nv: Add EL2 system registers to vcpu context
KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
KVM: arm64: nv: Handle virtual EL2 registers in
vcpu_read/write_sys_reg()
KVM: arm64: nv: Handle SPSR_EL2 specially
KVM: arm64: nv: Handle HCR_EL2.E2H specially
KVM: arm64: nv: Save/Restore vEL2 sysregs
KVM: arm64: nv: Forward debug traps to the nested guest
KVM: arm64: nv: Filter out unsupported features from ID regs
KVM: arm64: nv: Hide RAS from nested guests
KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
KVM: arm64: nv: Handle shadow stage 2 page faults
KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's
KVM: arm64: nv: Add handling of EL2-specific timer registers
KVM: arm64: nv: Load timer before the GIC
KVM: arm64: nv: Don't load the GICv4 context on entering a nested
guest
KVM: arm64: nv: Implement maintenance interrupt forwarding
KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT
KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation
KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like
information
KVM: arm64: Allow populating S2 SW bits
KVM: arm64: nv: Tag shadow S2 entries with nested level
KVM: arm64: nv: Add include containing the VNCR_EL2 offsets
KVM: arm64: Map VNCR-capable registers to a separate page
KVM: arm64: nv: Move nested vgic state into the sysreg file
KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature
KVM: arm64: nv: Synchronize PSTATE early on exit
KVM: arm64: nv: Allocate VNCR page when required
KVM: arm64: nv: Enable ARMv8.4-NV support
KVM: arm64: nv: Fast-track 'InHost' exception returns
KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests
.../admin-guide/kernel-parameters.txt | 4 + .../virt/kvm/devices/arm-vgic-v3.rst | 12 +- arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/esr.h | 6 + arch/arm64/include/asm/kvm_arm.h | 28 +- arch/arm64/include/asm/kvm_asm.h | 4 + arch/arm64/include/asm/kvm_emulate.h | 145 +- arch/arm64/include/asm/kvm_host.h | 175 ++- arch/arm64/include/asm/kvm_hyp.h | 2 + arch/arm64/include/asm/kvm_mmu.h | 17 +- arch/arm64/include/asm/kvm_nested.h | 152 ++ arch/arm64/include/asm/kvm_pgtable.h | 10 + arch/arm64/include/asm/sysreg.h | 104 +- arch/arm64/include/asm/vncr_mapping.h | 73 + arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kernel/cpufeature.c | 35 + arch/arm64/kvm/Makefile | 4 +- arch/arm64/kvm/arch_timer.c | 189 ++- arch/arm64/kvm/arm.c | 34 +- arch/arm64/kvm/at.c | 231 ++++ arch/arm64/kvm/emulate-nested.c | 186 +++ arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/handle_exit.c | 81 +- arch/arm64/kvm/hyp/exception.c | 44 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 31 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 28 +- arch/arm64/kvm/hyp/nvhe/switch.c | 10 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 6 + arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 207 ++- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 +- arch/arm64/kvm/hyp/vhe/tlb.c | 83 ++ arch/arm64/kvm/inject_fault.c | 62 +- arch/arm64/kvm/mmu.c | 183 ++- arch/arm64/kvm/nested.c | 908 ++++++++++++ arch/arm64/kvm/reset.c | 14 +- arch/arm64/kvm/sys_regs.c | 1226 ++++++++++++++++- arch/arm64/kvm/sys_regs.h | 6 + arch/arm64/kvm/trace_arm.h | 65 +- arch/arm64/kvm/vgic/vgic-init.c | 30 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 + arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 240 ++++ arch/arm64/kvm/vgic/vgic-v3.c | 39 +- arch/arm64/kvm/vgic/vgic.c | 44 + arch/arm64/kvm/vgic/vgic.h | 10 + include/kvm/arm_arch_timer.h | 7 + include/kvm/arm_vgic.h | 16 + tools/arch/arm/include/uapi/asm/kvm.h | 1 + 50 files changed, 4890 insertions(+), 160 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/vncr_mapping.h create mode 100644 arch/arm64/kvm/at.c create mode 100644 arch/arm64/kvm/emulate-nested.c create mode 100644 arch/arm64/kvm/nested.c create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 1/66 arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
From: Jintack Lim <[email protected]>
Add a new ARM64_HAS_NESTED_VIRT feature to indicate that the CPU has the ARMv8.3 nested virtualization capability.
This will be used to support nested virtualization in KVM.
--- .../admin-guide/kernel-parameters.txt | 4 +++ arch/arm64/include/asm/cpucaps.h | 1 + arch/arm64/kernel/cpufeature.c | 25 +++++++++++++++++++ 3 files changed, 30 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +4 -0
@@ -2285,6 +2285,10 @@ [KVM,ARM] Allow use of GICv4 for direct injection of LPIs. + kvm-arm.nested= + [KVM,ARM] Allow nested virtualization in KVM/ARM. + Default is 0 (disabled) + kvm_cma_resv_ratio=n [PPC] Reserves given percentage from system memory area for contiguous memory allocation for KVM hash pagetable
arch/arm64/include/asm/cpucaps.h +1 -0
@@ -16,6 +16,7 @@#define ARM64_WORKAROUND_CAVIUM_23154 6#define ARM64_WORKAROUND_834220 7#define ARM64_HAS_NO_HW_PREFETCH 8 +#define ARM64_HAS_NESTED_VIRT 9#define ARM64_HAS_VIRT_HOST_EXTN 11#define ARM64_WORKAROUND_CAVIUM_27456 12#define ARM64_HAS_32BIT_EL0 13
arch/arm64/kernel/cpufeature.c +25 -0
@@ -1577,6 +1577,21 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused)if (!alternative_is_applied (ARM64_HAS_VIRT_HOST_EXTN )) write_sysreg(read_sysreg(tpidr_el1), tpidr_el2); } + +static bool nested_param; +static bool has_nested_virt_support (const struct arm64_cpu_capabilities *cap, +int scope) +{ +return has_cpuid_feature (cap, scope) && + nested_param; +} + +static int __init kvmarm_nested_cfg (char *buf) +{ +return strtobool (buf, &nested_param); +} + +early_param ("kvm-arm.nested" , kvmarm_nested_cfg);#endif static void cpu_has_fwb (const struct arm64_cpu_capabilities *__unused)
@@ -1790,6 +1805,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = runs_at_el2, .cpu_enable = cpu_copy_el2regs, }, + { + .desc = "Nested Virtualization Support", + .capability = ARM64_HAS_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_NV_SHIFT, + .min_field_value = 1, + }, #endif /* CONFIG_ARM64_VHE */ { .desc = "32-bit EL0 Support",
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 2/66 KVM: arm64: nv: Introduce nested virtualization VCPU feature
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check.
Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead.
We don't yet allow userspace to actually set this feature.
--- arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + 2 files changed, 15 insertions(+) create mode 100644 arch/arm64/include/asm/kvm_nested.h
arch/arm64/include/asm/kvm_nested.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NESTED_H +#define __ARM64_KVM_NESTED_H + +#include <linux/kvm_host.h> + +static inline bool nested_virt_in_use (const struct kvm_vcpu *vcpu) +{ +return (!__is_defined (__KVM_NVHE_HYPERVISOR__) && +cpus_have_final_cap (ARM64_HAS_NESTED_VIRT ) && +test_bit (KVM_ARM_VCPU_HAS_EL2 , vcpu->arch.features)); +} + +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/uapi/asm/kvm.h +1 -0
@@ -106,6 +106,7 @@ struct kvm_regs {#define KVM_ARM_VCPU_SVE 4/* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5/* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6/* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_HAS_EL2 7/* Support nested virtualization */ struct kvm_vcpu_init {__u32 target;
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 3/66 KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU.
--- arch/arm64/kvm/reset.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/arm64/kvm/reset.c +12 -1
@@ -27,6 +27,7 @@#include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include <asm/virt.h> /* Maximum phys_shift supported for any VM on this host */
@@ -38,6 +39,9 @@ static u32 kvm_ipa_limit;#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ PSR_F_BIT | PSR_D_BIT) +#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) +#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ PSR_AA32_I_BIT | PSR_AA32_F_BIT)
@@ -224,11 +228,18 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)switch (vcpu->arch.target) {default :if (test_bit (KVM_ARM_VCPU_EL1_32BIT , vcpu->arch.features)) { -if (!cpus_have_const_cap (ARM64_HAS_32BIT_EL1 )) { +/* +* The CPU must support 32bit EL1, and 32bit +* NV is just not a thing... +*/ +if (!cpus_have_const_cap (ARM64_HAS_32BIT_EL1 ) || +nested_virt_in_use (vcpu)) { ret = -EINVAL ; goto out; } pstate =VCPU_RESET_PSTATE_SVC ; + }else if (nested_virt_in_use (vcpu)) { + pstate =VCPU_RESET_PSTATE_EL2 ; }else { pstate =VCPU_RESET_PSTATE_EL1 ; }
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 4/66 KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
We were not allowing userspace to set a more privileged mode for the VCPU than EL1, but we should allow this when nested virtualization is enabled for the VCPU.
--- arch/arm64/kvm/guest.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/kvm/guest.c +6 -0
@@ -24,6 +24,7 @@#include <asm/fpsimd.h> #include <asm/kvm.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_nested.h> #include <asm/sigcontext.h> #include "trace.h"
@@ -242,6 +243,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)if (vcpu_el1_is_32bit (vcpu))return -EINVAL ;break ; +case PSR_MODE_EL2h: +case PSR_MODE_EL2t: +if (vcpu_el1_is_32bit (vcpu) || !nested_virt_in_use (vcpu)) +return -EINVAL ; +break ;default : err = -EINVAL ; goto out;
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 5/66 KVM: arm64: nv: Add EL2 system registers to vcpu context
In reply to Marc ZyngierAdd the minimal set of EL2 system registers to the vcpu context. Nothing uses them just yet.
--- arch/arm64/include/asm/kvm_host.h | 34 ++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +33 -1
@@ -206,12 +206,44 @@ enum vcpu_sysreg {CNTP_CVAL_EL0 ,CNTP_CTL_EL0 , -/* 32bit specific registers. Keep them at the end of the range */ +/* 32bit specific registers. */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2 ,/* Instruction Fault Status Register */ FPEXC32_EL2 ,/* Floating-Point Exception Control Register */ DBGVCR32_EL2 ,/* Debug Vector Catch Register */ +/* EL2 registers */ +VPIDR_EL2 ,/* Virtualization Processor ID Register */ +VMPIDR_EL2 ,/* Virtualization Multiprocessor ID Register */ +SCTLR_EL2 ,/* System Control Register (EL2) */ +ACTLR_EL2 ,/* Auxiliary Control Register (EL2) */ +HCR_EL2 ,/* Hypervisor Configuration Register */ +MDCR_EL2 ,/* Monitor Debug Configuration Register (EL2) */ +CPTR_EL2 ,/* Architectural Feature Trap Register (EL2) */ +HSTR_EL2 ,/* Hypervisor System Trap Register */ +HACR_EL2 ,/* Hypervisor Auxiliary Control Register */ +TTBR0_EL2 ,/* Translation Table Base Register 0 (EL2) */ +TTBR1_EL2 ,/* Translation Table Base Register 1 (EL2) */ +TCR_EL2 ,/* Translation Control Register (EL2) */ +VTTBR_EL2 ,/* Virtualization Translation Table Base Register */ +VTCR_EL2 ,/* Virtualization Translation Control Register */ +SPSR_EL2 ,/* EL2 saved program status register */ +ELR_EL2 ,/* EL2 exception link register */ +AFSR0_EL2 ,/* Auxiliary Fault Status Register 0 (EL2) */ +AFSR1_EL2 ,/* Auxiliary Fault Status Register 1 (EL2) */ +ESR_EL2 ,/* Exception Syndrome Register (EL2) */ +FAR_EL2 ,/* Hypervisor IPA Fault Address Register */ +HPFAR_EL2 ,/* Hypervisor IPA Fault Address Register */ +MAIR_EL2 ,/* Memory Attribute Indirection Register (EL2) */ +AMAIR_EL2 ,/* Auxiliary Memory Attribute Indirection Register (EL2) */ +VBAR_EL2 ,/* Vector Base Address Register (EL2) */ +RVBAR_EL2 ,/* Reset Vector Base Address Register */ +RMR_EL2 ,/* Reset Management Register */ +CONTEXTIDR_EL2 ,/* Context ID Register (EL2) */ +TPIDR_EL2 ,/* EL2 Software Thread ID Register */ +CNTHCTL_EL2 ,/* Counter-timer Hypervisor Control register */ +SP_EL2 ,/* EL2 Stack Pointer */ +NR_SYS_REGS /* Nothing after this line! */ };
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 05/66] KVM: arm64: nv: Add EL2 system registers to vcpu context
On Thu, 10 Dec 2020 15:59:01 +0000
Marc Zyngier <[email protected]> wrote:
Hi,
Quoted 4 lines
Add the minimal set of EL2 system registers to the vcpu context. Nothing uses them just yet.
Signed-off-by Marc Zyngier <[email protected]>
Checked against the ARM ARM that this list contains the _EL2 registers available in ARMv8.1, minus timer and GIC registers.
Cheers,
Andre
Quoted 44 lines
--- arch/arm64/include/asm/kvm_host.h | 34 ++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)arch/arm64/include/asm/kvm_host.h +26
-1@@ -206,12 +206,44 @@ enum vcpu_sysreg {CNTP_CVAL_EL0 ,CNTP_CTL_EL0 , -/* 32bit specific registers. Keep them at the end of the range */ +/* 32bit specific registers. */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2 ,/* Instruction Fault Status Register */ FPEXC32_EL2 ,/* Floating-Point Exception Control Register */ DBGVCR32_EL2 ,/* Debug Vector Catch Register */ +/* EL2 registers */ +VPIDR_EL2 ,/* Virtualization Processor ID Register */ +VMPIDR_EL2 ,/* Virtualization Multiprocessor ID Register */ +SCTLR_EL2 ,/* System Control Register (EL2) */ +ACTLR_EL2 ,/* Auxiliary Control Register (EL2) */ +HCR_EL2 ,/* Hypervisor Configuration Register */ +MDCR_EL2 ,/* Monitor Debug Configuration Register (EL2) */ +CPTR_EL2 ,/* Architectural Feature Trap Register (EL2) */ +HSTR_EL2 ,/* Hypervisor System Trap Register */ +HACR_EL2 ,/* Hypervisor Auxiliary Control Register */ +TTBR0_EL2 ,/* Translation Table Base Register 0 (EL2) */ +TTBR1_EL2 ,/* Translation Table Base Register 1 (EL2) */ +TCR_EL2 ,/* Translation Control Register (EL2) */ +VTTBR_EL2 ,/* Virtualization Translation Table Base Register */ +VTCR_EL2 ,/* Virtualization Translation Control Register */ +SPSR_EL2 ,/* EL2 saved program status register */ +ELR_EL2 ,/* EL2 exception link register */ +AFSR0_EL2 ,/* Auxiliary Fault Status Register 0 (EL2) */ +AFSR1_EL2 ,/* Auxiliary Fault Status Register 1 (EL2) */ +ESR_EL2 ,/* Exception Syndrome Register (EL2) */ +FAR_EL2 ,/* Hypervisor IPA Fault Address Register */ +HPFAR_EL2 ,/* Hypervisor IPA Fault Address Register */ +MAIR_EL2 ,/* Memory Attribute Indirection Register (EL2) */ +AMAIR_EL2 ,/* Auxiliary Memory Attribute Indirection Register (EL2) */ + VBAR_EL2, /* Vector Base Address Register (EL2) */
arch/arm64/include/asm/kvm_host.h enum vcpu_sysreg { · line 249
Quoted 10 lines
arch/arm64/include/asm/kvm_host.h +7
-0+RVBAR_EL2 ,/* Reset Vector Base Address Register */ +RMR_EL2 ,/* Reset Management Register */ +CONTEXTIDR_EL2 ,/* Context ID Register (EL2) */ +TPIDR_EL2 ,/* EL2 Software Thread ID Register */ +CNTHCTL_EL2 ,/* Counter-timer Hypervisor Control register */ +SP_EL2 ,/* EL2 Stack Pointer */ + NR_SYS_REGS /* Nothing after this line! */ };
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 6/66 KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest.
Add convenient primitives for this.
--- arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +55 -0
@@ -173,6 +173,61 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,vcpu_gp_regs (vcpu)->regs[reg_num] = val; } +static inline bool vcpu_mode_el2_ctxt (const struct kvm_cpu_context *ctxt) +{ +unsigned long cpsr = ctxt->regs.pstate; + +switch (cpsr & (PSR_MODE32_BIT |PSR_MODE_MASK )) { +case PSR_MODE_EL2h: +case PSR_MODE_EL2t: +return true; +default : +return false; + } +} + +static inline bool vcpu_mode_el2 (const struct kvm_vcpu *vcpu) +{ +return vcpu_mode_el2_ctxt (&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_e2h_is_set (const struct kvm_cpu_context *ctxt) +{ +return ctxt_sys_reg (ctxt,HCR_EL2 ) &HCR_E2H ; +} + +static inline bool vcpu_el2_e2h_is_set (const struct kvm_vcpu *vcpu) +{ +return __vcpu_el2_e2h_is_set (&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_tge_is_set (const struct kvm_cpu_context *ctxt) +{ +return ctxt_sys_reg (ctxt,HCR_EL2 ) &HCR_TGE ; +} + +static inline bool vcpu_el2_tge_is_set (const struct kvm_vcpu *vcpu) +{ +return __vcpu_el2_tge_is_set (&vcpu->arch.ctxt); +} + +static inline bool __is_hyp_ctxt (const struct kvm_cpu_context *ctxt) +{ +/* +* We are in a hypervisor context if the vcpu mode is EL2 or +* E2H and TGE bits are set. The latter means we are in the user space +* of the VHE kernel. ARMv8.1 ARM describes this as 'InHost' +*/ +return vcpu_mode_el2_ctxt (ctxt) || + (__vcpu_el2_e2h_is_set (ctxt) &&__vcpu_el2_tge_is_set (ctxt)) || +WARN_ON (__vcpu_el2_tge_is_set (ctxt)); +} + +static inline bool is_hyp_ctxt (const struct kvm_vcpu *vcpu) +{ +return __is_hyp_ctxt (&vcpu->arch.ctxt); +} +/* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 7/66 KVM: arm64: nv: Handle HCR_EL2.NV system register traps
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.
This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.
This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.
--- arch/arm64/include/asm/sysreg.h | 44 ++++++++++++- arch/arm64/kvm/sys_regs.c | 112 ++++++++++++++++++++++++++++++-- 2 files changed, 150 insertions(+), 6 deletions(-)
arch/arm64/include/asm/sysreg.h +43 -1
@@ -469,19 +469,50 @@#define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7) +#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0) +#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5) +#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0) +#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1) +#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0) +#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1) +#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2) +#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3) +#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7) +#define SYS_ZCR_EL2 sys_reg(3, 4, 1, 2, 0) + +#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0) +#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1) +#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) +#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) +#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0) +#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)#define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) +#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0) +#define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1) +#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0) +#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1)#define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0)#define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3)#define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0)#define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0)#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) -#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) +#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) +#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4) + +#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0) +#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0) + +#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0) +#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1) +#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2) +#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1)#define __SYS__AP0Rx_EL2 (x) sys_reg(3, 4, 12, 8, x)#define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0)#define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1)
@@ -523,15 +554,24 @@#define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6)#define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7) +#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1) +#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2) + +#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) +#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) +/* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)#define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2)#define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0) +#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)#define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1)#define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2) +#define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0)#define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1) +#define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0)#define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1)#define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0)
@@ -549,6 +589,8 @@#define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1)#define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2) +#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44))#define SCTLR_ELx_ATA (BIT(43))
arch/arm64/kvm/sys_regs.c +107 -5
140 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 07/66] KVM: arm64: nv: Handle HCR_EL2.NV system register traps
On Thu, 10 Dec 2020 15:59:03 +0000
Marc Zyngier <[email protected]> wrote:
Hi,
Quoted 13 lines
From: Jintack Lim <[email protected]>
ARM v8.3 introduces a new bit in the
HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to accessSP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.This patch sets up a handler for EL2 registers and
SP_EL1register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.
Quoted 6 lines
This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.
Signed-off-by Jintack Lim <[email protected]>[maz: added SCTLR_EL2 RES0/RES1 handling]Signed-off-by Marc Zyngier <[email protected]>
I compared the system register encodings in the first hunk against the ARMv8 ARM, they are all fine.
I also checked the last hunk for (copy&paste) typos, all good as well.
Some comments below:
Quoted 125 lines
--- arch/arm64/include/asm/sysreg.h | 44 ++++++++++++- arch/arm64/kvm/sys_regs.c | 112 ++++++++++++++++++++++++++++++-- 2 files changed, 150 insertions(+), 6 deletions(-)arch/arm64/include/asm/sysreg.h +43
-1@@ -469,19 +469,50 @@#define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7) +#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0) +#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5) +#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0) +#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1) +#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0) +#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1) +#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2) +#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3) +#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7) +#define SYS_ZCR_EL2 sys_reg(3, 4, 1, 2, 0) + +#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0) +#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1) +#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) +#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) +#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0) +#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)#define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) +#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0) +#define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1) +#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0) +#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1)#define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0)#define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3)#define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0)#define SYS_TFSR_EL2 sys_reg(3, 4, 5, 6, 0)#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) -#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) +#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) +#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4) + +#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0) +#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0) + +#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0) +#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1) +#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2) +#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1)#define __SYS__AP0Rx_EL2 (x) sys_reg(3, 4, 12, 8, x)#define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0)#define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1)@@ -523,15 +554,24 @@#define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6)#define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7) +#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1) +#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2) + +#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) +#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) +/* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)#define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2)#define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0) +#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)#define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1)#define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2) +#define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0)#define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1) +#define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0)#define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1)#define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0)@@ -549,6 +589,8 @@#define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1)#define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2) +#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44))#define SCTLR_ELx_ATA (BIT(43))arch/arm64/kvm/sys_regs.c +22
-0@@ -104,6 +104,57 @@ static u32 get_ccsidr(u32 csselr)return ccsidr; } +static bool access_rw (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +vcpu_write_sys_reg (vcpu, p->regval, r->reg); +else + p->regval =vcpu_read_sys_reg (vcpu, r->reg); + +return true; +} + +static bool access_wi (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +return ignore_write (vcpu, p); + + p->regval =vcpu_read_sys_reg (vcpu, r->reg); +return true; +}
arch/arm64/kvm/sys_regs.c static u32 get_ccsidr(u32 csselr) · line 138
Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10
-0+ +static bool access_sctlr_el2 (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) { +u64 val = p->regval; + +if (vcpu_el2_e2h_is_set (vcpu) &&vcpu_el2_tge_is_set (vcpu)) { + val &= ~(GENMASK_ULL(63,45) | GENMASK_ULL(34, 32) |
In ARMv8 ARM F.c bits 49-45 are used for the TWE delay value. The manual says it's RES0 when FEAT_TWED is not implemented, but this is true for a lot of other feature bits we don't implement? So shall we let a guest set them, reducing the mask to (63,50)?
arch/arm64/kvm/sys_regs.c static u32 get_ccsidr(u32 csselr) · line 139
Quoted 1 line
arch/arm64/kvm/sys_regs.c +1
-0+ BIT_ULL(17));
The ARMv8 ARM F.c lists bit 9 as RES0 as well for (E2H,TGE)=(1,1).
Quoted 86 lines
arch/arm64/kvm/sys_regs.c +56
-5+ val |=SCTLR_EL1_RES1 ; + }else { + val &= ~(GENMASK_ULL (63 ,45 ) |BIT_ULL (42 ) | +GENMASK_ULL (39 ,38 ) |GENMASK_ULL (35 ,32 ) | +BIT_ULL (26 ) |BIT_ULL (24 ) |BIT_ULL (20 ) | +BIT_ULL (17 ) |GENMASK_ULL (15 ,14 ) | +GENMASK (10 ,7 )); + val |=SCTLR_EL2_RES1 ; + } + +vcpu_write_sys_reg (vcpu, val, r->reg); + }else { + p->regval =vcpu_read_sys_reg (vcpu, r->reg); + } + +return true; +} +/* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -342,12 +393,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu,struct sys_reg_params *p, const struct sys_reg_desc *r) { -if (p->is_write) { -vcpu_write_sys_reg (vcpu, p->regval, r->reg); +access_rw (vcpu ,p ,r ); +if (p->is_write) vcpu->arch.flags |=KVM_ARM64_DEBUG_DIRTY ; - }else { - p->regval =vcpu_read_sys_reg (vcpu, r->reg); - }trace_trap_reg (__func__, r->reg, p->is_write, p->regval);@@ -1314,6 +1362,18 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, .set_user = set_raz_id_reg, \ } +static bool access_sp_el1 (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +__vcpu_sys_reg (vcpu,SP_EL1 ) = p->regval; +else + p->regval =__vcpu_sys_reg (vcpu,SP_EL1 ); + +return true; +} +/* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -1692,9 +1752,51 @@ static const struct sys_reg_desc sys_reg_descs[] = {*/ { SYS_DESC(SYS_PMCCFILTR_EL0), access_pmu_evtyper, reset_val, PMCCFILTR_EL0, 0 }, + { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 }, + { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 }, + + { SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 }, + { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 }, + { SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 }, + { SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 }, + { SYS_DESC(SYS_CPTR_EL2), access_rw, reset_val, CPTR_EL2, CPTR_EL2_RES1 }, + { SYS_DESC(SYS_HSTR_EL2), access_rw, reset_val, HSTR_EL2, 0 }, + { SYS_DESC(SYS_HACR_EL2), access_rw, reset_val, HACR_EL2, 0 }, + + { SYS_DESC(SYS_TTBR0_EL2), access_rw, reset_val, TTBR0_EL2, 0 }, + { SYS_DESC(SYS_TTBR1_EL2), access_rw, reset_val, TTBR1_EL2, 0 }, + { SYS_DESC(SYS_TCR_EL2), access_rw, reset_val, TCR_EL2, TCR_EL2_RES1 }, + { SYS_DESC(SYS_VTTBR_EL2), access_rw, reset_val, VTTBR_EL2, 0 }, + { SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 }, + { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, + { SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 }, + { SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 }, + { SYS_DESC(SYS_SP_EL1), access_sp_el1}, + { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 }, + { SYS_DESC(SYS_AFSR0_EL2), access_rw, reset_val, AFSR0_EL2, 0 }, + { SYS_DESC(SYS_AFSR1_EL2), access_rw, reset_val, AFSR1_EL2, 0 }, + { SYS_DESC(SYS_ESR_EL2), access_rw, reset_val, ESR_EL2, 0 }, { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
arch/arm64/kvm/sys_regs.c static const struct sys_reg_desc sys_reg_descs[] = { · line 1791
Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10
-0+ + { SYS_DESC(SYS_FAR_EL2), access_rw, reset_val, FAR_EL2, 0 }, + { SYS_DESC(SYS_HPFAR_EL2), access_rw, reset_val, HPFAR_EL2, 0 }, + + { SYS_DESC(SYS_MAIR_EL2), access_rw, reset_val, MAIR_EL2, 0 }, + { SYS_DESC(SYS_AMAIR_EL2), access_rw, reset_val, AMAIR_EL2, 0 }, + + { SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 }, + { SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 }, + { SYS_DESC(SYS_RMR_EL2), access_wi, reset_val, RMR_EL2, 1 },
Why is this WI, exactly? Isn't EL2 the highest implemented EL from a guest's point of view, and thus a write to RMR_EL2 with bit 1 set should trigger a reset (to AArch64)?
I see that we don't NEED to implement this register (as we don't support AArch32), but that should look differently here then?
The rest looks alright to me.
Cheers,
Andre
arch/arm64/kvm/sys_regs.c static const struct sys_reg_desc sys_reg_descs[] = { · line 1802
Quoted 11 lines
arch/arm64/kvm/sys_regs.c +8
-0+ + { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, + { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, + + { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, + { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; static bool trap_dbgidr(struct kvm_vcpu *vcpu,
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 8/66 KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
The VMPIDR_EL2 and VPIDR_EL2 are architecturally UNKNOWN at reset, but let's be nice to a guest hypervisor behaving foolishly and reset these to something reasonable anyway.
--- arch/arm64/kvm/sys_regs.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)
arch/arm64/kvm/sys_regs.c +21 -4
@@ -621,7 +621,7 @@ static void reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)vcpu_write_sys_reg (vcpu, actlr,ACTLR_EL1 ); } -static void reset_mpidr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) +static u64 compute_reset_mpidr (struct kvm_vcpu *vcpu) {u64 mpidr;
@@ -635,7 +635,24 @@ static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) mpidr = (vcpu->vcpu_id &0x0f ) <<MPIDR_LEVEL_SHIFT (0 ); mpidr |= ((vcpu->vcpu_id >>4 ) &0xff ) <<MPIDR_LEVEL_SHIFT (1 ); mpidr |= ((vcpu->vcpu_id >>12 ) &0xff ) <<MPIDR_LEVEL_SHIFT (2 ); -vcpu_write_sys_reg (vcpu, (1ULL <<31 ) | mpidr,MPIDR_EL1 ); + mpidr |= (1ULL <<31 ); + +return mpidr; +} + +static void reset_mpidr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) +{ +vcpu_write_sys_reg (vcpu,compute_reset_mpidr (vcpu),MPIDR_EL1 ); +} + +static void reset_vmpidr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) +{ +vcpu_write_sys_reg (vcpu,compute_reset_mpidr (vcpu),VMPIDR_EL2 ); +} + +static void reset_vpidr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r) +{ +vcpu_write_sys_reg (vcpu,read_cpuid_id (),VPIDR_EL2 ); }static void reset_pmcr (struct kvm_vcpu *vcpu,const struct sys_reg_desc *r)
@@ -1752,8 +1769,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {*/ { SYS_DESC(SYS_PMCCFILTR_EL0), access_pmu_evtyper, reset_val, PMCCFILTR_EL0, 0 }, - { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 }, - { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 }, + { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_vpidr, VPIDR_EL2 }, + { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_vmpidr, VMPIDR_EL2 }, { SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 }, { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 },
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 9/66 KVM: arm64: nv: Support virtual EL2 exceptions
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).
--- arch/arm64/include/asm/kvm_arm.h | 17 +++ arch/arm64/include/asm/kvm_emulate.h | 10 ++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/emulate-nested.c | 176 +++++++++++++++++++++++++++ arch/arm64/kvm/hyp/exception.c | 44 +++++-- arch/arm64/kvm/inject_fault.c | 62 ++++++++-- arch/arm64/kvm/trace_arm.h | 59 +++++++++ 7 files changed, 352 insertions(+), 18 deletions(-) create mode 100644 arch/arm64/kvm/emulate-nested.c
arch/arm64/include/asm/kvm_arm.h +17 -0
@@ -332,4 +332,21 @@#define CPACR_EL1_TTA (1 << 28)#define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN) +#define kvm_mode_names \ + { PSR_MODE_EL0t, "EL0t" }, \ + { PSR_MODE_EL1t, "EL1t" }, \ + { PSR_MODE_EL1h, "EL1h" }, \ + { PSR_MODE_EL2t, "EL2t" }, \ + { PSR_MODE_EL2h, "EL2h" }, \ + { PSR_MODE_EL3t, "EL3t" }, \ + { PSR_MODE_EL3h, "EL3h" }, \ + { PSR_AA32_MODE_USR, "32-bit USR" }, \ + { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \ + { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \ + { PSR_AA32_MODE_SVC, "32-bit SVC" }, \ + { PSR_AA32_MODE_ABT, "32-bit ABT" }, \ + { PSR_AA32_MODE_HYP, "32-bit HYP" }, \ + { PSR_AA32_MODE_UND, "32-bit UND" }, \ + { PSR_AA32_MODE_SYS, "32-bit SYS" } +#endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +10 -0
@@ -33,6 +33,12 @@ enum exception_type { except_type_serror = 0x180, }; +#define kvm_exception_type_names \ + { except_type_sync, "SYNC" }, \ + { except_type_irq, "IRQ" }, \ + { except_type_fiq, "FIQ" }, \ + { except_type_serror, "SERROR" } +bool kvm_condition_valid32 (const struct kvm_vcpu *vcpu);void kvm_skip_instr32 (struct kvm_vcpu *vcpu);
@@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);void kvm_inject_dabt (struct kvm_vcpu *vcpu,unsigned long addr);void kvm_inject_pabt (struct kvm_vcpu *vcpu,unsigned long addr); +void kvm_emulate_nested_eret (struct kvm_vcpu *vcpu); +int kvm_inject_nested_sync (struct kvm_vcpu *vcpu,u64 esr_el2); +int kvm_inject_nested_irq (struct kvm_vcpu *vcpu); + static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) {return !(vcpu->arch.hcr_el2 &HCR_RW );
arch/arm64/kvm/Makefile +1 -1
@@ -17,11 +17,11 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ arch_timer.o \ + emulate-nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o - kvm-$(CONFIG_KVM_ARM_PMU) += pmu-emul.o
arch/arm64/kvm/emulate-nested.c new file +176 -0
176 lines not on this page · raw message
arch/arm64/kvm/hyp/exception.c +37 -7
82 lines not on this page · raw message
arch/arm64/kvm/inject_fault.c +52 -10
92 lines not on this page · raw message
arch/arm64/kvm/trace_arm.h +59 -0
71 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 09/66] KVM: arm64: nv: Support virtual EL2 exceptions
On Thu, 10 Dec 2020 15:59:05 +0000
Marc Zyngier <[email protected]> wrote:
Hi,
Quoted 371 lines
From: Jintack Lim <[email protected]>
Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).
Signed-off-by Jintack Lim <[email protected]>Signed-off-by Christoffer Dall <[email protected]>[maz: switch to common exception injection framework]Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_arm.h | 17 +++ arch/arm64/include/asm/kvm_emulate.h | 10 ++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/emulate-nested.c | 176 +++++++++++++++++++++++++++ arch/arm64/kvm/hyp/exception.c | 44 +++++-- arch/arm64/kvm/inject_fault.c | 62 ++++++++-- arch/arm64/kvm/trace_arm.h | 59 +++++++++ 7 files changed, 352 insertions(+), 18 deletions(-) create mode 100644 arch/arm64/kvm/emulate-nested.carch/arm64/include/asm/kvm_arm.h +17
-0@@ -332,4 +332,21 @@#define CPACR_EL1_TTA (1 << 28)#define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN) +#define kvm_mode_names \ + { PSR_MODE_EL0t, "EL0t" }, \ + { PSR_MODE_EL1t, "EL1t" }, \ + { PSR_MODE_EL1h, "EL1h" }, \ + { PSR_MODE_EL2t, "EL2t" }, \ + { PSR_MODE_EL2h, "EL2h" }, \ + { PSR_MODE_EL3t, "EL3t" }, \ + { PSR_MODE_EL3h, "EL3h" }, \ + { PSR_AA32_MODE_USR, "32-bit USR" }, \ + { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \ + { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \ + { PSR_AA32_MODE_SVC, "32-bit SVC" }, \ + { PSR_AA32_MODE_ABT, "32-bit ABT" }, \ + { PSR_AA32_MODE_HYP, "32-bit HYP" }, \ + { PSR_AA32_MODE_UND, "32-bit UND" }, \ + { PSR_AA32_MODE_SYS, "32-bit SYS" } +#endif /* __ARM64_KVM_ARM_H__ */ arch/arm64/include/asm/kvm_emulate.h +10
-0@@ -33,6 +33,12 @@ enum exception_type { except_type_serror = 0x180, }; +#define kvm_exception_type_names \ + { except_type_sync, "SYNC" }, \ + { except_type_irq, "IRQ" }, \ + { except_type_fiq, "FIQ" }, \ + { except_type_serror, "SERROR" } +bool kvm_condition_valid32 (const struct kvm_vcpu *vcpu);void kvm_skip_instr32 (struct kvm_vcpu *vcpu);@@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);void kvm_inject_dabt (struct kvm_vcpu *vcpu,unsigned long addr);void kvm_inject_pabt (struct kvm_vcpu *vcpu,unsigned long addr); +void kvm_emulate_nested_eret (struct kvm_vcpu *vcpu); +int kvm_inject_nested_sync (struct kvm_vcpu *vcpu,u64 esr_el2); +int kvm_inject_nested_irq (struct kvm_vcpu *vcpu); + static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) {return !(vcpu->arch.hcr_el2 &HCR_RW );arch/arm64/kvm/Makefile +1
-1@@ -17,11 +17,11 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ arch_timer.o \ + emulate-nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o - kvm-$(CONFIG_KVM_ARM_PMU) += pmu-emul.oarch/arm64/kvm/emulate-nested.c new file +176
-0@@ -0,0 +1,176 @@ +/* +* Copyright (C) 2016 - Linaro and Columbia University +* Author: Jintack Lim <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <linux/kvm.h> +#include <linux/kvm_host.h> + +#include <asm/kvm_emulate.h> +#include <asm/kvm_nested.h> + +#include "hyp/include/hyp/adjust_pc.h" + +#include "trace.h" + +void kvm_emulate_nested_eret (struct kvm_vcpu *vcpu) +{ +u64 spsr, elr, mode; +bool direct_eret; + +/* +* Going through the whole put/load motions is a waste of time +* if this is a VHE guest hypervisor returning to its own +* userspace, or the hypervisor performing a local exception +* return. No need to save/restore registers, no need to +* switch S2 MMU. Just do the canonical ERET. +*/ + spsr =vcpu_read_sys_reg (vcpu,SPSR_EL2 ); + mode = spsr & (PSR_MODE_MASK |PSR_MODE32_BIT ); + + direct_eret = (mode == PSR_MODE_EL0t && +vcpu_el2_e2h_is_set (vcpu) && +vcpu_el2_tge_is_set (vcpu)); + direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + +if (direct_eret) { + *vcpu_pc (vcpu) =vcpu_read_sys_reg (vcpu,ELR_EL2 ); + *vcpu_cpsr (vcpu) = spsr; +trace_kvm_nested_eret (vcpu, *vcpu_pc (vcpu), spsr); +return ; + } + +preempt_disable (); +kvm_arch_vcpu_put (vcpu); + + elr =__vcpu_sys_reg (vcpu,ELR_EL2 ); + +trace_kvm_nested_eret (vcpu, elr, spsr); + +/* +* Note that the current exception level is always the virtual EL2, +* since we set HCR_EL2.NV bit only when entering the virtual EL2. +*/ + *vcpu_pc (vcpu) = elr; + *vcpu_cpsr (vcpu) = spsr; + +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); +} + +static void kvm_inject_el2_exception (struct kvm_vcpu *vcpu,u64 esr_el2, +enum exception_type type) +{ +trace_kvm_inject_nested_exception (vcpu, esr_el2, type); + +switch (type) { +case except_type_sync: + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_SYNC; +break ; +case except_type_irq: + vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_IRQ; +break ; +default : +WARN_ONCE (1 ,"Unsupported EL2 exception injection %d\n" , type); + } + + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL2 | +KVM_ARM64_PENDING_EXCEPTION ); + +vcpu_write_sys_reg (vcpu, esr_el2,ESR_EL2 ); +} + +/* +* Emulate taking an exception to EL2. +* See ARM ARM J8.1.2 AArch64.TakeException() +*/ +static int kvm_inject_nested (struct kvm_vcpu *vcpu,u64 esr_el2, +enum exception_type type) +{ +u64 pstate, mode; +bool direct_inject; + +if (!nested_virt_in_use (vcpu)) { +kvm_err ("Unexpected call to %s for the non-nesting configuration\n" , + __func__); +return -EINVAL ; + } + +/* +* As for ERET, we can avoid doing too much on the injection path by +* checking that we either took the exception from a VHE host +* userspace or from vEL2. In these cases, there is no change in +* translation regime (or anything else), so let's do as little as +* possible. +*/ + pstate = *vcpu_cpsr (vcpu); + mode = pstate & (PSR_MODE_MASK |PSR_MODE32_BIT ); + + direct_inject = (mode == PSR_MODE_EL0t && +vcpu_el2_e2h_is_set (vcpu) && +vcpu_el2_tge_is_set (vcpu)); + direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + +if (direct_inject) { +kvm_inject_el2_exception (vcpu, esr_el2, type); +return 1 ; + } + +preempt_disable (); +kvm_arch_vcpu_put (vcpu); + +kvm_inject_el2_exception (vcpu, esr_el2, type); + +/* +* A hard requirement is that a switch between EL1 and EL2 +* contexts has to happen between a put/load, so that we can +* pick the correct timer and interrupt configuration, among +* other things. +* +* Make sure the exception actually took place before we load +* the new context. +*/ +__adjust_pc (vcpu); + +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); + +return 1 ; +} + +int kvm_inject_nested_sync (struct kvm_vcpu *vcpu,u64 esr_el2) +{ +return kvm_inject_nested (vcpu, esr_el2, except_type_sync); +} + +int kvm_inject_nested_irq (struct kvm_vcpu *vcpu) +{ +/* +* Do not inject an irq if the: +* - Current exception level is EL2, and +* - virtual HCR_EL2.TGE == 0 +* - virtual HCR_EL2.IMO == 0 +* +* See Table D1-17 "Physical interrupt target and masking when EL3 is +* not implemented and EL2 is implemented" in ARM DDI 0487C.a. +*/ + +if (vcpu_mode_el2 (vcpu) && !vcpu_el2_tge_is_set (vcpu) && + !(__vcpu_sys_reg (vcpu,HCR_EL2 ) &HCR_IMO )) +return 1 ; + +/* esr_el2 value doesn't matter for exits due to irqs. */ +return kvm_inject_nested (vcpu,0 , except_type_irq); +}arch/arm64/kvm/hyp/ renamed from arch/arm64/kvm/hyp/exception.c +37
-7@@ -22,7 +22,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { u64 val; - if (__vcpu_read_sys_reg_from_cpu(reg, &val)) + if (unlikely(nested_virt_in_use(vcpu))) + return vcpu_read_sys_reg(vcpu, reg); + else if (__vcpu_read_sys_reg_from_cpu(reg, &val)) return val; return __vcpu_sys_reg(vcpu, reg);@@ -30,14 +32,26 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) { - if (__vcpu_write_sys_reg_to_cpu(val, reg)) + if (unlikely(nested_virt_in_use(vcpu))) + vcpu_write_sys_reg(vcpu, val, reg); + else if (__vcpu_write_sys_reg_to_cpu(val, reg)) return; __vcpu_sys_reg(vcpu, reg) = val; } -static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val) +static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode, + u64 val) { + if (unlikely(nested_virt_in_use(vcpu))) { + if (target_mode == PSR_MODE_EL1h) + vcpu_write_sys_reg(vcpu, val, SPSR_EL1); + else + vcpu_write_sys_reg(vcpu, val, SPSR_EL2); + + return; + } + write_sysreg_el1(val, SYS_SPSR); }@@ -97,6 +111,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1); __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1); break; + case PSR_MODE_EL2h: + vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2); + sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2); + __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); + break; default: /* Don't do that */ BUG();@@ -148,7 +167,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, new |= target_mode; *vcpu_cpsr(vcpu) = new; - __vcpu_write_spsr(vcpu, old); + __vcpu_write_spsr(vcpu, target_mode, old); } /*@@ -319,11 +338,22 @@ void kvm_inject_exception(struct kvm_vcpu *vcpu) KVM_ARM64_EXCEPT_AA64_EL1): enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync); break; + + case (KVM_ARM64_EXCEPT_AA64_ELx_SYNC | + KVM_ARM64_EXCEPT_AA64_EL2): + enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync); + break; + + case (KVM_ARM64_EXCEPT_AA64_ELx_IRQ | + KVM_ARM64_EXCEPT_AA64_EL2): + enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq); + break; + default: /* - * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ} - * will be implemented at some point. Everything - * else gets silently ignored. + * Only EL1_SYNC and EL2_{SYNC,IRQ} makes + * sense so far. Everything else gets silently + * ignored. */ break; }arch/arm64/kvm/inject_fault.c +0
-0
arch/arm64/kvm/inject_fault.c line 23
Quoted 10 lines
arch/arm64/kvm/inject_fault.c +7
-0@@ -14,17 +14,50 @@#include <asm/kvm_emulate.h> #include <asm/esr.h> +static void pend_sync_exception (struct kvm_vcpu *vcpu) +{ + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC | +KVM_ARM64_PENDING_EXCEPTION ); + +/* If not nesting, EL1 is the only possible exception target */ +if (likely (!nested_virt_in_use (vcpu))) {
This breaks compilation for the next 14 patches, because nested_virt_in_use() is unknown here. Patch 23/66 fixes this by adding kvm_nested.h to kvm_emulate.h.
Shall we do this already in this patch here, or pull in kvm_nested.h in every C file we use nested_virt_in_use(), like exception.c above?
Cheers,
Andre
Quoted 151 lines
arch/arm64/kvm/inject_fault.c +45
-10+ vcpu->arch.flags |=KVM_ARM64_EXCEPT_AA64_EL1 ; +return ; + } + +/* +* With NV, we need to pick between EL1 and EL2. Note that we +* never deal with a nesting exception here, hence never +* changing context, and the exception itself can be delayed +* until the next entry. +*/ +switch (*vcpu_cpsr (vcpu) &PSR_MODE_MASK ) { +case PSR_MODE_EL2h: +case PSR_MODE_EL2t: + vcpu->arch.flags |=KVM_ARM64_EXCEPT_AA64_EL2 ; +break ; +case PSR_MODE_EL1h: +case PSR_MODE_EL1t: + vcpu->arch.flags |=KVM_ARM64_EXCEPT_AA64_EL1 ; +break ; +case PSR_MODE_EL0t: +if (vcpu_el2_tge_is_set (vcpu) &HCR_TGE ) + vcpu->arch.flags |=KVM_ARM64_EXCEPT_AA64_EL2 ; +else + vcpu->arch.flags |=KVM_ARM64_EXCEPT_AA64_EL1 ; +break ; +default : +BUG (); + } +} +static void inject_abt64 (struct kvm_vcpu *vcpu,bool is_iabt,unsigned long addr) {unsigned long cpsr = *vcpu_cpsr (vcpu);bool is_aarch32 =vcpu_mode_is_32bit (vcpu);u32 esr =0 ; - vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1 | - KVM_ARM64_EXCEPT_AA64_ELx_SYNC | -KVM_ARM64_PENDING_EXCEPTION ); - -vcpu_write_sys_reg (vcpu, addr,FAR_EL1 ); +pend_sync_exception (vcpu);/* * Build an {i,d}abort, depending on the level and the @@ -45,16 +78,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addrif (!is_iabt) esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT; -vcpu_write_sys_reg (vcpu, esr | ESR_ELx_FSC_EXTABT,ESR_EL1 ); + esr |= ESR_ELx_FSC_EXTABT; + +if (vcpu->arch.flags &KVM_ARM64_EXCEPT_AA64_EL1 ) { +vcpu_write_sys_reg (vcpu, addr,FAR_EL1 ); +vcpu_write_sys_reg (vcpu, esr,ESR_EL1 ); + }else { +vcpu_write_sys_reg (vcpu, addr,FAR_EL2 ); +vcpu_write_sys_reg (vcpu, esr,ESR_EL2 ); + } }static void inject_undef64 (struct kvm_vcpu *vcpu) {u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT); - vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1 | - KVM_ARM64_EXCEPT_AA64_ELx_SYNC | -KVM_ARM64_PENDING_EXCEPTION ); +pend_sync_exception (vcpu);/* * Build an unknown exception, depending on the instruction @@ -63,7 +102,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu)if (kvm_vcpu_trap_il_is32bit (vcpu)) esr |= ESR_ELx_IL; -vcpu_write_sys_reg (vcpu, esr,ESR_EL1 ); +if (vcpu->arch.flags &KVM_ARM64_EXCEPT_AA64_EL1 ) +vcpu_write_sys_reg (vcpu, esr,ESR_EL1 ); +else +vcpu_write_sys_reg (vcpu, esr,ESR_EL2 ); }#define DFSR_FSC_EXTABT_LPAE 0x10arch/arm64/kvm/trace_arm.h +52
-0@@ -2,6 +2,7 @@#if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ )#define _TRACE_ARM_ARM64_KVM_H +#include <asm/kvm_emulate.h> #include <kvm/arm_arch_timer.h> #include <linux/tracepoint.h> @@ -367,6 +368,64 @@ TRACE_EVENT(kvm_timer_emulate, __entry->timer_idx, __entry->should_fire) ); +TRACE_EVENT (kvm_nested_eret, + TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2, + unsigned long spsr_el2), +TP_ARGS (vcpu, elr_el2, spsr_el2), + +TP_STRUCT__entry ( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, elr_el2) + __field(unsigned long, spsr_el2) + __field(unsigned long, target_mode) + __field(unsigned long, hcr_el2) + ), + +TP_fast_assign ( + __entry->vcpu = vcpu; + __entry->elr_el2 = elr_el2; + __entry->spsr_el2 = spsr_el2; + __entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + +TP_printk ("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx" , + __entry->elr_el2, __entry->spsr_el2, +__print_symbolic (__entry->target_mode, kvm_mode_names), + __entry->hcr_el2) +); + +TRACE_EVENT(kvm_inject_nested_exception, + TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type), + TP_ARGS(vcpu, esr_el2, type), + + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, esr_el2) + __field(int, type) + __field(unsigned long, spsr_el2) + __field(unsigned long, pc) + __field(unsigned long, source_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->esr_el2 = esr_el2; + __entry->type = type; + __entry->spsr_el2 = *vcpu_cpsr(vcpu); + __entry->pc = *vcpu_pc(vcpu); + __entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), +
arch/arm64/kvm/trace_arm.h TRACE_EVENT(kvm_timer_emulate, · line 431
Quoted 10 lines
arch/arm64/kvm/trace_arm.h +7
-0+TP_printk ("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx" , +__print_symbolic (__entry->type, kvm_exception_type_names), + __entry->esr_el2, __entry->pc, __entry->spsr_el2, +__print_symbolic (__entry->source_mode, kvm_mode_names), + __entry->hcr_el2) +); +#endif /* _TRACE_ARM_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 09/66] KVM: arm64: nv: Support virtual EL2 exceptions
Quoted 3 lines · Andre Przywara
On Thu, 10 Dec 2020 15:59:05 +0000
Marc Zyngier <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 8 lines
arch/arm64/kvm/inject_fault.c +0
-0@@ -14,17 +14,50 @@#include <asm/kvm_emulate.h> #include <asm/esr.h>
Quoted 11 lines
arch/arm64/kvm/inject_fault.c line 23
Quoted 7 lines
arch/arm64/kvm/inject_fault.c +7
-0+static void pend_sync_exception (struct kvm_vcpu *vcpu) +{ + vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC | +KVM_ARM64_PENDING_EXCEPTION ); + +/* If not nesting, EL1 is the only possible exception target */ +if (likely (!nested_virt_in_use (vcpu))) {This breaks compilation for the next 14 patches, because
nested_virt_in_use()is unknown here. Patch 23/66 fixes this by addingkvm_nested.htokvm_emulate.h.
Indeed, well caught.
Quoted 2 lines
Shall we do this already in this patch here, or pull in
kvm_nested.hin every C file we usenested_virt_in_use(), likeexception.cabove?
I've just added kvm_nested.h in the two places that were required at this stage (and verified that it all bisects correctly now).
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 10/66 KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2.
Forward these to EL2 as required.
--- arch/arm64/kvm/handle_exit.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/arm64/kvm/handle_exit.c +11 -0
@@ -16,6 +16,7 @@#include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include <asm/debug-monitors.h> #include <asm/traps.h>
@@ -40,6 +41,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu) kvm_vcpu_hvc_get_imm(vcpu)); vcpu->stat.hvc_exit_stat++; +/* Forward hvc instructions to the virtual EL2 if the guest has EL2. */ +if (nested_virt_in_use (vcpu)) { +if (vcpu_read_sys_reg (vcpu,HCR_EL2 ) &HCR_HCD ) +kvm_inject_undefined (vcpu); +else +kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu)); + +return 1 ; + } + ret =kvm_hvc_call_handler (vcpu);if (ret <0 ) {vcpu_set_reg (vcpu,0 , ~0UL );
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 11/66 KVM: arm64: nv: Handle trapped ERET from virtual EL2
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software.
--- arch/arm64/include/asm/esr.h | 5 +++++ arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/handle_exit.c | 10 ++++++++++ 3 files changed, 16 insertions(+), 1 deletion(-)
arch/arm64/include/asm/esr.h +5 -0
@@ -251,6 +251,11 @@ (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \ ESR_ELx_SYS64_ISS_OP2_SHIFT)) +/* ISS field definitions for ERET/ERETAA/ERETAB trapping */ + +#define ESR_ELx_ERET_ISS_ERET_ERETAx 0x2 +#define ESR_ELx_ERET_ISS_ERETA_ERATAB 0x1 +/* * ISS field definitions for floating-point exception traps * (FP_EXC_32/FP_EXC_64).
arch/arm64/include/asm/kvm_arm.h +1 -1
@@ -326,7 +326,7 @@ECN (SP_ALIGN ),ECN (FP_EXC32 ),ECN (FP_EXC64 ),ECN (SERROR ), \ECN (BREAKPT_LOW ),ECN (BREAKPT_CUR ),ECN (SOFTSTP_LOW ), \ECN (SOFTSTP_CUR ),ECN (WATCHPT_LOW ),ECN (WATCHPT_CUR ), \ - ECN(BKPT32), ECN(VECTOR32), ECN(BRK64) + ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET)#define CPACR_EL1_FPEN (3 << 20)#define CPACR_EL1_TTA (1 << 28)
arch/arm64/kvm/handle_exit.c +10 -0
@@ -183,6 +183,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu)return 1 ; } +static int kvm_handle_eret (struct kvm_vcpu *vcpu) +{ +if (kvm_vcpu_get_esr (vcpu) & ESR_ELx_ERET_ISS_ERET_ERETAx) +return kvm_handle_ptrauth (vcpu); + +kvm_emulate_nested_eret (vcpu); +return 1 ; +} +static exit_handle_fn arm_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec, [ESR_ELx_EC_WFx] = kvm_handle_wfx,
@@ -197,6 +206,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC64] = handle_smc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, [ESR_ELx_EC_SVE] = handle_sve, + [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 12/66 KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
In reply to Marc ZyngierSome EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. In general, this only affects non-VHE guest hypervisors, as VHE system registers are compatible with the EL1 counterparts.
These helpers will get used in subsequent patches.
--- arch/arm64/include/asm/kvm_nested.h | 50 +++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
arch/arm64/include/asm/kvm_nested.h +50 -0
@@ -2,6 +2,7 @@#ifndef __ARM64_KVM_NESTED_H#define __ARM64_KVM_NESTED_H +#include <linux/bitfield.h> #include <linux/kvm_host.h> static inline bool nested_virt_in_use (const struct kvm_vcpu *vcpu)
@@ -11,4 +12,53 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +/* Translation helpers from non-VHE EL2 to EL1 */ +static inline u64 tcr_el2_ips_to_tcr_el1_ps (u64 tcr_el2) +{ +return (u64 )FIELD_GET (TCR_EL2_PS_MASK , tcr_el2) <<TCR_IPS_SHIFT ; +} + +static inline u64 translate_tcr_el2_to_tcr_el1 (u64 tcr) +{ +return TCR_EPD1_MASK |/* disable TTBR1_EL1 */ + ((tcr &TCR_EL2_TBI ) ?TCR_TBI0 :0 ) | +tcr_el2_ips_to_tcr_el1_ps (tcr) | + (tcr &TCR_EL2_TG0_MASK ) | + (tcr &TCR_EL2_ORGN0_MASK ) | + (tcr &TCR_EL2_IRGN0_MASK ) | + (tcr &TCR_EL2_T0SZ_MASK ); +} + +static inline u64 translate_cptr_el2_to_cpacr_el1 (u64 cptr_el2) +{ +u64 cpacr_el1 =0 ; + +if (!(cptr_el2 &CPTR_EL2_TFP )) + cpacr_el1 |=CPACR_EL1_FPEN ; +if (cptr_el2 &CPTR_EL2_TTA ) + cpacr_el1 |=CPACR_EL1_TTA ; +if (!(cptr_el2 &CPTR_EL2_TZ )) + cpacr_el1 |=CPACR_EL1_ZEN ; + +return cpacr_el1; +} + +static inline u64 translate_sctlr_el2_to_sctlr_el1 (u64 sctlr) +{ +/* Bit 20 is RES1 in SCTLR_EL1, but RES0 in SCTLR_EL2 */ +return sctlr |BIT (20 ); +} + +static inline u64 translate_ttbr0_el2_to_ttbr0_el1 (u64 ttbr0) +{ +/* Force ASID to 0 (ASID 0 or RES0) */ +return ttbr0 & ~GENMASK_ULL (63 ,48 ); +} + +static inline u64 translate_cnthctl_el2_to_cntkctl_el1 (u64 cnthctl) +{ +return ((FIELD_GET (CNTHCTL_EL1PCTEN |CNTHCTL_EL1PCEN , cnthctl) <<10 ) | + (cnthctl & (CNTHCTL_EVNTI |CNTHCTL_EVNTDIR |CNTHCTL_EVNTEN ))); +} +#endif /* __ARM64_KVM_NESTED_H */
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 13/66 KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()
In reply to Marc ZyngierKVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.
With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must also use the stored version, since the CPU carries the EL1 copy.
- Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map.
- Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map.
- There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch).
All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in.
This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL.
--- arch/arm64/kvm/sys_regs.c | 144 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 140 insertions(+), 4 deletions(-)
arch/arm64/kvm/sys_regs.c +140 -4
@@ -23,6 +23,7 @@#include <asm/kvm_emulate.h> #include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include <asm/perf_event.h> #include <asm/sysreg.h>
@@ -63,23 +64,158 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu,return false; } +#define PURE_EL2_SYSREG (el2) \ + case el2: { \ + *el1r = el2; \ + return true; \ + } + +#define MAPPED_EL2_SYSREG (el2, el1, fn) \ + case el2: { \ + *xlate = fn; \ + *el1r = el1; \ + return true; \ + } + +static bool get_el2_mapping (unsigned int reg, +unsigned int *el1r,u64 (**xlate)(u64 )) +{ +switch (reg) { +PURE_EL2_SYSREG (VPIDR_EL2 ); +PURE_EL2_SYSREG (VMPIDR_EL2 ); +PURE_EL2_SYSREG (ACTLR_EL2 ); +PURE_EL2_SYSREG (HCR_EL2 ); +PURE_EL2_SYSREG (MDCR_EL2 ); +PURE_EL2_SYSREG (HSTR_EL2 ); +PURE_EL2_SYSREG (HACR_EL2 ); +PURE_EL2_SYSREG (VTTBR_EL2 ); +PURE_EL2_SYSREG (VTCR_EL2 ); +PURE_EL2_SYSREG (RVBAR_EL2 ); +PURE_EL2_SYSREG (RMR_EL2 ); +PURE_EL2_SYSREG (TPIDR_EL2 ); +PURE_EL2_SYSREG (HPFAR_EL2 ); +PURE_EL2_SYSREG (ELR_EL2 ); +PURE_EL2_SYSREG (SPSR_EL2 ); +MAPPED_EL2_SYSREG (SCTLR_EL2 ,SCTLR_EL1 , + translate_sctlr_el2_to_sctlr_el1 ); +MAPPED_EL2_SYSREG (CPTR_EL2 ,CPACR_EL1 , + translate_cptr_el2_to_cpacr_el1 ); +MAPPED_EL2_SYSREG (TTBR0_EL2 ,TTBR0_EL1 , + translate_ttbr0_el2_to_ttbr0_el1 ); +MAPPED_EL2_SYSREG (TTBR1_EL2 ,TTBR1_EL1 ,NULL ); +MAPPED_EL2_SYSREG (TCR_EL2 ,TCR_EL1 , + translate_tcr_el2_to_tcr_el1 ); +MAPPED_EL2_SYSREG (VBAR_EL2 ,VBAR_EL1 ,NULL ); +MAPPED_EL2_SYSREG (AFSR0_EL2 ,AFSR0_EL1 ,NULL ); +MAPPED_EL2_SYSREG (AFSR1_EL2 ,AFSR1_EL1 ,NULL ); +MAPPED_EL2_SYSREG (ESR_EL2 ,ESR_EL1 ,NULL ); +MAPPED_EL2_SYSREG (FAR_EL2 ,FAR_EL1 ,NULL ); +MAPPED_EL2_SYSREG (MAIR_EL2 ,MAIR_EL1 ,NULL ); +MAPPED_EL2_SYSREG (AMAIR_EL2 ,AMAIR_EL1 ,NULL ); +MAPPED_EL2_SYSREG (CNTHCTL_EL2 ,CNTKCTL_EL1 , + translate_cnthctl_el2_to_cntkctl_el1 ); +default : +return false; + } +} +u64 vcpu_read_sys_reg (const struct kvm_vcpu *vcpu,int reg) {u64 val =0x8badf00d8badf00d ; +u64 (*xlate)(u64) =NULL ; +unsigned int el1r; + +if (!vcpu->arch.sysregs_loaded_on_cpu) + goto memory_read; + +if (unlikely (get_el2_mapping (reg, &el1r, &xlate))) { +if (!is_hyp_ctxt (vcpu)) + goto memory_read; + +/* +* ELR_EL2 is special cased for now. +*/ +switch (reg) { +case ELR_EL2 : +return read_sysreg_el1 (SYS_ELR ); + } + +/* +* If this register does not have an EL1 counterpart, +* then read the stored EL2 version. +*/ +if (reg == el1r) + goto memory_read; + +/* +* If we have a non-VHE guest and that the sysreg +* requires translation to be used at EL1, use the +* in-memory copy instead. +*/ +if (!vcpu_el2_e2h_is_set (vcpu) && xlate) + goto memory_read; + +/* Get the current version of the EL1 counterpart. */ +WARN_ON (!__vcpu_read_sys_reg_from_cpu (el1r, &val)); +return val; + } + +/* EL1 register can't be on the CPU if the guest is in vEL2. */ +if (unlikely (is_hyp_ctxt (vcpu))) + goto memory_read; -if (vcpu->arch.sysregs_loaded_on_cpu && -__vcpu_read_sys_reg_from_cpu (reg, &val)) +if (__vcpu_read_sys_reg_from_cpu (reg, &val))return val; +memory_read:return __vcpu_sys_reg (vcpu, reg); }void vcpu_write_sys_reg (struct kvm_vcpu *vcpu,u64 val,int reg) { -if (vcpu->arch.sysregs_loaded_on_cpu && -__vcpu_write_sys_reg_to_cpu (val, reg)) +u64 (*xlate)(u64) =NULL ; +unsigned int el1r; + +if (!vcpu->arch.sysregs_loaded_on_cpu) + goto memory_write; + +if (unlikely (get_el2_mapping (reg, &el1r, &xlate))) { +if (!is_hyp_ctxt (vcpu)) + goto memory_write; + +/* +* Always store a copy of the write to memory to avoid having +* to reverse-translate virtual EL2 system registers for a +* non-VHE guest hypervisor. +*/ +__vcpu_sys_reg (vcpu, reg) = val; + +switch (reg) { +case ELR_EL2 : +write_sysreg_el1 (val,SYS_ELR ); +return ; + } + +/* No EL1 counterpart? We're done here.? */ +if (reg == el1r) +return ; + +if (!vcpu_el2_e2h_is_set (vcpu) && xlate) + val =xlate (val); + +/* Redirect this to the EL1 version of the register. */ +WARN_ON (!__vcpu_write_sys_reg_to_cpu (val, el1r)); +return ; + } + +/* EL1 register can't be on the CPU if the guest is in vEL2. */ +if (unlikely (is_hyp_ctxt (vcpu))) + goto memory_write; + +if (__vcpu_write_sys_reg_to_cpu (val, reg))return ; +memory_write:__vcpu_sys_reg (vcpu, reg) = val; }
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 14/66 KVM: arm64: nv: Handle SPSR_EL2 specially
In reply to Marc ZyngierSPSR_EL2 needs special attention when running nested on ARMv8.3:
If taking an exception while running at vEL2 (actually EL1), the HW will update the SPSR_EL1 register with the EL1 mode. We need to track this in order to make sure that accesses to the virtual view of SPSR_EL2 is correct.
To do so, we place an illegal value in SPSR_EL1.M, and patch it accordingly if required when accessing it.
--- arch/arm64/include/asm/kvm_emulate.h | 37 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 23 +++++++++++++++-- 2 files changed, 58 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +37 -0
@@ -238,6 +238,43 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)return __is_hyp_ctxt (&vcpu->arch.ctxt); } +static inline u64 __fixup_spsr_el2_write (struct kvm_cpu_context *ctxt,u64 val) +{ +if (!__vcpu_el2_e2h_is_set (ctxt)) { +/* +* Clear the .M field when writing SPSR to the CPU, so that we +* can detect when the CPU clobbered our SPSR copy during a +* local exception. +*/ + val &= ~0xc ; + } + +return val; +} + +static inline u64 __fixup_spsr_el2_read (const struct kvm_cpu_context *ctxt,u64 val) +{ +if (__vcpu_el2_e2h_is_set (ctxt)) +return val; + +/* +* SPSR.M == 0 means the CPU has not touched the SPSR, so the +* register has still the value we saved on the last write. +*/ +if ((val &0xc ) ==0 ) +return ctxt_sys_reg (ctxt,SPSR_EL2 ); + +/* +* Otherwise there was a "local" exception on the CPU, +* which from the guest's point of view was being taken from +* EL2 to EL2, although it actually happened to be from +* EL1 to EL1. +* So we need to fix the .M field in SPSR, to make it look +* like EL2, which is what the guest would expect. +*/ +return (val & ~0x0c ) | CurrentEL_EL2; +} +/* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
arch/arm64/kvm/sys_regs.c +21 -2
@@ -133,11 +133,14 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) goto memory_read;/* -* ELR_EL2 is special cased for now. +* ELR_EL2 and SPSR_EL2 are special cased for now. */ switch (reg) {case ELR_EL2 :return read_sysreg_el1 (SYS_ELR ); +case SPSR_EL2 : + val =read_sysreg_el1 (SYS_SPSR ); +return __fixup_spsr_el2_read (&vcpu->arch.ctxt, val); }/*
@@ -194,6 +197,10 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)case ELR_EL2 :write_sysreg_el1 (val,SYS_ELR );return ; +case SPSR_EL2 : + val =__fixup_spsr_el2_write (&vcpu->arch.ctxt, val); +write_sysreg_el1 (val,SYS_SPSR ); +return ; }/* No EL1 counterpart? We're done here.? */
@@ -1527,6 +1534,18 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,return true; } +static bool access_spsr_el2 (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +vcpu_write_sys_reg (vcpu, p->regval,SPSR_EL2 ); +else + p->regval =vcpu_read_sys_reg (vcpu,SPSR_EL2 ); + +return true; +} +/* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -1923,7 +1942,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 }, { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, - { SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 }, + { SYS_DESC(SYS_SPSR_EL2), access_spsr_el2, reset_val, SPSR_EL2, 0 }, { SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 }, { SYS_DESC(SYS_SP_EL1), access_sp_el1},
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 15/66 KVM: arm64: nv: Handle HCR_EL2.E2H specially
In reply to Marc ZyngierHCR_EL2.E2H is nasty, as a flip of this bit completely changes the way we deal with a lot of the state. So when the guest flips this bit (sysregs are live), do the put/load dance so that we have a consistent state.
Yes, this is slow. Don't do it.
--- arch/arm64/kvm/sys_regs.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
arch/arm64/kvm/sys_regs.c +20 -0
@@ -183,9 +183,24 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) goto memory_write;if (unlikely (get_el2_mapping (reg, &el1r, &xlate))) { +bool need_put_load; +if (!is_hyp_ctxt (vcpu)) goto memory_write; +/* +* HCR_EL2.E2H is nasty: it changes the way we interpret a +* lot of the EL2 state, so treat is as a full state +* transition. +*/ + need_put_load = ((reg ==HCR_EL2 ) && +vcpu_el2_e2h_is_set (vcpu) != !!(val &HCR_E2H )); + +if (need_put_load) { +preempt_disable (); +kvm_arch_vcpu_put (vcpu); + } +/* * Always store a copy of the write to memory to avoid having * to reverse-translate virtual EL2 system registers for a
@@ -193,6 +208,11 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)*/ __vcpu_sys_reg (vcpu, reg) = val; +if (need_put_load) { +kvm_arch_vcpu_load (vcpu,smp_processor_id ()); +preempt_enable (); + } +switch (reg) {case ELR_EL2 :write_sysreg_el1 (val,SYS_ELR );
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 16/66 KVM: arm64: nv: Save/Restore vEL2 sysregs
In reply to Marc ZyngierWhenever we need to restore the guest's system registers to the CPU, we now need to take care of the EL2 system registers as well. Most of them are accessed via traps only, but some have an immediate effect and also a guest running in VHE mode would expect them to be accessible via their EL1 encoding, which we do not trap.
For vEL2 we write the virtual EL2 registers with an identical format directly into their EL1 counterpart, and translate the few registers that have a different format for the same effect on the execution when running a non-VHE guest guest hypervisor.
Based on an initial patch from Andre Przywara, rewritten many times since.
--- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 5 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 ++++++++++++++++++++- 3 files changed, 127 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +3 -2
@@ -71,9 +71,10 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)write_sysreg (ctxt_sys_reg (ctxt,TPIDRRO_EL0 ), tpidrro_el0); } -static inline void __sysreg_restore_el1_state (struct kvm_cpu_context *ctxt) +static inline void __sysreg_restore_el1_state (struct kvm_cpu_context *ctxt, +u64 mpidr) { -write_sysreg (ctxt_sys_reg (ctxt,MPIDR_EL1 ), vmpidr_el2); +write_sysreg (mpidr, vmpidr_el2);write_sysreg (ctxt_sys_reg (ctxt,CSSELR_EL1 ), csselr_el1); if (has_vhe() ||
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c +1 -1
@@ -28,7 +28,7 @@ void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)void __sysreg_restore_state_nvhe (struct kvm_cpu_context *ctxt) { -__sysreg_restore_el1_state (ctxt); +__sysreg_restore_el1_state (ctxt,ctxt_sys_reg (ctxt,MPIDR_EL1 ));__sysreg_restore_common_state (ctxt);__sysreg_restore_user_state (ctxt);__sysreg_restore_el2_return_state (ctxt);
arch/arm64/kvm/hyp/vhe/sysreg-sr.c +123 -2
154 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 17/66 KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
We can no longer blindly copy the VCPU's PSTATE into SPSR_EL2 and return to the guest and vice versa when taking an exception to the hypervisor, because we emulate virtual EL2 in EL1 and therefore have to translate the mode field from EL2 to EL1 and vice versa.
--- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 43 +++++++++++++++++++++- 1 file changed, 41 insertions(+), 2 deletions(-)
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +41 -2
@@ -51,10 +51,32 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,SPSR_EL1 ) =read_sysreg_el1 (SYS_SPSR ); } +static inline u64 from_hw_pstate (const struct kvm_cpu_context *ctxt) +{ +u64 reg =read_sysreg_el2 (SYS_SPSR ); + +if (__is_hyp_ctxt (ctxt)) { +u64 mode = reg & (PSR_MODE_MASK |PSR_MODE32_BIT ); + +switch (mode) { +case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; +break ; +case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; +break ; + } + +return (reg & ~(PSR_MODE_MASK |PSR_MODE32_BIT )) | mode; + } + +return reg; +} +static inline void __sysreg_save_el2_return_state (struct kvm_cpu_context *ctxt) { ctxt->regs.pc =read_sysreg_el2 (SYS_ELR ); - ctxt->regs.pstate =read_sysreg_el2 (SYS_SPSR ); + ctxt->regs.pstate =from_hw_pstate (ctxt);if (cpus_have_final_cap (ARM64_HAS_RAS_EXTN ))ctxt_sys_reg (ctxt,DISR_EL1 ) =read_sysreg_s (SYS_VDISR_EL2 );
@@ -131,9 +153,26 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,write_sysreg_el1 (ctxt_sys_reg (ctxt,SPSR_EL1 ),SYS_SPSR ); } +/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ +static inline u64 to_hw_pstate (const struct kvm_cpu_context *ctxt) +{ +u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK |PSR_MODE32_BIT ); + +switch (mode) { +case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; +break ; +case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; +break ; + } + +return (ctxt->regs.pstate & ~(PSR_MODE_MASK |PSR_MODE32_BIT )) | mode; +} +static inline void __sysreg_restore_el2_return_state (struct kvm_cpu_context *ctxt) { -u64 pstate = ctxt->regs.pstate; +u64 pstate =to_hw_pstate (ctxt);u64 mode = pstate &PSR_AA32_MODE_MASK ;/*
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 18/66 KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
When running in virtual EL2 mode, we actually run the hardware in EL1 and therefore have to use the EL1 registers to ensure correct operation.
By setting the HCR.TVM and HCR.TVRM we ensure that the virtual EL2 mode doesn't shoot itself in the foot when setting up what it believes to be a different mode's system register state (for example when preparing to switch to a VM).
We can leverage the existing sysregs infrastructure to support trapped accesses to these registers.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +--- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 7 ++++++- arch/arm64/kvm/sys_regs.c | 19 ++++++++++++++++--- 4 files changed, 24 insertions(+), 8 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +1 -3
@@ -101,10 +101,8 @@ static inline void __deactivate_traps_common(void)write_sysreg (0 , pmuserenr_el0); } -static inline void ___activate_traps (struct kvm_vcpu *vcpu) +static inline void ___activate_traps (struct kvm_vcpu *vcpu,u64 hcr) { -u64 hcr = vcpu->arch.hcr_el2; -if (cpus_have_final_cap (ARM64_WORKAROUND_CAVIUM_TX2_219_TVM )) hcr |=HCR_TVM ;
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
@@ -37,7 +37,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) {u64 val; -___activate_traps (vcpu); +___activate_traps (vcpu, vcpu->arch.hcr_el2);__activate_traps_common (vcpu); val =CPTR_EL2_DEFAULT ;
arch/arm64/kvm/hyp/vhe/switch.c +6 -1
@@ -36,9 +36,14 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector);static void __activate_traps (struct kvm_vcpu *vcpu) { +u64 hcr = vcpu->arch.hcr_el2;u64 val; -___activate_traps (vcpu); +/* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ +if (vcpu_mode_el2 (vcpu) && !vcpu_el2_e2h_is_set (vcpu)) + hcr |=HCR_TVM |HCR_TRVM ; + +___activate_traps (vcpu, hcr); val =read_sysreg (cpacr_el1); val |=CPACR_EL1_TTA ;
arch/arm64/kvm/sys_regs.c +16 -3
@@ -361,8 +361,15 @@ static void get_access_mask(const struct sys_reg_desc *r, u64 *mask, u64 *shift)/* * Generic accessor for VM registers. Only called as long as HCR_TVM -* is set. If the guest enables the MMU, we stop trapping the VM -* sys_regs and leave it in complete control of the caches. +* is set. +* +* This is set in two cases: either (1) we're running at vEL2, or (2) +* we're running at EL1 and the guest has its MMU off. +* +* (1) TVM/TRVM is set, as we need to virtualise some of the VM +* registers for the guest hypervisor +* (2) Once the guest enables the MMU, we stop trapping the VM sys_regs +* and leave it in complete control of the caches. */ static bool access_vm_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
@@ -371,7 +378,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,bool was_enabled =vcpu_has_cache_enabled (vcpu);u64 val, mask, shift; -BUG_ON (!p->is_write); +/* We don't expect TRVM on the host */ +BUG_ON (!vcpu_mode_el2 (vcpu) && !p->is_write); + +if (!p->is_write) { + p->regval =vcpu_read_sys_reg (vcpu, r->reg); +return true; + }get_access_mask (r, &mask, &shift);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 19/66 KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
For the same reason we trap virtual memory register accesses at virtual EL2, we need to trap SPSR_EL1, ELR_EL1 and VBAR_EL1 accesses. ARM v8.3 introduces the HCR_EL2.NV1 bit to be able to trap on those register accesses in EL1. Do not set this bit until the whole nesting support is completed.
--- arch/arm64/kvm/sys_regs.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-)
arch/arm64/kvm/sys_regs.c +28 -1
@@ -1567,6 +1567,30 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,return true; } +static bool access_elr (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +vcpu_write_sys_reg (vcpu, p->regval,ELR_EL1 ); +else + p->regval =vcpu_read_sys_reg (vcpu,ELR_EL1 ); + +return true; +} + +static bool access_spsr (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (p->is_write) +__vcpu_sys_reg (vcpu,SPSR_EL1 ) = p->regval; +else + p->regval =__vcpu_sys_reg (vcpu,SPSR_EL1 ); + +return true; +} +static bool access_spsr_el2 (struct kvm_vcpu *vcpu,struct sys_reg_params *p,const struct sys_reg_desc *r)
@@ -1728,6 +1752,9 @@ static const struct sys_reg_desc sys_reg_descs[] = {PTRAUTH_KEY (APDB ),PTRAUTH_KEY (APGA ), + { SYS_DESC(SYS_SPSR_EL1), access_spsr}, + { SYS_DESC(SYS_ELR_EL1), access_elr}, + { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 }, { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 }, { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 },
@@ -1759,7 +1786,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 20/66 KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
For the same reason we trap virtual memory register accesses in virtual EL2, we trap CPACR_EL1 access too; We allow the virtual EL2 mode to access EL1 system register state instead of the virtual EL2 one.
--- arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 3 +++ arch/arm64/kvm/sys_regs.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -1
@@ -269,7 +269,7 @@#define CPTR_EL2_TFP_SHIFT 10/* Hyp Coprocessor Trap Register */ -#define CPTR_EL2_TCPAC (1 << 31) +#define CPTR_EL2_TCPAC (1U << 31)#define CPTR_EL2_TAM (1 << 30)#define CPTR_EL2_TTA (1 << 20)#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT)
arch/arm64/kvm/hyp/vhe/switch.c +3 -0
@@ -68,6 +68,9 @@ static void __activate_traps(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +if (vcpu_mode_el2 (vcpu) && !vcpu_el2_e2h_is_set (vcpu)) + val |=CPTR_EL2_TCPAC ; +write_sysreg (val, cpacr_el1);write_sysreg (__this_cpu_read (kvm_hyp_vector), vbar_el1);
arch/arm64/kvm/sys_regs.c +1 -1
@@ -1736,7 +1736,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, { SYS_DESC(SYS_ACTLR_EL1), access_actlr, reset_actlr, ACTLR_EL1 }, - { SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_CPACR_EL1), access_rw, reset_val, CPACR_EL1, 0 }, { SYS_DESC(SYS_RGSR_EL1), undef_access }, { SYS_DESC(SYS_GCR_EL1), undef_access },
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 21/66 KVM: arm64: nv: Handle PSCI call via smc from the guest
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
VMs used to execute hvc #0 for the psci call if EL3 is not implemented. However, when we come to provide the virtual EL2 mode to the VM, the host OS inside the VM calls kvm_call_hyp() which is also hvc #0. So, it's hard to differentiate between them from the host hypervisor's point of view.
So, let the VM execute smc instruction for the psci call. On ARMv8.3, even if EL3 is not implemented, a smc instruction executed at non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than being treated as UNDEFINED. So, the host hypervisor can handle this psci call without any confusion.
--- arch/arm64/kvm/handle_exit.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
arch/arm64/kvm/handle_exit.c +22 -2
@@ -62,6 +62,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu)static int handle_smc (struct kvm_vcpu *vcpu) { +int ret; +/* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
@@ -69,10 +71,28 @@ static int handle_smc(struct kvm_vcpu *vcpu)* * We need to advance the PC after the trap, as it would * otherwise return to the same address... +* +* If imm is non-zero, it's not defined, so just skip it. +*/ +if (kvm_vcpu_hvc_get_imm (vcpu)) { +vcpu_set_reg (vcpu,0 , ~0UL ); +kvm_incr_pc (vcpu); +return 1 ; + } + +/* +* If imm is zero, it's a psci call. +* Note that on ARMv8.3, even if EL3 is not implemented, SMC executed +* at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than +* being treated as UNDEFINED. */ -vcpu_set_reg (vcpu,0 , ~0UL ); + ret =kvm_hvc_call_handler (vcpu); +if (ret <0 ) +vcpu_set_reg (vcpu,0 , ~0UL ); +kvm_incr_pc (vcpu); -return 1 ; + +return ret; }/*
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 22/66 KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
Forward exceptions due to WFI or WFE instructions to the virtual EL2 if they are not coming from the virtual EL2 and virtual HCR_EL2.TWX is set.
--- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/handle_exit.c | 11 +++++++- arch/arm64/kvm/nested.c | 40 +++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/kvm/nested.c
arch/arm64/include/asm/kvm_nested.h +2 -0
@@ -61,4 +61,6 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) (cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN))); } +int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe); +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/Makefile +1 -1
@@ -17,7 +17,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ arch_timer.o \ - emulate-nested.o \ + emulate-nested.o nested.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/handle_exit.c +10 -1
@@ -119,7 +119,16 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu)*/ static int kvm_handle_wfx (struct kvm_vcpu *vcpu) { -if (kvm_vcpu_get_esr (vcpu) & ESR_ELx_WFx_ISS_WFE) { +bool is_wfe = !!(kvm_vcpu_get_esr (vcpu) & ESR_ELx_WFx_ISS_WFE); + +if (nested_virt_in_use (vcpu)) { +int ret =handle_wfx_nested (vcpu, is_wfe); + +if (ret != -EINVAL ) +return ret; + } + +if (is_wfe) {trace_kvm_wfx_arm64 (*vcpu_pc (vcpu), true); vcpu->stat.wfe_exit_stat++;kvm_vcpu_on_spin (vcpu,vcpu_mode_priv (vcpu));
arch/arm64/kvm/nested.c new file +40 -0
@@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* Copyright (C) 2017 - Columbia University and Linaro Ltd. +* Author: Jintack Lim <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <linux/kvm.h> +#include <linux/kvm_host.h> + +#include <asm/kvm_emulate.h> + +/* +* Inject wfx to the virtual EL2 if this is not from the virtual EL2 and +* the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor +* handle this. +*/ +int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe) +{ +u64 hcr_el2 =__vcpu_sys_reg (vcpu,HCR_EL2 ); + +if (vcpu_mode_el2 (vcpu)) +return -EINVAL ; + +if ((is_wfe && (hcr_el2 &HCR_TWE )) || (!is_wfe && (hcr_el2 &HCR_TWI ))) +return kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu)); + +return -EINVAL ; +}
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 23/66 KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
Forward traps due to FP/ASIMD register accesses to the virtual EL2 if virtual CPTR_EL2.TFP is set (with HCR_EL2.E2H == 0) or CPTR_EL2.FPEN is configure to do so (with HCR_EL2.E2h == 1).
[maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with
all the hard work actually being done by Chase Conklin]
--- arch/arm64/include/asm/kvm_emulate.h | 26 +++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 16 +++++++++++---- arch/arm64/kvm/hyp/include/hyp/switch.h | 11 +++++++++-- 3 files changed, 47 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +26 -0
@@ -11,12 +11,14 @@#ifndef __ARM64_KVM_EMULATE_H__#define __ARM64_KVM_EMULATE_H__ +#include <linux/bitfield.h> #include <linux/kvm_host.h> #include <asm/debug-monitors.h> #include <asm/esr.h> #include <asm/kvm_arm.h> #include <asm/kvm_hyp.h> +#include <asm/kvm_nested.h> #include <asm/ptrace.h> #include <asm/cputype.h> #include <asm/virt.h>
@@ -321,6 +323,30 @@ static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu)return mode != PSR_MODE_EL0t; } +static inline bool guest_hyp_fpsimd_traps_enabled (const struct kvm_vcpu *vcpu) +{ +u64 val; + +if (!nested_virt_in_use (vcpu)) +return false; + + val =vcpu_read_sys_reg (vcpu,CPTR_EL2 ); + +if (!vcpu_el2_e2h_is_set (vcpu)) +return (val &CPTR_EL2_TFP ); + +switch (FIELD_GET (CPACR_EL1_FPEN , val)) { +case 0b00 : +case 0b10 : +return true; +case 0b01 : +return vcpu_el2_tge_is_set (vcpu) && !vcpu_mode_el2 (vcpu); +case 0b11 : +default :/* GCC is dumb */ +return false; + } +} + static __always_inline u32 kvm_vcpu_get_esr(const struct kvm_vcpu *vcpu) {return vcpu->arch.fault.esr_el2;
arch/arm64/kvm/handle_exit.c +12 -4
@@ -96,11 +96,19 @@ static int handle_smc(struct kvm_vcpu *vcpu) }/* -* Guest access to FP/ASIMD registers are routed to this handler only -* when the system doesn't support FP/ASIMD. +* This handles the cases where the system does not support FP/ASIMD or when +* we are running nested virtualization and the guest hypervisor is trapping +* FP/ASIMD accesses by its guest guest. +* +* All other handling of guest vs. host FP/ASIMD register state is handled in +* fixup_guest_exit(). */ -static int handle_no_fpsimd (struct kvm_vcpu *vcpu) +static int kvm_handle_fpasimd (struct kvm_vcpu *vcpu) { +if (guest_hyp_fpsimd_traps_enabled (vcpu)) +return kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu)); + +/* This is the case when the system doesn't support FP/ASIMD. */ kvm_inject_undefined (vcpu);return 1 ; }
@@ -243,7 +251,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_BREAKPT_LOW]= kvm_handle_guest_debug, [ESR_ELx_EC_BKPT32] = kvm_handle_guest_debug, [ESR_ELx_EC_BRK64] = kvm_handle_guest_debug, - [ESR_ELx_EC_FP_ASIMD] = handle_no_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_fpasimd, [ESR_ELx_EC_PAC] = kvm_handle_ptrauth, };
arch/arm64/kvm/hyp/include/hyp/switch.h +9 -2
@@ -223,11 +223,18 @@ static inline bool __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) esr_ec != ESR_ELx_EC_SVE)return false; -/* Don't handle SVE traps for non-SVE vcpus here: */ -if (!sve_guest) +/* +* Don't handle SVE traps for non-SVE vcpus here. This +* includes NV guests for the time being. +*/ +if (!sve_guest) {if (esr_ec != ESR_ELx_EC_FP_ASIMD)return false; +if (guest_hyp_fpsimd_traps_enabled (vcpu)) +return false; + } +/* Valid trap. Switch the context: */ if (vhe) {
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 24/66 KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set.
In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set.
This is for recursive nested virtualization.
--- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 27 +++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 7 +++++++ arch/arm64/kvm/sys_regs.c | 24 ++++++++++++++++++++++++ 5 files changed, 61 insertions(+)
arch/arm64/include/asm/kvm_arm.h +1 -0
@@ -14,6 +14,7 @@/* Hyp Configuration Register (HCR) bits */ #define HCR_ATA (UL(1) << 56)#define HCR_FWB (UL(1) << 46) +#define HCR_NV (UL(1) << 42)#define HCR_API (UL(1) << 41)#define HCR_APK (UL(1) << 40)#define HCR_TEA (UL(1) << 37)
arch/arm64/include/asm/kvm_nested.h +2 -0
@@ -62,5 +62,7 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) }int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe); +extern bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit); +extern bool forward_nv_traps (struct kvm_vcpu *vcpu);#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/emulate-nested.c +27 -0
@@ -25,11 +25,38 @@#include "trace.h" +bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit) +{ +bool control_bit_set; + +if (!nested_virt_in_use (vcpu)) +return false; + + control_bit_set =__vcpu_sys_reg (vcpu,HCR_EL2 ) & control_bit; +if (!vcpu_mode_el2 (vcpu) && control_bit_set) { +kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu)); +return true; + } +return false; +} + +bool forward_nv_traps (struct kvm_vcpu *vcpu) +{ +return forward_traps (vcpu,HCR_NV ); +} +void kvm_emulate_nested_eret (struct kvm_vcpu *vcpu) {u64 spsr, elr, mode;bool direct_eret; +/* +* Forward this trap to the virtual EL2 if the virtual +* HCR_EL2.NV bit is set and this is coming from !EL2. +*/ +if (forward_nv_traps (vcpu)) +return ; +/* * Going through the whole put/load motions is a waste of time * if this is a VHE guest hypervisor returning to its own
arch/arm64/kvm/handle_exit.c +7 -0
@@ -64,6 +64,13 @@ static int handle_smc(struct kvm_vcpu *vcpu) {int ret; +/* +* Forward this trapped smc instruction to the virtual EL2 if +* the guest has asked for it. +*/ +if (forward_traps (vcpu,HCR_TSC )) +return 1 ; +/* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
arch/arm64/kvm/sys_regs.c +24 -0
@@ -267,10 +267,19 @@ static u32 get_ccsidr(u32 csselr)return ccsidr; } +static bool el12_reg (struct sys_reg_params *p) +{ +/* All *_EL12 registers have Op1=5. */ +return (p->Op1 ==5 ); +} +static bool access_rw (struct kvm_vcpu *vcpu,struct sys_reg_params *p,const struct sys_reg_desc *r) { +if (el12_reg (p) &&forward_nv_traps (vcpu)) +return false; +if (p->is_write)vcpu_write_sys_reg (vcpu, p->regval, r->reg);else
@@ -294,6 +303,9 @@ static bool access_sctlr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; +if (p->is_write) {u64 val = p->regval;
@@ -378,6 +390,9 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,bool was_enabled =vcpu_has_cache_enabled (vcpu);u64 val, mask, shift; +if (el12_reg (p) &&forward_nv_traps (vcpu)) +return false; +/* We don't expect TRVM on the host */ BUG_ON (!vcpu_mode_el2 (vcpu) && !p->is_write);
@@ -1571,6 +1586,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; +if (p->is_write)vcpu_write_sys_reg (vcpu, p->regval,ELR_EL1 );else
@@ -1583,6 +1601,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; +if (p->is_write)__vcpu_sys_reg (vcpu,SPSR_EL1 ) = p->regval;else
@@ -1595,6 +1616,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; +if (p->is_write)vcpu_write_sys_reg (vcpu, p->regval,SPSR_EL2 );else
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 25/66 KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
Forward the EL1 virtual memory register traps to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.TVM or TRVM bit is set.
This is for recursive nested virtualization.
--- arch/arm64/kvm/sys_regs.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/arm64/kvm/sys_regs.c +7 -0
@@ -393,6 +393,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,if (el12_reg (p) &&forward_nv_traps (vcpu))return false; +if (!el12_reg (p)) { +u64 bit = p->is_write ?HCR_TVM :HCR_TRVM ; + +if (forward_traps (vcpu, bit)) +return false; + } +/* We don't expect TRVM on the host */ BUG_ON (!vcpu_mode_el2 (vcpu) && !p->is_write);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 26/66 KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.NV bit is set.
This is for recursive nested virtualization.
--- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/kvm/sys_regs.c | 28 +++++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
@@ -14,6 +14,7 @@/* Hyp Configuration Register (HCR) bits */ #define HCR_ATA (UL(1) << 56)#define HCR_FWB (UL(1) << 46) +#define HCR_NV1 (UL(1) << 43)#define HCR_NV (UL(1) << 42)#define HCR_API (UL(1) << 41)#define HCR_APK (UL(1) << 40)
arch/arm64/kvm/sys_regs.c +27 -1
@@ -299,6 +299,22 @@ static bool access_wi(struct kvm_vcpu *vcpu,return true; } +/* This function is to support the recursive nested virtualization */ +static bool forward_nv1_traps (struct kvm_vcpu *vcpu,struct sys_reg_params *p) +{ +return forward_traps (vcpu,HCR_NV1 ); +} + +static bool access_vbar_el1 (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +if (forward_nv1_traps (vcpu, p)) +return false; + +return access_rw (vcpu, p, r); +} +static bool access_sctlr_el2 (struct kvm_vcpu *vcpu,struct sys_reg_params *p,const struct sys_reg_desc *r)
@@ -1589,6 +1605,7 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,return true; } +static bool access_elr (struct kvm_vcpu *vcpu,struct sys_reg_params *p,const struct sys_reg_desc *r)
@@ -1596,6 +1613,9 @@ static bool access_elr(struct kvm_vcpu *vcpu,if (el12_reg (p) &&forward_nv_traps (vcpu))return false; +if (!el12_reg (p) &&forward_nv1_traps (vcpu, p)) +return false; +if (p->is_write)vcpu_write_sys_reg (vcpu, p->regval,ELR_EL1 );else
@@ -1611,6 +1631,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu,if (el12_reg (p) &&forward_nv_traps (vcpu))return false; +if (!el12_reg (p) &&forward_nv1_traps (vcpu, p)) +return false; +if (p->is_write)__vcpu_sys_reg (vcpu,SPSR_EL1 ) = p->regval;else
@@ -1626,6 +1649,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu,if (el12_reg (p) &&forward_nv_traps (vcpu))return false; +if (!el12_reg (p) &&forward_nv1_traps (vcpu, p)) +return false; +if (p->is_write)vcpu_write_sys_reg (vcpu, p->regval,SPSR_EL2 );else
@@ -1817,7 +1843,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_vbar_el1, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 27/66 KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
With HCR_EL2.NV bit set, accesses to EL12 registers in the virtual EL2 trap to EL2. Handle those traps just like we do for EL1 registers.
One exception is CNTKCTL_EL12. We don't trap on CNTKCTL_EL1 for non-VHE virtual EL2 because we don't have to. However, accessing CNTKCTL_EL12 will trap since it's one of the EL12 registers controlled by HCR_EL2.NV bit. Therefore, add a handler for it and don't treat it as a non-trap-registers when preparing a shadow context.
--- arch/arm64/kvm/sys_regs.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
arch/arm64/kvm/sys_regs.c +17 -0
@@ -2085,6 +2085,23 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, + { SYS_DESC(SYS_CPACR_EL12), access_rw, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_TTBR0_EL12), access_vm_reg, reset_unknown, TTBR0_EL1 }, + { SYS_DESC(SYS_TTBR1_EL12), access_vm_reg, reset_unknown, TTBR1_EL1 }, + { SYS_DESC(SYS_TCR_EL12), access_vm_reg, reset_val, TCR_EL1, 0 }, + { SYS_DESC(SYS_SPSR_EL12), access_spsr}, + { SYS_DESC(SYS_ELR_EL12), access_elr}, + { SYS_DESC(SYS_AFSR0_EL12), access_vm_reg, reset_unknown, AFSR0_EL1 }, + { SYS_DESC(SYS_AFSR1_EL12), access_vm_reg, reset_unknown, AFSR1_EL1 }, + { SYS_DESC(SYS_ESR_EL12), access_vm_reg, reset_unknown, ESR_EL1 }, + { SYS_DESC(SYS_FAR_EL12), access_vm_reg, reset_unknown, FAR_EL1 }, + { SYS_DESC(SYS_MAIR_EL12), access_vm_reg, reset_unknown, MAIR_EL1 }, + { SYS_DESC(SYS_AMAIR_EL12), access_vm_reg, reset_amair_el1, AMAIR_EL1 }, + { SYS_DESC(SYS_VBAR_EL12), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_CONTEXTIDR_EL12), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 }, + { SYS_DESC(SYS_CNTKCTL_EL12), access_rw, reset_val, CNTKCTL_EL1, 0 }, + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, };
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 28/66 KVM: arm64: nv: Forward debug traps to the nested guest
In reply to Marc ZyngierOn handling a debug trap, check whether we need to forward it to the guest before handling it.
--- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 9 +++++++-- arch/arm64/kvm/sys_regs.c | 3 +++ 3 files changed, 12 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_nested.h +2 -0
@@ -62,6 +62,8 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) }int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe); +extern bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg, +u64 control_bit);extern bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit);extern bool forward_nv_traps (struct kvm_vcpu *vcpu);
arch/arm64/kvm/emulate-nested.c +7 -2
@@ -25,14 +25,14 @@#include "trace.h" -bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit) +bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg,u64 control_bit) {bool control_bit_set;if (!nested_virt_in_use (vcpu))return false; - control_bit_set =__vcpu_sys_reg (vcpu,HCR_EL2 ) & control_bit; + control_bit_set =__vcpu_sys_reg (vcpu, reg) & control_bit;if (!vcpu_mode_el2 (vcpu) && control_bit_set) {kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu));return true;
@@ -40,6 +40,11 @@ bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit)return false; } +bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit) +{ +return __forward_traps (vcpu,HCR_EL2 , control_bit); +} +bool forward_nv_traps (struct kvm_vcpu *vcpu) {return forward_traps (vcpu,HCR_NV );
arch/arm64/kvm/sys_regs.c +3 -0
@@ -607,6 +607,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu,struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (__forward_traps(vcpu, MDCR_EL2, MDCR_EL2_TDA | MDCR_EL2_TDE)) + return false; +access_rw (vcpu ,p ,r );if (p->is_write) vcpu->arch.flags |=KVM_ARM64_DEBUG_DIRTY ;
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 29/66 KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
We enable nested virtualization by setting the HCR NV and NV1 bit.
When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states.
--- arch/arm64/kvm/hyp/vhe/switch.c | 36 ++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/vhe/switch.c +33 -3
@@ -39,9 +39,39 @@ static void __activate_traps(struct kvm_vcpu *vcpu)u64 hcr = vcpu->arch.hcr_el2;u64 val; -/* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ -if (vcpu_mode_el2 (vcpu) && !vcpu_el2_e2h_is_set (vcpu)) - hcr |=HCR_TVM |HCR_TRVM ; +if (is_hyp_ctxt (vcpu)) { + hcr |=HCR_NV ; + +if (!vcpu_el2_e2h_is_set (vcpu)) { +/* +* For a guest hypervisor on v8.0, trap and emulate +* the EL1 virtual memory control register accesses. +*/ + hcr |=HCR_TVM |HCR_TRVM |HCR_NV1 ; + }else { +/* +* For a guest hypervisor on v8.1 (VHE), allow to +* access the EL1 virtual memory control registers +* natively. These accesses are to access EL2 register +* states. +* Note that we still need to respect the virtual +* HCR_EL2 state. +*/ +u64 vhcr_el2 =__vcpu_sys_reg (vcpu,HCR_EL2 ); + +/* +* We already set TVM to handle set/way cache maint +* ops traps, this somewhat collides with the nested +* virt trapping for nVHE. So turn this off for now +* here, in the hope that VHE guests won't ever do this. +* TODO: find out whether it's worth to support both +* cases at the same time. +*/ + hcr &= ~HCR_TVM ; + + hcr |= vhcr_el2 & (HCR_TVM |HCR_TRVM ); + } + }___activate_traps (vcpu, hcr);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 30/66 KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
So far we were flushing almost the entire universe whenever a VM would load/unload the SCTLR_EL1 and the two versions of that register had different MMU enabled settings. This turned out to be so slow that it prevented forward progress for a nested VM, because a scheduler timer tick interrupt would always be pending when we reached the nested VM.
To avoid this problem, we consider the SCTLR_EL2 when evaluating if caches are on or off when entering virtual EL2 (because this is the value that we end up shadowing onto the hardware EL1 register).
--- arch/arm64/include/asm/kvm_mmu.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_mmu.h +5 -1
@@ -125,6 +125,7 @@ alternative_cb_end#include <asm/cache.h> #include <asm/cacheflush.h> #include <asm/mmu_context.h> +#include <asm/kvm_emulate.h> void kvm_update_va_mask (struct alt_instr *alt,__le32 *origptr,__le32 *updptr,int nr_inst);
@@ -201,7 +202,10 @@ struct kvm;static inline bool vcpu_has_cache_enabled (struct kvm_vcpu *vcpu) { -return (vcpu_read_sys_reg (vcpu,SCTLR_EL1 ) &0b101 ) ==0b101 ; +if (vcpu_mode_el2 (vcpu)) +return (__vcpu_sys_reg (vcpu,SCTLR_EL2 ) &0b101 ) ==0b101 ; +else +return (vcpu_read_sys_reg (vcpu,SCTLR_EL1 ) &0b101 ) ==0b101 ; }static inline void __clean_dcache_guest_page (kvm_pfn_t pfn,unsigned long size)
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 31/66 KVM: arm64: nv: Filter out unsupported features from ID regs
In reply to Marc ZyngierAs there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.
Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.
--- arch/arm64/include/asm/kvm_nested.h | 6 ++ arch/arm64/kvm/nested.c | 152 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 4 +- 3 files changed, 161 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +6 -0
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,extern bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit);extern bool forward_nv_traps (struct kvm_vcpu *vcpu); +struct sys_reg_params ; +struct sys_reg_desc ; + +void access_nested_id_reg (struct kvm_vcpu *v,struct sys_reg_params *p, +const struct sys_reg_desc *r); +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +152 -0
161 lines not on this page · raw message
arch/arm64/kvm/sys_regs.c +3 -1
@@ -1372,8 +1372,10 @@ static bool access_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) {bool raz =sysreg_visible_as_raz (vcpu, r); +bool ret =__access_id_reg (vcpu, p, r, raz); -return __access_id_reg (vcpu, p, r, raz); +access_nested_id_reg (vcpu, p, r); +return ret; } static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 32/66 KVM: arm64: nv: Hide RAS from nested guests
In reply to Marc ZyngierWe don't want to expose complicated features to guests until we have a good grasp on the basic CPU emulation. So let's pretend that RAS, doesn't exist in a nested guest. We already hide the feature bits, let's now make sure VDISR_EL1 will UNDEF.
--- arch/arm64/kvm/sys_regs.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/arm64/kvm/sys_regs.c +9 -0
@@ -527,6 +527,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu,return read_zero (vcpu, p); } +static bool trap_undef (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +kvm_inject_undefined (vcpu); +return false; +} +/* * ARMv8.1 mandates at least a trivial LORegion implementation, where all the * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0
@@ -2083,6 +2091,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 }, { SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 }, { SYS_DESC(SYS_RMR_EL2), access_wi, reset_val, RMR_EL2, 1 }, + { SYS_DESC(SYS_VDISR_EL2), trap_undef }, { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 },
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 33/66 KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
In reply to Marc ZyngierAdd Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd.
We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables.
--- arch/arm64/include/asm/kvm_host.h | 29 +++++ arch/arm64/include/asm/kvm_mmu.h | 8 ++ arch/arm64/include/asm/kvm_nested.h | 7 ++ arch/arm64/kvm/arm.c | 16 ++- arch/arm64/kvm/mmu.c | 18 ++- arch/arm64/kvm/nested.c | 183 ++++++++++++++++++++++++++++ 6 files changed, 250 insertions(+), 11 deletions(-)
arch/arm64/include/asm/kvm_host.h +29 -0
@@ -95,14 +95,43 @@ struct kvm_s2_mmu {int __percpu *last_vcpu_ran;struct kvm *kvm; + +/* +* For a shadow stage-2 MMU, the virtual vttbr programmed by the guest +* hypervisor. Unused for kvm_arch->mmu. Set to 1 when the structure +* contains no valid information. +*/ +u64 vttbr; + +/* true when this represents a nested context where virtual HCR_EL2.VM == 1 */ +bool nested_stage2_enabled; + +/* +* 0: Nobody is currently using this, check vttbr for validity +* >0: Somebody is actively using this. +*/ +atomic_t refcnt; }; +static inline bool kvm_s2_mmu_valid (struct kvm_s2_mmu *mmu) +{ +return !(mmu->vttbr &1 ); +} +struct kvm_arch_memory_slot { };struct kvm_arch {struct kvm_s2_mmu mmu; +/* +* Stage 2 paging stage for VMs with nested virtual using a virtual +* VMID. +*/ +struct kvm_s2_mmu *nested_mmus; +size_t nested_mmus_size; +int nested_mmus_next; +/* VTCR_EL2 value for this VM */ u64 vtcr;
arch/arm64/include/asm/kvm_mmu.h +8 -0
@@ -126,6 +126,7 @@ alternative_cb_end#include <asm/cacheflush.h> #include <asm/mmu_context.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_nested.h> void kvm_update_va_mask (struct alt_instr *alt,__le32 *origptr,__le32 *updptr,int nr_inst);
@@ -184,6 +185,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr);void free_hyp_pgds (void ); +void kvm_unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size);void stage2_unmap_vm (struct kvm *kvm);int kvm_init_stage2_mmu (struct kvm *kvm,struct kvm_s2_mmu *mmu);void kvm_free_stage2_pgd (struct kvm_s2_mmu *mmu);
@@ -306,5 +308,11 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static inline u64 get_vmid (u64 vttbr) +{ +return (vttbr &VTTBR_VMID_MASK (kvm_get_vmid_bits ())) >> +VTTBR_VMID_SHIFT ; +} +#endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */
arch/arm64/include/asm/kvm_nested.h +7 -0
@@ -61,6 +61,13 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) (cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN))); } +extern void kvm_init_nested (struct kvm *kvm); +extern int kvm_vcpu_init_nested (struct kvm_vcpu *vcpu); +extern void kvm_init_nested_s2_mmu (struct kvm_s2_mmu *mmu); +extern struct kvm_s2_mmu *lookup_s2_mmu (struct kvm *kvm,u64 vttbr,u64 hcr); +extern void kvm_vcpu_load_hw_mmu (struct kvm_vcpu *vcpu); +extern void kvm_vcpu_put_hw_mmu (struct kvm_vcpu *vcpu); +int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe);extern bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg,u64 control_bit);
arch/arm64/kvm/arm.c +15 -1
@@ -35,6 +35,7 @@#include <asm/kvm_arm.h> #include <asm/kvm_asm.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include <asm/kvm_emulate.h> #include <asm/sections.h>
@@ -142,6 +143,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)if (ret)return ret; +kvm_init_nested (kvm); + ret =create_hyp_mappings (kvm, kvm +1 ,PAGE_HYP );if (ret) goto out_free_stage2_pgd;
@@ -385,6 +388,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)struct kvm_s2_mmu *mmu;int *last_ran; +if (nested_virt_in_use (vcpu)) +kvm_vcpu_load_hw_mmu (vcpu); + mmu = vcpu->arch.hw_mmu; last_ran =this_cpu_ptr (mmu->last_vcpu_ran);
@@ -426,6 +432,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)kvm_vgic_put (vcpu);kvm_vcpu_pmu_restore_host (vcpu); +if (nested_virt_in_use (vcpu)) +kvm_vcpu_put_hw_mmu (vcpu); + vcpu->cpu =-1 ; }
@@ -1026,8 +1035,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, vcpu->arch.target = phys_target; +/* Prepare for nested if required */ + ret =kvm_vcpu_init_nested (vcpu); +/* Now we know what it is, we can reset it. */ - ret =kvm_reset_vcpu (vcpu); +if (!ret) + ret =kvm_reset_vcpu (vcpu); +if (ret) { vcpu->arch.target =-1 ;bitmap_zero (vcpu->arch.features,KVM_VCPU_MAX_FEATURES );
arch/arm64/kvm/mmu.c +8 -10
@@ -113,7 +113,7 @@ static bool kvm_is_device_pfn(unsigned long pfn)* does. */ /** -* unmap_stage2_range -- Clear stage2 page table entries to unmap a range +* kvm_unmap_stage2_range -- Clear stage2 page table entries to unmap a range * @mmu: The KVM stage-2 MMU pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap
@@ -136,7 +136,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 may_block)); } -static void unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size) +void kvm_unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size) {__unmap_stage2_range (mmu, start, size, true); }
@@ -391,6 +391,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) mmu->pgt = pgt; mmu->pgd_phys =__pa (pgt->pgd); mmu->vmid.vmid_gen =0 ; + +kvm_init_nested_s2_mmu (mmu); +return 0 ; out_destroy_pgtable:
@@ -435,7 +438,7 @@ static void stage2_unmap_memslot(struct kvm *kvm,if (!(vma->vm_flags &VM_PFNMAP )) {gpa_t gpa = addr + (vm_start - memslot->userspace_addr); -unmap_stage2_range (&kvm->arch.mmu, gpa, vm_end - vm_start); +kvm_unmap_stage2_range (&kvm->arch.mmu, gpa, vm_end - vm_start); } hva = vm_end; }while (hva < reg_end);
@@ -1360,7 +1363,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,spin_lock (&kvm->mmu_lock);if (ret) -unmap_stage2_range (&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size); +kvm_unmap_stage2_range (&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size);else if (!cpus_have_final_cap (ARM64_HAS_STAGE2_FWB ))stage2_flush_memslot (kvm, memslot);spin_unlock (&kvm->mmu_lock);
@@ -1377,11 +1380,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) { } -void kvm_arch_flush_shadow_all (struct kvm *kvm) -{ -kvm_free_stage2_pgd (&kvm->arch.mmu); -} -void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) {
@@ -1389,7 +1387,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,phys_addr_t size = slot->npages <<PAGE_SHIFT ;spin_lock (&kvm->mmu_lock); -unmap_stage2_range (&kvm->arch.mmu, gpa, size); +kvm_unmap_stage2_range (&kvm->arch.mmu, gpa, size);spin_unlock (&kvm->mmu_lock); }
arch/arm64/kvm/nested.c +183 -0
201 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 33/66] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
Quoted 270 lines · Marc Zyngier
Add Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd.
We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables.
Co-developed-by Christoffer Dall <[email protected]>Signed-off-by Christoffer Dall <[email protected]>Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 29 +++++ arch/arm64/include/asm/kvm_mmu.h | 8 ++ arch/arm64/include/asm/kvm_nested.h | 7 ++ arch/arm64/kvm/arm.c | 16 ++- arch/arm64/kvm/mmu.c | 18 ++- arch/arm64/kvm/nested.c | 183 ++++++++++++++++++++++++++++ 6 files changed, 250 insertions(+), 11 deletions(-)arch/arm64/include/asm/kvm_host.h +29
-0@@ -95,14 +95,43 @@ struct kvm_s2_mmu {int __percpu *last_vcpu_ran;struct kvm *kvm; + +/* +* For a shadow stage-2 MMU, the virtual vttbr programmed by the guest +* hypervisor. Unused for kvm_arch->mmu. Set to 1 when the structure +* contains no valid information. +*/ +u64 vttbr; + +/* true when this represents a nested context where virtual HCR_EL2.VM == 1 */ +bool nested_stage2_enabled; + +/* +* 0: Nobody is currently using this, check vttbr for validity +* >0: Somebody is actively using this. +*/ +atomic_t refcnt; }; +static inline bool kvm_s2_mmu_valid (struct kvm_s2_mmu *mmu) +{ +return !(mmu->vttbr &1 ); +} +struct kvm_arch_memory_slot { };struct kvm_arch {struct kvm_s2_mmu mmu; +/* +* Stage 2 paging stage for VMs with nested virtual using a virtual +* VMID. +*/ +struct kvm_s2_mmu *nested_mmus; +size_t nested_mmus_size; +int nested_mmus_next; +/* VTCR_EL2 value for this VM */ u64 vtcr;arch/arm64/include/asm/kvm_mmu.h +8
-0@@ -126,6 +126,7 @@ alternative_cb_end#include <asm/cacheflush.h> #include <asm/mmu_context.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_nested.h> void kvm_update_va_mask (struct alt_instr *alt,__le32 *origptr,__le32 *updptr,int nr_inst);@@ -184,6 +185,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr);void free_hyp_pgds (void ); +void kvm_unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size);void stage2_unmap_vm (struct kvm *kvm);int kvm_init_stage2_mmu (struct kvm *kvm,struct kvm_s2_mmu *mmu);void kvm_free_stage2_pgd (struct kvm_s2_mmu *mmu);@@ -306,5 +308,11 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT)); } +static inline u64 get_vmid (u64 vttbr) +{ +return (vttbr &VTTBR_VMID_MASK (kvm_get_vmid_bits ())) >> +VTTBR_VMID_SHIFT ; +} +#endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ arch/arm64/include/asm/kvm_nested.h +7
-0@@ -61,6 +61,13 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl) (cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN))); } +extern void kvm_init_nested (struct kvm *kvm); +extern int kvm_vcpu_init_nested (struct kvm_vcpu *vcpu); +extern void kvm_init_nested_s2_mmu (struct kvm_s2_mmu *mmu); +extern struct kvm_s2_mmu *lookup_s2_mmu (struct kvm *kvm,u64 vttbr,u64 hcr); +extern void kvm_vcpu_load_hw_mmu (struct kvm_vcpu *vcpu); +extern void kvm_vcpu_put_hw_mmu (struct kvm_vcpu *vcpu); +int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe);extern bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg,u64 control_bit);arch/arm64/kvm/arm.c +15
-1@@ -35,6 +35,7 @@#include <asm/kvm_arm.h> #include <asm/kvm_asm.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include <asm/kvm_emulate.h> #include <asm/sections.h> @@ -142,6 +143,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)if (ret)return ret; +kvm_init_nested (kvm); + ret =create_hyp_mappings (kvm, kvm +1 ,PAGE_HYP );if (ret) goto out_free_stage2_pgd;@@ -385,6 +388,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)struct kvm_s2_mmu *mmu;int *last_ran; +if (nested_virt_in_use (vcpu)) +kvm_vcpu_load_hw_mmu (vcpu); + mmu = vcpu->arch.hw_mmu; last_ran =this_cpu_ptr (mmu->last_vcpu_ran);@@ -426,6 +432,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)kvm_vgic_put (vcpu);kvm_vcpu_pmu_restore_host (vcpu); +if (nested_virt_in_use (vcpu)) +kvm_vcpu_put_hw_mmu (vcpu); + vcpu->cpu =-1 ; }@@ -1026,8 +1035,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, vcpu->arch.target = phys_target; +/* Prepare for nested if required */ + ret =kvm_vcpu_init_nested (vcpu); +/* Now we know what it is, we can reset it. */ - ret =kvm_reset_vcpu (vcpu); +if (!ret) + ret =kvm_reset_vcpu (vcpu); +if (ret) { vcpu->arch.target =-1 ;bitmap_zero (vcpu->arch.features,KVM_VCPU_MAX_FEATURES );arch/arm64/kvm/mmu.c +8
-10@@ -113,7 +113,7 @@ static bool kvm_is_device_pfn(unsigned long pfn)* does. */ /** -* unmap_stage2_range -- Clear stage2 page table entries to unmap a range +* kvm_unmap_stage2_range -- Clear stage2 page table entries to unmap a range * @mmu: The KVM stage-2 MMU pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap @@ -136,7 +136,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 may_block)); } -static void unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size) +void kvm_unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size) {__unmap_stage2_range (mmu, start, size, true); }@@ -391,6 +391,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) mmu->pgt = pgt; mmu->pgd_phys =__pa (pgt->pgd); mmu->vmid.vmid_gen =0 ; + +kvm_init_nested_s2_mmu (mmu); +return 0 ; out_destroy_pgtable:@@ -435,7 +438,7 @@ static void stage2_unmap_memslot(struct kvm *kvm,if (!(vma->vm_flags &VM_PFNMAP )) {gpa_t gpa = addr + (vm_start - memslot->userspace_addr); -unmap_stage2_range (&kvm->arch.mmu, gpa, vm_end - vm_start); +kvm_unmap_stage2_range (&kvm->arch.mmu, gpa, vm_end - vm_start); } hva = vm_end; }while (hva < reg_end);@@ -1360,7 +1363,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,spin_lock (&kvm->mmu_lock);if (ret) -unmap_stage2_range (&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size); +kvm_unmap_stage2_range (&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size);else if (!cpus_have_final_cap (ARM64_HAS_STAGE2_FWB ))stage2_flush_memslot (kvm, memslot);spin_unlock (&kvm->mmu_lock);@@ -1377,11 +1380,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) { } -void kvm_arch_flush_shadow_all (struct kvm *kvm) -{ -kvm_free_stage2_pgd (&kvm->arch.mmu); -} -void kvm_arch_flush_shadow_memslot (struct kvm *kvm,struct kvm_memory_slot *slot) {@@ -1389,7 +1387,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,phys_addr_t size = slot->npages <<PAGE_SHIFT ;spin_lock (&kvm->mmu_lock); -unmap_stage2_range (&kvm->arch.mmu, gpa, size); +kvm_unmap_stage2_range (&kvm->arch.mmu, gpa, size);spin_unlock (&kvm->mmu_lock); }arch/arm64/kvm/nested.c +10
-0@@ -19,12 +19,177 @@#include <linux/kvm.h> #include <linux/kvm_host.h> +#include <asm/kvm_arm.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_mmu.h> #include <asm/kvm_nested.h> #include <asm/sysreg.h> #include "sys_regs.h" +void kvm_init_nested (struct kvm *kvm) +{ + kvm->arch.nested_mmus =NULL ; + kvm->arch.nested_mmus_size =0 ; +} + +int kvm_vcpu_init_nested (struct kvm_vcpu *vcpu) +{
arch/arm64/kvm/nested.c line 47
Quoted 10 lines
arch/arm64/kvm/nested.c +10
-0+struct kvm *kvm = vcpu->kvm; +struct kvm_s2_mmu *tmp; +int num_mmus; +int ret = -ENOMEM ; + +if (!test_bit (KVM_ARM_VCPU_HAS_EL2 , vcpu->arch.features)) +return 0 ; + +if (!cpus_have_final_cap (ARM64_HAS_NESTED_VIRT )) +return -EINVAL ;
nit: returning a "not supported" kind of errno?
Quoted 176 lines
arch/arm64/kvm/nested.c +163
-1+ +mutex_lock (&kvm->lock); + +/* +* Let's treat memory allocation failures as benign: If we fail to +* allocate anything, return an error and keep the allocated array +* alive. Userspace may try to recover by intializing the vcpu +* again, and there is no reason to affect the whole VM for this. +*/ + num_mmus =atomic_read (&kvm->online_vcpus) *2 ; + tmp =krealloc (kvm->arch.nested_mmus, + num_mmus *sizeof (*kvm->arch.nested_mmus), +GFP_KERNEL | __GFP_ZERO); +if (tmp) { +if (kvm_init_stage2_mmu (kvm, &tmp[num_mmus -1 ]) || +kvm_init_stage2_mmu (kvm, &tmp[num_mmus -2 ])) { +kvm_free_stage2_pgd (&tmp[num_mmus -1 ]); +kvm_free_stage2_pgd (&tmp[num_mmus -2 ]); + }else { + kvm->arch.nested_mmus_size = num_mmus; + ret =0 ; + } + + kvm->arch.nested_mmus = tmp; + } + +mutex_unlock (&kvm->lock); +return ret; +} + +/* Must be called with kvm->lock held */ +struct kvm_s2_mmu *lookup_s2_mmu (struct kvm *kvm,u64 vttbr,u64 hcr) +{ +bool nested_stage2_enabled = hcr &HCR_VM ; +int i; + +/* Don't consider the CnP bit for the vttbr match */ + vttbr = vttbr & ~VTTBR_CNP_BIT ; + +/* +* Two possibilities when looking up a S2 MMU context: +* +* - either S2 is enabled in the guest, and we need a context that +* is S2-enabled and matches the full VTTBR (VMID+BADDR), which +* makes it safe from a TLB conflict perspective (a broken guest +* won't be able to generate them), +* +* - or S2 is disabled, and we need a context that is S2-disabled +* and matches the VMID only, as all TLBs are tagged by VMID even +* if S2 translation is enabled. +*/ +for (i =0 ; i < kvm->arch.nested_mmus_size; i++) { +struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + +if (!kvm_s2_mmu_valid (mmu)) +continue ; + +if (nested_stage2_enabled && + mmu->nested_stage2_enabled && + vttbr == mmu->vttbr) +return mmu; + +if (!nested_stage2_enabled && + !mmu->nested_stage2_enabled && +get_vmid (vttbr) ==get_vmid (mmu->vttbr)) +return mmu; + } +return NULL ; +} + +static struct kvm_s2_mmu *get_s2_mmu_nested (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +u64 vttbr =vcpu_read_sys_reg (vcpu,VTTBR_EL2 ); +u64 hcr=vcpu_read_sys_reg (vcpu,HCR_EL2 ); +struct kvm_s2_mmu *s2_mmu; +int i; + + s2_mmu =lookup_s2_mmu (kvm, vttbr, hcr); +if (s2_mmu) + goto out; + +/* +* Make sure we don't always search from the same point, or we +* will always reuse a potentially active context, leaving +* free contexts unused. +*/ +for (i = kvm->arch.nested_mmus_next; + i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next); + i++) { + s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size]; + +if (atomic_read (&s2_mmu->refcnt) ==0 ) +break ; + } +BUG_ON (atomic_read (&s2_mmu->refcnt));/* We have struct MMUs to spare */ + +/* Set the scene for the next search */ + kvm->arch.nested_mmus_next = (i +1 ) % kvm->arch.nested_mmus_size; + +if (kvm_s2_mmu_valid (s2_mmu)) { +/* Clear the old state */ +kvm_unmap_stage2_range (s2_mmu,0 ,kvm_phys_size (kvm)); +if (s2_mmu->vmid.vmid_gen) +kvm_call_hyp (__kvm_tlb_flush_vmid, s2_mmu); + } + +/* +* The virtual VMID (modulo CnP) will be used as a key when matching +* an existing kvm_s2_mmu. +*/ + s2_mmu->vttbr = vttbr & ~VTTBR_CNP_BIT ; + s2_mmu->nested_stage2_enabled = hcr &HCR_VM ; + +out: +atomic_inc (&s2_mmu->refcnt); +return s2_mmu; +} + +void kvm_init_nested_s2_mmu (struct kvm_s2_mmu *mmu) +{ + mmu->vttbr =1 ; + mmu->nested_stage2_enabled = false; +atomic_set (&mmu->refcnt,0 ); +} + +void kvm_vcpu_load_hw_mmu (struct kvm_vcpu *vcpu) +{ +if (is_hyp_ctxt (vcpu)) { + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + }else { +spin_lock (&vcpu->kvm->mmu_lock); + vcpu->arch.hw_mmu =get_s2_mmu_nested (vcpu); +spin_unlock (&vcpu->kvm->mmu_lock); + } +} + +void kvm_vcpu_put_hw_mmu (struct kvm_vcpu *vcpu) +{ +if (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu) { +atomic_dec (&vcpu->arch.hw_mmu->refcnt); + vcpu->arch.hw_mmu =NULL ; + } +} +/* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor @@ -43,6 +208,24 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)return -EINVAL ; } +void kvm_arch_flush_shadow_all (struct kvm *kvm) +{ +int i; + +for (i =0 ; i < kvm->arch.nested_mmus_size; i++) { +struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + +WARN_ON (atomic_read (&mmu->refcnt)); + +if (!atomic_read (&mmu->refcnt)) +kvm_free_stage2_pgd (mmu); + } +kfree (kvm->arch.nested_mmus); + kvm->arch.nested_mmus =NULL ; + kvm->arch.nested_mmus_size =0 ; +kvm_free_stage2_pgd (&kvm->arch.mmu); +} +#define FEATURE (x) (GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT))/* -- 2.29.2
Quoted 4 lines
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 33/66] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
Quoted 2 lines · Haibo Xu
On Fri, 11 Dec 2020 at 00:04, Marc Zyngier <[email protected]> wrote:
[...]
Quoted 28 lines
Quoted 28 lines
arch/arm64/kvm/nested.c +14
-0@@ -19,12 +19,177 @@#include <linux/kvm.h> #include <linux/kvm_host.h> +#include <asm/kvm_arm.h> #include <asm/kvm_emulate.h> +#include <asm/kvm_mmu.h> #include <asm/kvm_nested.h> #include <asm/sysreg.h> #include "sys_regs.h" +void kvm_init_nested (struct kvm *kvm) +{ + kvm->arch.nested_mmus =NULL ; + kvm->arch.nested_mmus_size =0 ; +} + +int kvm_vcpu_init_nested (struct kvm_vcpu *vcpu) +{ +struct kvm *kvm = vcpu->kvm; +struct kvm_s2_mmu *tmp; +int num_mmus; +int ret = -ENOMEM ;
Quoted 8 lines
arch/arm64/kvm/nested.c line 47
Quoted 6 lines
arch/arm64/kvm/nested.c +6
-0+ +if (!test_bit (KVM_ARM_VCPU_HAS_EL2 , vcpu->arch.features)) +return 0 ; + +if (!cpus_have_final_cap (ARM64_HAS_NESTED_VIRT )) +return -EINVAL ;nit: returning a "not supported" kind of errno?
I think this is consistent with what we otherwise return when there is a mismatch between requested features from userspace and host capabilities (such as asking for a 32bit guest on a 64bit only CPU).
What seems to be missing though is a way to *advertise* the feature to userspace as such:
arch/arm64/kvm/arm.c +3 -0
@@ -243,6 +243,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_ARM_EL1_32BIT : r =cpus_have_const_cap (ARM64_HAS_32BIT_EL1 );break ; +case KVM_CAP_ARM_EL2 : + r =cpus_have_const_cap (ARM64_HAS_NESTED_VIRT ); +break ;case KVM_CAP_GUEST_DEBUG_HW_BPS : r =get_num_brps ();break ;
include/uapi/linux/kvm.h +1 -0
@@ -1058,6 +1058,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_ENFORCE_PV_FEATURE_CPUID 190#define KVM_CAP_SYS_HYPERV_CPUID 191#define KVM_CAP_DIRTY_LOG_RING 192 +#define KVM_CAP_ARM_EL2 193#ifdef KVM_CAP_IRQ_ROUTING
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 34/66 KVM: arm64: nv: Implement nested Stage-2 page table walk logic
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Based on the pseudo-code in the ARM ARM, implement a stage 2 software page table walker.
--- arch/arm64/include/asm/esr.h | 1 + arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_nested.h | 13 ++ arch/arm64/kvm/nested.c | 276 ++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+)
arch/arm64/include/asm/esr.h +1 -0
@@ -127,6 +127,7 @@#define ESR_ELx_CM (UL(1) << ESR_ELx_CM_SHIFT)/* ISS field definitions for exceptions taken in to Hyp */ +#define ESR_ELx_FSC_ADDRSZ (0x00)#define ESR_ELx_CV (UL(1) << 24)#define ESR_ELx_COND_SHIFT (20)#define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT)
arch/arm64/include/asm/kvm_arm.h +2 -0
@@ -264,6 +264,8 @@#define VTTBR_VMID_SHIFT (UL(48))#define VTTBR_VMID_MASK (size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) +#define SCTLR_EE (UL(1) << 25) +/* Hyp System Trap Register */ #define HSTR_EL2_T (x) (1 << x)
arch/arm64/include/asm/kvm_nested.h +13 -0
@@ -68,6 +68,19 @@ extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr);extern void kvm_vcpu_load_hw_mmu (struct kvm_vcpu *vcpu);extern void kvm_vcpu_put_hw_mmu (struct kvm_vcpu *vcpu); +struct kvm_s2_trans { +phys_addr_t output; +unsigned long block_size; +bool writable; +bool readable; +int level; +u32 esr; +u64 upper_attr; +}; + +extern int kvm_walk_nested_s2 (struct kvm_vcpu *vcpu,phys_addr_t gipa, +struct kvm_s2_trans *result); +int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe);extern bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg,u64 control_bit);
arch/arm64/kvm/nested.c +276 -0
282 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 35/66 KVM: arm64: nv: Handle shadow stage 2 page faults
In reply to Marc ZyngierIf we are faulting on a shadow stage 2 translation, we first walk the guest hypervisor's stage 2 page table to see if it has a mapping. If not, we inject a stage 2 page fault to the virtual EL2. Otherwise, we create a mapping in the shadow stage 2 page table.
Note that we have to deal with two IPAs when we got a shadow stage 2 page fault. One is the address we faulted on, and is in the L2 guest phys space. The other is from the guest stage-2 page table walk, and is in the L1 guest phys space. To differentiate them, we rename variables so that fault_ipa is used for the former and ipa is used for the latter.
--- arch/arm64/include/asm/kvm_emulate.h | 6 ++ arch/arm64/include/asm/kvm_nested.h | 18 ++++++ arch/arm64/kvm/mmu.c | 93 ++++++++++++++++++++++------ arch/arm64/kvm/nested.c | 39 ++++++++++++ 4 files changed, 138 insertions(+), 18 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +6 -0
@@ -586,4 +586,10 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu) vcpu->arch.flags |= KVM_ARM64_INCREMENT_PC; } +static inline bool kvm_is_shadow_s2_fault (struct kvm_vcpu *vcpu) +{ +return (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu && + vcpu->arch.hw_mmu->nested_stage2_enabled); +} +#endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/include/asm/kvm_nested.h +18 -0
@@ -78,9 +78,27 @@ struct kvm_s2_trans {u64 upper_attr; }; +static inline phys_addr_t kvm_s2_trans_output (struct kvm_s2_trans *trans) +{ +return trans->output; +} + +static inline unsigned long kvm_s2_trans_size (struct kvm_s2_trans *trans) +{ +return trans->block_size; +} + +static inline u32 kvm_s2_trans_esr (struct kvm_s2_trans *trans) +{ +return trans->esr; +} +extern int kvm_walk_nested_s2 (struct kvm_vcpu *vcpu,phys_addr_t gipa,struct kvm_s2_trans *result); +extern int kvm_s2_handle_perm_fault (struct kvm_vcpu *vcpu, +struct kvm_s2_trans *trans); +extern int kvm_inject_s2_fault (struct kvm_vcpu *vcpu,u64 esr_el2);int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe);extern bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg,u64 control_bit);
arch/arm64/kvm/mmu.c +75 -18
196 lines not on this page · raw message
arch/arm64/kvm/nested.c +39 -0
@@ -466,6 +466,45 @@ void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) } } +/* +* Returns non-zero if permission fault is handled by injecting it to the next +* level hypervisor. +*/ +int kvm_s2_handle_perm_fault (struct kvm_vcpu *vcpu,struct kvm_s2_trans *trans) +{ +unsigned long fault_status =kvm_vcpu_trap_get_fault_type (vcpu); +bool forward_fault = false; + + trans->esr =0 ; + +if (fault_status !=FSC_PERM ) +return 0 ; + +if (kvm_vcpu_trap_is_iabt (vcpu)) { + forward_fault = (trans->upper_attr &BIT (54 )); + }else { +bool write_fault =kvm_is_write_fault (vcpu); + + forward_fault = ((write_fault && !trans->writable) || + (!write_fault && !trans->readable)); + } + +if (forward_fault) { + trans->esr =esr_s2_fault (vcpu, trans->level, ESR_ELx_FSC_PERM); +return 1 ; + } + +return 0 ; +} + +int kvm_inject_s2_fault (struct kvm_vcpu *vcpu,u64 esr_el2) +{ +vcpu_write_sys_reg (vcpu, vcpu->arch.fault.far_el2,FAR_EL2 ); +vcpu_write_sys_reg (vcpu, vcpu->arch.fault.hpfar_el2,HPFAR_EL2 ); + +return kvm_inject_nested_sync (vcpu, esr_el2); +} +/* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 36/66 KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
In reply to Marc ZyngierWhen mapping a page in a shadow stage-2, special care must be taken not to be more permissive than the guest is (writable or readable page when the guest hasn't set that permission).
--- arch/arm64/include/asm/kvm_nested.h | 15 +++++++++++++++ arch/arm64/kvm/mmu.c | 14 +++++++++++++- arch/arm64/kvm/nested.c | 2 +- 3 files changed, 29 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_nested.h +15 -0
@@ -93,6 +93,21 @@ static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans)return trans->esr; } +static inline bool kvm_s2_trans_readable (struct kvm_s2_trans *trans) +{ +return trans->readable; +} + +static inline bool kvm_s2_trans_writable (struct kvm_s2_trans *trans) +{ +return trans->writable; +} + +static inline bool kvm_s2_trans_executable (struct kvm_s2_trans *trans) +{ +return !(trans->upper_attr &BIT (54 )); +} +extern int kvm_walk_nested_s2 (struct kvm_vcpu *vcpu,phys_addr_t gipa,struct kvm_s2_trans *result);
arch/arm64/kvm/mmu.c +13 -1
@@ -887,6 +887,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,if (exec_fault && device)return -ENOEXEC ; +/* +* Potentially reduce shadow S2 permissions to match the guest's own +* S2. For exec faults, we'd only reach this point if the guest +* actually allowed it (see kvm_s2_handle_perm_fault). +*/ +if (kvm_is_shadow_s2_fault (vcpu)) { + writable &=kvm_s2_trans_writable (nested); +if (!kvm_s2_trans_readable (nested)) + prot &= ~KVM_PGTABLE_PROT_R ; + } +spin_lock (&kvm->mmu_lock); pgt = vcpu->arch.hw_mmu->pgt;if (mmu_notifier_retry (kvm, mmu_seq))
@@ -915,7 +926,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,if (device) prot |=KVM_PGTABLE_PROT_DEVICE ; -else if (cpus_have_const_cap (ARM64_HAS_CACHE_DIC )) +else if (cpus_have_const_cap (ARM64_HAS_CACHE_DIC ) && +kvm_s2_trans_executable (nested)) prot |=KVM_PGTABLE_PROT_X ;if (fault_status ==FSC_PERM && !(logging_active && writable)) {
arch/arm64/kvm/nested.c +1 -1
@@ -481,7 +481,7 @@ int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans)return 0 ;if (kvm_vcpu_trap_is_iabt (vcpu)) { - forward_fault = (trans->upper_attr &BIT (54 )); + forward_fault = !kvm_s2_trans_executable (trans); }else {bool write_fault =kvm_is_write_fault (vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 37/66 KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Unmap/flush shadow stage 2 page tables for the nested VMs as well as the stage 2 page table for the guest hypervisor.
Note: A bunch of the code in mmu.c relating to MMU notifiers is currently dealt with in an extremely abrupt way, for example by clearing out an entire shadow stage-2 table. This will be handled in a more efficient way using the reverse mapping feature in a later version of the patch series.
--- arch/arm64/include/asm/kvm_mmu.h | 3 +++ arch/arm64/include/asm/kvm_nested.h | 3 +++ arch/arm64/kvm/mmu.c | 34 ++++++++++++++++++++++--- arch/arm64/kvm/nested.c | 39 +++++++++++++++++++++++++++++ 4 files changed, 75 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_mmu.h +3 -0
@@ -183,6 +183,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **haddr);int create_hyp_exec_mappings (phys_addr_t phys_addr,size_t size,void **haddr); +void kvm_stage2_flush_range (struct kvm_s2_mmu *mmu, +phys_addr_t addr,phys_addr_t end);void free_hyp_pgds (void );void kvm_unmap_stage2_range (struct kvm_s2_mmu *mmu,phys_addr_t start,u64 size);
@@ -191,6 +193,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu);void kvm_free_stage2_pgd (struct kvm_s2_mmu *mmu);int kvm_phys_addr_ioremap (struct kvm *kvm,phys_addr_t guest_ipa,phys_addr_t pa,unsigned long size,bool writable); +void kvm_stage2_wp_range (struct kvm_s2_mmu *mmu,phys_addr_t addr,phys_addr_t end);int kvm_handle_guest_abort (struct kvm_vcpu *vcpu);
arch/arm64/include/asm/kvm_nested.h +3 -0
@@ -114,6 +114,9 @@ extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,extern int kvm_s2_handle_perm_fault (struct kvm_vcpu *vcpu,struct kvm_s2_trans *trans);extern int kvm_inject_s2_fault (struct kvm_vcpu *vcpu,u64 esr_el2); +extern void kvm_nested_s2_wp (struct kvm *kvm); +extern void kvm_nested_s2_clear (struct kvm *kvm); +extern void kvm_nested_s2_flush (struct kvm *kvm);int handle_wfx_nested (struct kvm_vcpu *vcpu,bool is_wfe);extern bool __forward_traps (struct kvm_vcpu *vcpu,unsigned int reg,u64 control_bit);
arch/arm64/kvm/mmu.c +30 -4
@@ -141,13 +141,20 @@ void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)__unmap_stage2_range (mmu, start, size, true); } +void kvm_stage2_flush_range (struct kvm_s2_mmu *mmu, +phys_addr_t addr,phys_addr_t end) +{ +stage2_apply_range_resched (mmu->kvm, addr, end, kvm_pgtable_stage2_flush); +} +static void stage2_flush_memslot (struct kvm *kvm,struct kvm_memory_slot *memslot) {phys_addr_t addr = memslot->base_gfn <<PAGE_SHIFT ;phys_addr_t end = addr +PAGE_SIZE * memslot->npages; +struct kvm_s2_mmu *mmu = &kvm->arch.mmu; -stage2_apply_range_resched (kvm, addr, end, kvm_pgtable_stage2_flush); +kvm_stage2_flush_range (mmu, addr, end); }/**
@@ -170,6 +177,8 @@ static void stage2_flush_vm(struct kvm *kvm)kvm_for_each_memslot (memslot, slots)stage2_flush_memslot (kvm, memslot); +kvm_nested_s2_flush (kvm); +spin_unlock (&kvm->mmu_lock);srcu_read_unlock (&kvm->srcu, idx); }
@@ -465,6 +474,8 @@ void stage2_unmap_vm(struct kvm *kvm)kvm_for_each_memslot (memslot, slots)stage2_unmap_memslot (kvm, memslot); +kvm_nested_s2_clear (kvm); +spin_unlock (&kvm->mmu_lock);mmap_read_unlock (current->mm);srcu_read_unlock (&kvm->srcu, idx);
@@ -539,7 +550,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,* @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range (struct kvm_s2_mmu *mmu,phys_addr_t addr,phys_addr_t end) +void kvm_stage2_wp_range (struct kvm_s2_mmu *mmu,phys_addr_t addr,phys_addr_t end) {struct kvm *kvm = mmu->kvm;stage2_apply_range_resched (kvm, addr, end, kvm_pgtable_stage2_wrprotect);
@@ -571,7 +582,8 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) end = (memslot->base_gfn + memslot->npages) <<PAGE_SHIFT ;spin_lock (&kvm->mmu_lock); -stage2_wp_range (&kvm->arch.mmu, start, end); +kvm_stage2_wp_range (&kvm->arch.mmu, start, end); +kvm_nested_s2_wp (kvm);spin_unlock (&kvm->mmu_lock);kvm_flush_remote_tlbs (kvm); }
@@ -595,7 +607,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,phys_addr_t start = (base_gfn +__ffs (mask)) <<PAGE_SHIFT ;phys_addr_t end = (base_gfn +__fls (mask) +1 ) <<PAGE_SHIFT ; -stage2_wp_range (&kvm->arch.mmu, start, end); +kvm_stage2_wp_range (&kvm->arch.mmu, start, end); }/*
@@ -610,6 +622,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t gfn_offset, unsigned long mask) {kvm_mmu_write_protect_pt_masked (kvm ,slot ,gfn_offset ,mask ); +kvm_nested_s2_wp (kvm); }static void clean_dcache_guest_page (kvm_pfn_t pfn,unsigned long size)
@@ -1164,6 +1177,7 @@ static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *datbool may_block = flags &MMU_NOTIFIER_RANGE_BLOCKABLE ;__unmap_stage2_range (&kvm->arch.mmu, gpa, size, may_block); +kvm_nested_s2_clear (kvm);return 0 ; }
@@ -1192,6 +1206,7 @@ static int kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data*/ kvm_pgtable_stage2_map (kvm->arch.mmu.pgt, gpa,PAGE_SIZE ,__pfn_to_phys (*pfn),KVM_PGTABLE_PROT_R ,NULL ); +kvm_nested_s2_clear (kvm);return 0 ; }
@@ -1223,12 +1238,22 @@ static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) kpte =kvm_pgtable_stage2_mkold (kvm->arch.mmu.pgt, gpa); pte =__pte (kpte);return pte_valid (pte) &&pte_young (pte); + +/* +* TODO: Handle nested_mmu structures here using the reverse mapping in +* a later version of patch series. +*/ }static int kvm_test_age_hva_handler (struct kvm *kvm,gpa_t gpa,u64 size,void *data) {WARN_ON (size !=PAGE_SIZE && size !=PMD_SIZE && size !=PUD_SIZE );return kvm_pgtable_stage2_is_young (kvm->arch.mmu.pgt, gpa); + +/* +* TODO: Handle nested_mmu structures here using the reverse mapping in +* a later version of patch series. +*/ }int kvm_age_hva (struct kvm *kvm,unsigned long start,unsigned long end)
@@ -1457,6 +1482,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,spin_lock (&kvm->mmu_lock);kvm_unmap_stage2_range (&kvm->arch.mmu, gpa, size); +kvm_nested_s2_clear (kvm);spin_unlock (&kvm->mmu_lock); }
arch/arm64/kvm/nested.c +39 -0
@@ -505,6 +505,45 @@ int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2)return kvm_inject_nested_sync (vcpu, esr_el2); } +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_wp (struct kvm *kvm) +{ +int i; + +for (i =0 ; i < kvm->arch.nested_mmus_size; i++) { +struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + +if (kvm_s2_mmu_valid (mmu)) +kvm_stage2_wp_range (mmu,0 ,kvm_phys_size (kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_clear (struct kvm *kvm) +{ +int i; + +for (i =0 ; i < kvm->arch.nested_mmus_size; i++) { +struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + +if (kvm_s2_mmu_valid (mmu)) +kvm_unmap_stage2_range (mmu,0 ,kvm_phys_size (kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_flush (struct kvm *kvm) +{ +int i; + +for (i =0 ; i < kvm->arch.nested_mmus_size; i++) { +struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + +if (kvm_s2_mmu_valid (mmu)) +kvm_stage2_flush_range (mmu,0 ,kvm_phys_size (kvm)); + } +} +/* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 38/66 KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
This introduces a function prototype to determine if we need to forward system instruction traps to the virtual EL2. The implementation of forward_trap functions for each system instruction will be added in later patches.
--- arch/arm64/kvm/sys_regs.c | 8 ++++++++ arch/arm64/kvm/sys_regs.h | 6 ++++++ 2 files changed, 14 insertions(+)
arch/arm64/kvm/sys_regs.c +8 -0
@@ -2490,6 +2490,14 @@ static void perform_access(struct kvm_vcpu *vcpu,*/ BUG_ON (!r->access); +/* +* Forward this trap to the virtual EL2 if the guest hypervisor has +* configured to trap the current instruction. +*/ +if (nested_virt_in_use (vcpu) && r->forward_trap + &&unlikely (r->forward_trap (vcpu))) +return ; +/* Skip instruction if instructed so */ if (likely (r->access (vcpu, params, r)))kvm_incr_pc (vcpu);
arch/arm64/kvm/sys_regs.h +6 -0
@@ -58,6 +58,12 @@ struct sys_reg_desc {int (*set_user)(struct kvm_vcpu *vcpu,const struct sys_reg_desc *rd,const struct kvm_one_reg *reg,void __user *uaddr); +/* +* Forward the trap to the virtual EL2 if the guest hypervisor has +* configured to trap the current instruction. +*/ +bool (*forward_trap)(struct kvm_vcpu *vcpu); +/* Return mask of REG_* runtime visibility overrides */ unsigned int (*visibility)(const struct kvm_vcpu *vcpu,const struct sys_reg_desc *rd);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 39/66 KVM: arm64: nv: Set a handler for the system instruction traps
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
When HCR.NV bit is set, execution of the EL2 translation regime address aranslation instructions and TLB maintenance instructions are trapped to EL2. In addition, execution of the EL1 translation regime address aranslation instructions and TLB maintenance instructions that are only accessible from EL2 and above are trapped to EL2. In these cases, ESR_EL2.EC will be set to 0x18.
Rework the system instruction emulation framework to handle potentially all system instruction traps other than MSR/MRS instructions. Those system instructions would be AT and TLBI instructions controlled by HCR_EL2.NV, AT, and TTLB bits.
--- arch/arm64/include/asm/kvm_host.h | 4 +-- arch/arm64/kvm/handle_exit.c | 2 +- arch/arm64/kvm/sys_regs.c | 48 +++++++++++++++++++++++++------ 3 files changed, 42 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -2
@@ -369,7 +369,7 @@ struct kvm_vcpu_arch {/* * Guest registers we preserve during guest debugging. * -* These shadow registers are updated by the kvm_handle_sys_reg +* These shadow registers are updated by the kvm_handle_sys * trap handler if the guest accesses or updates them while we * are using guest debug. */
@@ -681,7 +681,7 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu);int kvm_handle_cp14_64 (struct kvm_vcpu *vcpu);int kvm_handle_cp15_32 (struct kvm_vcpu *vcpu);int kvm_handle_cp15_64 (struct kvm_vcpu *vcpu); -int kvm_handle_sys_reg (struct kvm_vcpu *vcpu); +int kvm_handle_sys (struct kvm_vcpu *vcpu);void kvm_reset_sys_regs (struct kvm_vcpu *vcpu);
arch/arm64/kvm/handle_exit.c +1 -1
@@ -248,7 +248,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC32] = handle_smc, [ESR_ELx_EC_HVC64] = handle_hvc, [ESR_ELx_EC_SMC64] = handle_smc, - [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, + [ESR_ELx_EC_SYS64] = kvm_handle_sys, [ESR_ELx_EC_SVE] = handle_sve, [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort,
arch/arm64/kvm/sys_regs.c +39 -9
87 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 40/66 KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
When supporting nested virtualization a guest hypervisor executing AT instructions must be trapped and emulated by the host hypervisor, because untrapped AT instructions operating on S1E1 will use the wrong translation regieme (the one used to emulate virtual EL2 in EL1 instead of virtual EL1) and AT instructions operating on S12 will not work from EL1.
This patch does several things.
1. List and define all AT system instructions to emulate and document the emulation design.
2. Implement AT instruction handling logic in EL2. This will be used to emulate AT instructions executed in the virtual EL2.
AT instruction emulation works by loading the proper processor context, which depends on the trapped instruction and the virtual HCR_EL2, to the EL1 virtual memory control registers and executing AT instructions. Note that ctxt->hw_sys_regs is expected to have the proper processor context before calling the handling function(__kvm_at_insn) implemented in this patch.
4. Emulate AT S1E[01] instructions by issuing the same instructions in EL2. We set the physical EL1 registers, NV and NV1 bits as described in the AT instruction emulation overview.
5. Emulate AT A12E[01] instructions in two steps: First, do the stage-1 translation by reusing the existing AT emulation functions. Second, do the stage-2 translation by walking the guest hypervisor's stage-2 page table in software. Record the translation result to PAR_EL1.
6. Emulate AT S1E2 instructions by issuing the corresponding S1E1 instructions in EL2. We set the physical EL1 registers and the HCR_EL2 register as described in the AT instruction emulation overview.
7. Forward system instruction traps to the virtual EL2 if the corresponding virtual AT bit is set in the virtual HCR_EL2.
[ Much logic above has been reworked by Marc Zyngier ]
--- arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 17 +++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/at.c | 231 +++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/switch.c | 13 +- arch/arm64/kvm/sys_regs.c | 201 ++++++++++++++++++++++++++- 7 files changed, 463 insertions(+), 5 deletions(-) create mode 100644 arch/arm64/kvm/at.c
arch/arm64/include/asm/kvm_arm.h +2 -0
@@ -14,6 +14,7 @@/* Hyp Configuration Register (HCR) bits */ #define HCR_ATA (UL(1) << 56)#define HCR_FWB (UL(1) << 46) +#define HCR_AT (UL(1) << 44)#define HCR_NV1 (UL(1) << 43)#define HCR_NV (UL(1) << 42)#define HCR_API (UL(1) << 41)
@@ -110,6 +111,7 @@#define VTCR_EL2_TG0_16K TCR_TG0_16K#define VTCR_EL2_TG0_64K TCR_TG0_64K#define VTCR_EL2_SH0_MASK TCR_SH0_MASK +#define VTCR_EL2_SH0_SHIFT TCR_SH0_SHIFT#define VTCR_EL2_SH0_INNER TCR_SH0_INNER#define VTCR_EL2_ORGN0_MASK TCR_ORGN0_MASK#define VTCR_EL2_ORGN0_WBWA TCR_ORGN0_WBWA
arch/arm64/include/asm/kvm_asm.h +2 -0
@@ -189,6 +189,8 @@ extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);extern void __kvm_tlb_flush_local_vmid (struct kvm_s2_mmu *mmu);extern void __kvm_timer_set_cntvoff (u64 cntvoff); +extern void __kvm_at_s1e01 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr); +extern void __kvm_at_s1e2 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr);extern int __kvm_vcpu_run (struct kvm_vcpu *vcpu);
arch/arm64/include/asm/sysreg.h +17 -0
@@ -591,6 +591,23 @@#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* AT instructions */ +#define AT_Op0 1 +#define AT_CRn 7 + +#define OP_AT_S1E1R sys_insn(AT_Op0, 0, AT_CRn, 8, 0) +#define OP_AT_S1E1W sys_insn(AT_Op0, 0, AT_CRn, 8, 1) +#define OP_AT_S1E0R sys_insn(AT_Op0, 0, AT_CRn, 8, 2) +#define OP_AT_S1E0W sys_insn(AT_Op0, 0, AT_CRn, 8, 3) +#define OP_AT_S1E1RP sys_insn(AT_Op0, 0, AT_CRn, 9, 0) +#define OP_AT_S1E1WP sys_insn(AT_Op0, 0, AT_CRn, 9, 1) +#define OP_AT_S1E2R sys_insn(AT_Op0, 4, AT_CRn, 8, 0) +#define OP_AT_S1E2W sys_insn(AT_Op0, 4, AT_CRn, 8, 1) +#define OP_AT_S12E1R sys_insn(AT_Op0, 4, AT_CRn, 8, 4) +#define OP_AT_S12E1W sys_insn(AT_Op0, 4, AT_CRn, 8, 5) +#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) +#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) +/* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44))#define SCTLR_ELx_ATA (BIT(43))
arch/arm64/kvm/Makefile +1 -1
@@ -17,7 +17,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ guest.o debug.o reset.o sys_regs.o \ vgic-sys-reg-v3.o fpsimd.o pmu.o \ arch_timer.o \ - emulate-nested.o nested.o \ + emulate-nested.o nested.o at.o \ vgic/vgic.o vgic/vgic-init.o \ vgic/vgic-irqfd.o vgic/vgic-v2.o \ vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/at.c new file +231 -0
231 lines not on this page · raw message
arch/arm64/kvm/hyp/vhe/switch.c +11 -2
@@ -45,9 +45,10 @@ static void __activate_traps(struct kvm_vcpu *vcpu)if (!vcpu_el2_e2h_is_set (vcpu)) {/* * For a guest hypervisor on v8.0, trap and emulate -* the EL1 virtual memory control register accesses. +* the EL1 virtual memory control register accesses +* as well as the AT S1 operations. */ - hcr |=HCR_TVM |HCR_TRVM |HCR_NV1 ; + hcr |=HCR_TVM |HCR_TRVM |HCR_AT |HCR_NV1 ; }else {/* * For a guest hypervisor on v8.1 (VHE), allow to
@@ -70,6 +71,14 @@ static void __activate_traps(struct kvm_vcpu *vcpu) hcr &= ~HCR_TVM ; hcr |= vhcr_el2 & (HCR_TVM |HCR_TRVM ); + +/* +* If we're using the EL1 translation regime +* (TGE clear), then ensure that AT S1 ops are +* trapped too. +*/ +if (!vcpu_el2_tge_is_set (vcpu)) + hcr |=HCR_AT ; } }
arch/arm64/kvm/sys_regs.c +199 -2
217 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 41/66 KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
When supporting nested virtualization a guest hypervisor executing TLBI instructions must be trapped and emulated by the host hypervisor, because the guest hypervisor can only affect physical TLB entries relating to its own execution environment (virtual EL2 in EL1) but not to the nested guests as required by the semantics of the instructions and TLBI instructions might also result in updates (invalidations) to shadow page tables.
This patch does several things.
1. List and define all TLBI system instructions to emulate.
2. Emulate TLBI ALLE2(IS) instruction executed in the virtual EL2. Since we emulate the virtual EL2 in the EL1, we invalidate EL1&0 regime stage 1 TLB entries with setting vttbr_el2 having the VMID of the virtual EL2.
3. Emulate TLBI VAE2* instruction executed in the virtual EL2. Based on the same principle as TLBI ALLE2 instruction, we can simply emulate those instructions by executing corresponding VAE1* instructions with the virtual EL2's VMID assigned by the host hypervisor.
Note that we are able to emulate TLBI ALLE2IS precisely by only invalidating stage 1 TLB entries via TLBI VMALL1IS instruction, but to make it simeple, we reuse the existing function, __kvm_tlb_flush_vmid(), which invalidates both of stage 1 and 2 TLB entries.
4. TLBI ALLE1(IS) instruction invalidates all EL1&0 regime stage 1 and 2 TLB entries (on all PEs in the same Inner Shareable domain). To emulate these instructions, we first need to clear all the mappings in the shadow page tables since executing those instructions implies the change of mappings in the stage 2 page tables maintained by the guest hypervisor. We then need to invalidate all EL1&0 regime stage 1 and 2 TLB entries of all VMIDs, which are assigned by the host hypervisor, for this VM.
5. Based on the same principle as TLBI ALLE1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. But this time we do it only for the current VMID from the guest hypervisor's perspective, not for all VMIDs.
6. Based on the same principle as TLBI ALLE1(IS) and TLBI VMALLS12E1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. We do it only for one mapping for the current VMID from the guest hypervisor's view.
7. Forward system instruction traps to the virtual EL2 if a corresponding bit in the virtual HCR_EL2 is set.
8. Even though a guest hypervisor can execute TLBI instructions that are accesible at EL1 without trap, it's wrong; All those TLBI instructions work based on current VMID, and when running a guest hypervisor current VMID is the one for itself, not the one from the virtual vttbr_el2. So letting a guest hypervisor execute those TLBI instructions results in invalidating its own TLB entries and leaving invalid TLB entries unhandled.
Therefore we trap and emulate those TLBI instructions. The emulation is simple; we find a shadow VMID mapped to the virtual vttbr_el2, set it in the physical vttbr_el2, then execute the same instruction in EL2.
We don't set HCR_EL2.TTLB bit yet.
[ Changes performed by Marc Zynger:
The TLBI handling code more or less directly execute the same instruction that has been trapped (with an EL2->EL1 conversion in the case of an EL2 TLBI), but that's unfortunately not enough:
- TLBIs must be upgraded to the Inner Shareable domain to account for vcpu migration, just like we already have with HCR_EL2.FB.
- The DSB instruction that synchronises these must thus be on the Inner Shareable domain as well.
- Prior to executing the TLBI, we need another DSB ISHST to make sure that the update to the page tables is now visible.
Ordering of system instructions fixed
- The current TLB invalidation code is pretty buggy, as it assume a page mapping. On the contrary, it is likely that TLB invalidation will cover more than a single page, and the size should be decided by the guests configuration (and not the host's).
Since we don't cache the guest mapping sizes in the shadow PT yet, let's assume the worse case (a block mapping) and invalidate that.
Take this opportunity to fix the decoding of the parameter (it isn't a straight IPA).
- In general, we always emulate local TBL invalidations as being as upgraded to the Inner Shareable domain so that we can easily deal with vcpu migration. This is consistent with the fact that we set HCR_EL2.FB when running non-nested VMs.
So let's emulate TLBI ALLE2 as ALLE2IS.
]
[ Changes performed by Christoffer Dall:
Sometimes when we are invalidating the TLB for a certain S2 MMU context, this context can also have EL2 context associated with it and we have to invalidate this too.
]
--- arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 36 ++++++ arch/arm64/kvm/hyp/vhe/switch.c | 8 +- arch/arm64/kvm/hyp/vhe/tlb.c | 81 ++++++++++++ arch/arm64/kvm/mmu.c | 18 ++- arch/arm64/kvm/sys_regs.c | 212 +++++++++++++++++++++++++++++++ 6 files changed, 352 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_asm.h +2 -0
@@ -187,6 +187,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, int level);extern void __kvm_tlb_flush_vmid (struct kvm_s2_mmu *mmu);extern void __kvm_tlb_flush_local_vmid (struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_vae2is (struct kvm_s2_mmu *mmu,u64 va,u64 sys_encoding); +extern void __kvm_tlb_el1_instr (struct kvm_s2_mmu *mmu,u64 val,u64 sys_encoding);extern void __kvm_timer_set_cntvoff (u64 cntvoff);extern void __kvm_at_s1e01 (struct kvm_vcpu *vcpu,u32 op,u64 vaddr);
arch/arm64/include/asm/sysreg.h +36 -0
@@ -608,6 +608,42 @@#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6)#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) +/* TLBI instructions */ +#define TLBI_Op0 1 +#define TLBI_Op1_EL1 0/* Accessible from EL1 or higher */ +#define TLBI_Op1_EL2 4/* Accessible from EL2 or higher */ +#define TLBI_CRn 8 +#define tlbi_insn_el1 (CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL1, TLBI_CRn, (CRm), (Op2)) +#define tlbi_insn_el2 (CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL2, TLBI_CRn, (CRm), (Op2)) + +#define OP_TLBI_VMALLE1IS tlbi_insn_el1(3, 0) +#define OP_TLBI_VAE1IS tlbi_insn_el1(3, 1) +#define OP_TLBI_ASIDE1IS tlbi_insn_el1(3, 2) +#define OP_TLBI_VAAE1IS tlbi_insn_el1(3, 3) +#define OP_TLBI_VALE1IS tlbi_insn_el1(3, 5) +#define OP_TLBI_VAALE1IS tlbi_insn_el1(3, 7) +#define OP_TLBI_VMALLE1 tlbi_insn_el1(7, 0) +#define OP_TLBI_VAE1 tlbi_insn_el1(7, 1) +#define OP_TLBI_ASIDE1 tlbi_insn_el1(7, 2) +#define OP_TLBI_VAAE1 tlbi_insn_el1(7, 3) +#define OP_TLBI_VALE1 tlbi_insn_el1(7, 5) +#define OP_TLBI_VAALE1 tlbi_insn_el1(7, 7) + +#define OP_TLBI_IPAS2E1IS tlbi_insn_el2(0, 1) +#define OP_TLBI_IPAS2LE1IS tlbi_insn_el2(0, 5) +#define OP_TLBI_ALLE2IS tlbi_insn_el2(3, 0) +#define OP_TLBI_VAE2IS tlbi_insn_el2(3, 1) +#define OP_TLBI_ALLE1IS tlbi_insn_el2(3, 4) +#define OP_TLBI_VALE2IS tlbi_insn_el2(3, 5) +#define OP_TLBI_VMALLS12E1IS tlbi_insn_el2(3, 6) +#define OP_TLBI_IPAS2E1 tlbi_insn_el2(4, 1) +#define OP_TLBI_IPAS2LE1 tlbi_insn_el2(4, 5) +#define OP_TLBI_ALLE2 tlbi_insn_el2(7, 0) +#define OP_TLBI_VAE2 tlbi_insn_el2(7, 1) +#define OP_TLBI_ALLE1 tlbi_insn_el2(7, 4) +#define OP_TLBI_VALE2 tlbi_insn_el2(7, 5) +#define OP_TLBI_VMALLS12E1 tlbi_insn_el2(7, 6) +/* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44))#define SCTLR_ELx_ATA (BIT(43))
arch/arm64/kvm/hyp/vhe/switch.c +4 -4
@@ -48,7 +48,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)* the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |=HCR_TVM |HCR_TRVM |HCR_AT |HCR_NV1 ; + hcr |=HCR_TVM |HCR_TRVM |HCR_AT |HCR_TTLB |HCR_NV1 ; }else {/* * For a guest hypervisor on v8.1 (VHE), allow to
@@ -74,11 +74,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu)/* * If we're using the EL1 translation regime -* (TGE clear), then ensure that AT S1 ops are -* trapped too. +* (TGE clear), then ensure that AT S1 and +* TLBI E1 ops are trapped too. */ if (!vcpu_el2_tge_is_set (vcpu)) - hcr |=HCR_AT ; + hcr |=HCR_AT |HCR_TTLB ; } }
arch/arm64/kvm/hyp/vhe/tlb.c +81 -0
84 lines not on this page · raw message
arch/arm64/kvm/mmu.c +17 -1
@@ -80,7 +80,23 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot)*/ void kvm_flush_remote_tlbs (struct kvm *kvm) { -kvm_call_hyp (__kvm_tlb_flush_vmid, &kvm->arch.mmu); +struct kvm_s2_mmu *mmu = &kvm->arch.mmu; + +if (mmu == &kvm->arch.mmu) { +/* +* For a normal (i.e. non-nested) guest, flush entries for the +* given VMID * +*/ +kvm_call_hyp (__kvm_tlb_flush_vmid, mmu); + }else { +/* +* When supporting nested virtualization, we can have multiple +* VMIDs in play for each VCPU in the VM, so it's really not +* worth it to try to quiesce the system and flush all the +* VMIDs that may be in use, instead just nuke the whole thing. +*/ +kvm_call_hyp (__kvm_flush_vm_context); + } }static bool kvm_is_device_pfn (unsigned long pfn)
arch/arm64/kvm/sys_regs.c +212 -0
236 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 42/66 KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's
In reply to Marc ZyngierWhen entering a L2 guest (nested virt enabled, but not in hypervisor context), we need to honor the traps the L1 guest has asked enabled.
For now, just OR the guest's HCR_EL2 into the host's. We may have to do some filtering in the future though.
--- arch/arm64/kvm/hyp/vhe/switch.c | 2 ++ 1 file changed, 2 insertions(+)
arch/arm64/kvm/hyp/vhe/switch.c +2 -0
@@ -80,6 +80,8 @@ static void __activate_traps(struct kvm_vcpu *vcpu)if (!vcpu_el2_tge_is_set (vcpu)) hcr |=HCR_AT |HCR_TTLB ; } + }else if (nested_virt_in_use (vcpu)) { + hcr |=__vcpu_sys_reg (vcpu,HCR_EL2 ); }___activate_traps (vcpu, hcr);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 43/66 KVM: arm64: nv: arch_timer: Support hyp timer emulation
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Emulating EL2 also means emulating the EL2 timers. To do so, we expand our timer framework to deal with at most 4 timers. At any given time, two timers are using the HW timers, and the two others are purely emulated.
The role of deciding which is which at any given time is left to a mapping function which is called every time we need to make such a decision.
--- arch/arm64/include/asm/kvm_host.h | 4 + arch/arm64/kvm/arch_timer.c | 152 +++++++++++++++++++++++++++++- arch/arm64/kvm/sys_regs.c | 7 +- arch/arm64/kvm/trace_arm.h | 6 +- arch/arm64/kvm/vgic/vgic.c | 15 +++ include/kvm/arm_arch_timer.h | 6 ++ include/kvm/arm_vgic.h | 1 + 7 files changed, 186 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_host.h +4 -0
@@ -272,6 +272,10 @@ enum vcpu_sysreg {TPIDR_EL2 ,/* EL2 Software Thread ID Register */ CNTHCTL_EL2 ,/* Counter-timer Hypervisor Control register */ SP_EL2 ,/* EL2 Stack Pointer */ +CNTHP_CTL_EL2 , +CNTHP_CVAL_EL2 , +CNTHV_CTL_EL2 , +CNTHV_CVAL_EL2 , NR_SYS_REGS /* Nothing after this line! */ };
arch/arm64/kvm/arch_timer.c +149 -3
364 lines not on this page · raw message
arch/arm64/kvm/sys_regs.c +6 -1
@@ -1290,6 +1290,11 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, tmr =TIMER_PTIMER ; treg =TIMER_REG_CVAL ;break ; +case SYS_CNTVOFF_EL2 : + tmr =TIMER_VTIMER ; + treg =TIMER_REG_VOFF ; +break ; +default :BUG (); }
@@ -2101,7 +2106,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, - { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, + { SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
arch/arm64/kvm/trace_arm.h +5 -1
@@ -272,6 +272,7 @@ TRACE_EVENT(kvm_get_timer_map, __field( unsigned long, vcpu_id ) __field( int, direct_vtimer ) __field( int, direct_ptimer ) + __field( int, emul_vtimer ) __field( int, emul_ptimer ) ),
@@ -280,14 +281,17 @@ TRACE_EVENT(kvm_get_timer_map, __entry->direct_vtimer =arch_timer_ctx_index (map->direct_vtimer); __entry->direct_ptimer = (map->direct_ptimer) ?arch_timer_ctx_index (map->direct_ptimer) :-1 ; + __entry->emul_vtimer = + (map->emul_vtimer) ?arch_timer_ctx_index (map->emul_vtimer) :-1 ; __entry->emul_ptimer = (map->emul_ptimer) ?arch_timer_ctx_index (map->emul_ptimer) :-1 ; ), -TP_printk ("VCPU: %ld, dv: %d, dp: %d, ep: %d" , +TP_printk ("VCPU: %ld, dv: %d, dp: %d, ev: %d, ep: %d" , __entry->vcpu_id, __entry->direct_vtimer, __entry->direct_ptimer, + __entry->emul_vtimer, __entry->emul_ptimer) );
arch/arm64/kvm/vgic/vgic.c +15 -0
@@ -573,6 +573,21 @@ int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid)return 0 ; } +int kvm_vgic_get_map (struct kvm_vcpu *vcpu,unsigned int vintid) +{ +struct vgic_irq *irq =vgic_get_irq (vcpu->kvm, vcpu, vintid); +unsigned long flags; +int ret =-1 ; + +raw_spin_lock_irqsave (&irq->irq_lock, flags); +if (irq->hw) + ret = irq->hwintid; +raw_spin_unlock_irqrestore (&irq->irq_lock, flags); + +vgic_put_irq (vcpu->kvm, irq); +return ret; +} +/** * kvm_vgic_set_owner - Set the owner of an interrupt for a VM *
include/kvm/arm_arch_timer.h +6 -0
@@ -13,6 +13,8 @@enum kvm_arch_timers {TIMER_PTIMER ,TIMER_VTIMER , +TIMER_HVTIMER , +TIMER_HPTIMER ,NR_KVM_TIMERS };
@@ -21,6 +23,7 @@ enum kvm_arch_timer_regs {TIMER_REG_CVAL ,TIMER_REG_TVAL ,TIMER_REG_CTL , + TIMER_REG_VOFF, }; struct arch_timer_context {
@@ -47,6 +50,7 @@ struct arch_timer_context {struct timer_map {struct arch_timer_context *direct_vtimer;struct arch_timer_context *direct_ptimer; +struct arch_timer_context *emul_vtimer;struct arch_timer_context *emul_ptimer; };
@@ -91,6 +95,8 @@ bool kvm_arch_timer_get_input_level(int vintid);#define vcpu_get_timer (v,t) (&vcpu_timer(v)->timers[(t)])#define vcpu_vtimer (v) (&(v)->arch.timer_cpu.timers[TIMER_VTIMER])#define vcpu_ptimer (v) (&(v)->arch.timer_cpu.timers[TIMER_PTIMER]) +#define vcpu_hvtimer (v) (&(v)->arch.timer_cpu.timers[TIMER_HVTIMER]) +#define vcpu_hptimer (v) (&(v)->arch.timer_cpu.timers[TIMER_HPTIMER])#define arch_timer_ctx_index (ctx) ((ctx) - vcpu_timer((ctx)->vcpu)->timers)
include/kvm/arm_vgic.h +1 -0
@@ -353,6 +353,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid,int kvm_vgic_map_phys_irq (struct kvm_vcpu *vcpu,unsigned int host_irq,u32 vintid,bool (*get_input_level)(int vindid));int kvm_vgic_unmap_phys_irq (struct kvm_vcpu *vcpu,unsigned int vintid); +int kvm_vgic_get_map (struct kvm_vcpu *vcpu,unsigned int vintid);bool kvm_vgic_map_is_active (struct kvm_vcpu *vcpu,unsigned int vintid);int kvm_vgic_vcpu_pending_irq (struct kvm_vcpu *vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
kernel test robot
build botRe: [PATCH v3 43/66] KVM: arm64: nv: arch_timer: Support hyp timer emulation
Thank you for the patch! Perhaps something to improve:
Hi Marc,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on kvmarm/next]
[also build test WARNING on next-20201210]
[cannot apply to arm64/for-next/core soc/for-next arm/for-next xlnx/master linus/master kvm/linux-next v5.10-rc7] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Marc-Zyngier/KVM-arm64-ARMv8-3-8-4-Nested-Virtualization-support/20201211-004741 base: https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git next config: arm64-defconfig (attached as .config) compiler: aarch64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/af7a3cada1fa577612a86dd0a7c4016ffed68acc
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Marc-Zyngier/KVM-arm64-ARMv8-3-8-4-Nested-Virtualization-support/20201211-004741 git checkout af7a3cada1fa577612a86dd0a7c4016ffed68acc
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm64
If you fix the issue, kindly add following tag as appropriate
All warnings (new ones prefixed by >>):
arch/arm64/kvm/arch_timer.c: In function 'kvm_timer_vcpu_load_nested_switch':
>> arch/arm64/kvm/arch_timer.c:656:10: warning: variable 'ret' set but not used [-Wunused-but-set-variable]
656 | int hw, ret;
| ^~~vim +/ret +656 arch/arm64/kvm/arch_timer.c
652
653 static void kvm_timer_vcpu_load_nested_switch(struct kvm_vcpu *vcpu,
654 struct timer_map *map)
655 {
> 656 int hw, ret;
657
658 if (!irqchip_in_kernel(vcpu->kvm))
659 return;
660
661 /*
662 * We only ever unmap the vtimer irq on a VHE system that runs nested
663 * virtualization, in which case we have both a valid emul_vtimer,
664 * emul_ptimer, direct_vtimer, and direct_ptimer.
665 *
666 * Since this is called from kvm_timer_vcpu_load(), a change between
667 * vEL2 and vEL1/0 will have just happened, and the timer_map will
668 * represent this, and therefore we switch the emul/direct mappings
669 * below.
670 */
671 hw = kvm_vgic_get_map(vcpu, map->direct_vtimer->irq.irq);
672 if (hw < 0) {
673 kvm_vgic_unmap_phys_irq(vcpu, map->emul_vtimer->irq.irq);
674 kvm_vgic_unmap_phys_irq(vcpu, map->emul_ptimer->irq.irq);
675
676 ret = kvm_vgic_map_phys_irq(vcpu,
677 map->direct_vtimer->host_timer_irq,
678 map->direct_vtimer->irq.irq,
679 kvm_arch_timer_get_input_level);
680 ret = kvm_vgic_map_phys_irq(vcpu,
681 map->direct_ptimer->host_timer_irq,
682 map->direct_ptimer->irq.irq,
683 kvm_arch_timer_get_input_level); 684 }
685 }
686
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Marc Zyngier
maintainerPATCH v3 44/66 KVM: arm64: nv: Add handling of EL2-specific timer registers
In reply to Marc ZyngierAdd the required handling for EL2 and EL02 registers, as well as EL1 registers used in the E2H context.
--- arch/arm64/include/asm/sysreg.h | 6 +++ arch/arm64/kvm/sys_regs.c | 87 +++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+)
arch/arm64/include/asm/sysreg.h +6 -0
@@ -559,6 +559,12 @@#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3)#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) +#define SYS_CNTHP_TVAL_EL2 sys_reg(3, 4, 14, 2, 0) +#define SYS_CNTHP_CTL_EL2 sys_reg(3, 4, 14, 2, 1) +#define SYS_CNTHP_CVAL_EL2 sys_reg(3, 4, 14, 2, 2) +#define SYS_CNTHV_TVAL_EL2 sys_reg(3, 4, 14, 3, 0) +#define SYS_CNTHV_CTL_EL2 sys_reg(3, 4, 14, 3, 1) +#define SYS_CNTHV_CVAL_EL2 sys_reg(3, 4, 14, 3, 2)/* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)
arch/arm64/kvm/sys_regs.c +87 -0
119 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 45/66 KVM: arm64: nv: Load timer before the GIC
In reply to Marc ZyngierIn order for vgic_v3_load_nested to be able to observe which timer interrupts have the HW bit set for the current context, the timers must have been loaded in the new mode and the right timer mapped to their corresponding HW IRQs.
At the moment, we load the GIC first, meaning that timer interrupts injected to an L2 guest will never have the HW bit set (we see the old configuration).
Swapping the two loads solves this particular problem.
--- arch/arm64/kvm/arm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/arm.c +1 -1
@@ -405,8 +405,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu->cpu = cpu; -kvm_vgic_load (vcpu);kvm_timer_vcpu_load (vcpu); +kvm_vgic_load (vcpu);if (has_vhe ())kvm_vcpu_load_sysregs_vhe (vcpu);kvm_arch_vcpu_load_fp (vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 46/66 KVM: arm64: nv: Nested GICv3 Support
In reply to Marc ZyngierFrom: Jintack Lim <[email protected]>
When entering a nested VM, we set up the hypervisor control interface based on what the guest hypervisor has set. Especially, we investigate each list register written by the guest hypervisor whether HW bit is set. If so, we translate hw irq number from the guest's point of view to the real hardware irq number if there is a mapping.
--- arch/arm64/include/asm/kvm_emulate.h | 8 +- arch/arm64/include/asm/kvm_host.h | 9 +- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arm.c | 13 +- arch/arm64/kvm/nested.c | 16 +++ arch/arm64/kvm/sys_regs.c | 179 ++++++++++++++++++++++++++- arch/arm64/kvm/vgic/vgic-v3-nested.c | 178 ++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3.c | 26 ++++ arch/arm64/kvm/vgic/vgic.c | 27 ++++ include/kvm/arm_vgic.h | 18 +++ 11 files changed, 468 insertions(+), 9 deletions(-) create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c
arch/arm64/include/asm/kvm_emulate.h +7 -1
@@ -497,7 +497,13 @@ static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu)static inline unsigned long kvm_vcpu_get_mpidr_aff (struct kvm_vcpu *vcpu) { -return vcpu_read_sys_reg (vcpu,MPIDR_EL1 ) &MPIDR_HWID_BITMASK ; +/* +* Use the in-memory view for MPIDR_EL1. It can't be changed by the +* guest, and is also accessed from the context of *another* vcpu, +* so anything using some other state (such as the NV state that is +* used by vcpu_read_sys_reg) will eventually go wrong. +*/ +return __vcpu_sys_reg (vcpu,MPIDR_EL1 ) &MPIDR_HWID_BITMASK ; }static inline void kvm_vcpu_set_be (struct kvm_vcpu *vcpu)
arch/arm64/include/asm/kvm_host.h +5 -4
@@ -42,10 +42,11 @@#define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) -#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) -#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) -#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) +#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) +#define KVM_REQ_RELOAD_GICv4 KVM_ARCH_REQ(4) +#define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(5)#define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET)
arch/arm64/include/asm/kvm_nested.h +1 -0
@@ -67,6 +67,7 @@ extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu);extern struct kvm_s2_mmu *lookup_s2_mmu (struct kvm *kvm,u64 vttbr,u64 hcr);extern void kvm_vcpu_load_hw_mmu (struct kvm_vcpu *vcpu);extern void kvm_vcpu_put_hw_mmu (struct kvm_vcpu *vcpu); +extern void check_nested_vcpu_requests (struct kvm_vcpu *vcpu);struct kvm_s2_trans {phys_addr_t output;
arch/arm64/kvm/Makefile +1 -1
@@ -23,5 +23,5 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ - vgic/vgic-its.o vgic/vgic-debug.o + vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o kvm-$(CONFIG_KVM_ARM_PMU) += pmu-emul.o
arch/arm64/kvm/arm.c +11 -2
@@ -692,6 +692,8 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)vgic_v4_load (vcpu);preempt_enable (); } + +check_nested_vcpu_requests (vcpu); } }
@@ -738,9 +740,16 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)*/ cond_resched (); -update_vmid (&vcpu->arch.hw_mmu->vmid); - +/* +* A nested exeption triggered by a vcpu request (such +* as an interrupt injected in a guest hypervisor) can +* change the currently used VMID (by switching to a +* different translation regime. It is thus necesary +* to update the VMID *after* all requests have been +* processed. +*/ check_vcpu_requests (vcpu); +update_vmid (&vcpu->arch.hw_mmu->vmid);/* * Preparing the interrupts to be injected also
arch/arm64/kvm/nested.c +16 -0
@@ -580,6 +580,22 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm)kvm_free_stage2_pgd (&kvm->arch.mmu); } +bool vgic_state_is_nested (struct kvm_vcpu *vcpu) +{ +bool imo =__vcpu_sys_reg (vcpu,HCR_EL2 ) &HCR_IMO ; +bool fmo =__vcpu_sys_reg (vcpu,HCR_EL2 ) &HCR_FMO ; + +WARN_ONCE (imo != fmo,"Separate virtual IRQ/FIQ settings not supported\n" ); + +return nested_virt_in_use (vcpu) && imo && fmo && !is_hyp_ctxt (vcpu); +} + +void check_nested_vcpu_requests (struct kvm_vcpu *vcpu) +{ +if (kvm_check_request (KVM_REQ_GUEST_HYP_IRQ_PENDING , vcpu)) +kvm_inject_nested_irq (vcpu); +} +#define FEATURE (x) (GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT))/*
arch/arm64/kvm/sys_regs.c +178 -1
215 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic-v3-nested.c new file +178 -0
178 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic-v3.c +26 -0
@@ -6,6 +6,7 @@#include <kvm/arm_vgic.h> #include <asm/kvm_hyp.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include <asm/kvm_asm.h> #include "vgic.h"
@@ -289,6 +290,12 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) vgic_v3->vgic_sre = (ICC_SRE_EL1_DIB |ICC_SRE_EL1_DFB |ICC_SRE_EL1_SRE ); +/* +* If nesting is allowed, force GICv3 onto the nested +* guests as well. +*/ +if (nested_virt_in_use (vcpu)) + vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser =INITIAL_PENDBASER_VALUE ; }else { vgic_v3->vgic_sre =0 ;
@@ -654,6 +661,13 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) {struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; +/* +* vgic_v3_load_nested only affects the LRs in the shadow +* state, so it is fine to pass the nested state around. +*/ +if (vgic_state_is_nested (vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; +/* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen * is dependent on ICC_SRE_EL1.SRE, and we have to perform the
@@ -667,6 +681,9 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)if (has_vhe ())__vgic_v3_activate_traps (cpu_if); +if (vgic_state_is_nested (vcpu)) +vgic_v3_load_nested (vcpu); +WARN_ON (vgic_v4_load (vcpu)); }
@@ -674,6 +691,9 @@ void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) {struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; +if (vgic_state_is_nested (vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; +if (likely (cpu_if->vgic_sre)) cpu_if->vgic_vmcr =kvm_call_hyp_ret (__vgic_v3_read_vmcr); }
@@ -686,8 +706,14 @@ void vgic_v3_put(struct kvm_vcpu *vcpu)vgic_v3_vmcr_sync (vcpu); +if (vgic_state_is_nested (vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; +kvm_call_hyp (__vgic_v3_save_aprs, cpu_if);if (has_vhe ())__vgic_v3_deactivate_traps (cpu_if); + +if (vgic_state_is_nested (vcpu)) +vgic_v3_put_nested (vcpu); }
arch/arm64/kvm/vgic/vgic.c +27 -0
@@ -876,6 +876,10 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) {int used_lrs; +/* If nesting, this is a load/put affair, not flush/sync. */ +if (vgic_state_is_nested (vcpu)) +return ; +/* An empty ap_list_head implies used_lrs == 0 */ if (list_empty (&vcpu->arch.vgic_cpu.ap_list_head))return ;
@@ -920,6 +924,29 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu) !vgic_supports_direct_msis(vcpu->kvm))return ; +/* +* If in a nested state, we must return early. Two possibilities: +* +* - If we have any pending IRQ for the guest and the guest +* expects IRQs to be handled in its virtual EL2 mode (the +* virtual IMO bit is set) and it is not already running in +* virtual EL2 mode, then we have to emulate an IRQ +* exception to virtual EL2. +* +* We do that by placing a request to ourselves which will +* abort the entry procedure and inject the exception at the +* beginning of the run loop. +* +* - Otherwise, do exactly *NOTHING*. The guest state is +* already loaded, and we can carry on with running it. +*/ +if (vgic_state_is_nested (vcpu)) { +if (kvm_vgic_vcpu_pending_irq (vcpu)) +kvm_make_request (KVM_REQ_GUEST_HYP_IRQ_PENDING , vcpu); + +return ; + } +DEBUG_SPINLOCK_BUG_ON (!irqs_disabled ());if (!list_empty (&vcpu->arch.vgic_cpu.ap_list_head)) {
include/kvm/arm_vgic.h +18 -0
@@ -306,6 +306,15 @@ struct vgic_cpu {struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS ]; +/* CPU vif control registers for the virtual GICH interface */ +struct vgic_v3_cpu_if nested_vgic_v3; + +/* +* The shadow vif control register loaded to the hardware when +* running a nested L2 guest with the virtual IMO/FMO bit set. +*/ +struct vgic_v3_cpu_if shadow_vgic_v3; +raw_spinlock_t ap_list_lock;/* Protects the ap_list */ /*
@@ -362,6 +371,13 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu);void kvm_vgic_put (struct kvm_vcpu *vcpu);void kvm_vgic_vmcr_sync (struct kvm_vcpu *vcpu); +void vgic_v3_load_nested (struct kvm_vcpu *vcpu); +void vgic_v3_put_nested (struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq (struct kvm_vcpu *vcpu); +u16 vgic_v3_get_eisr (struct kvm_vcpu *vcpu); +u16 vgic_v3_get_elrsr (struct kvm_vcpu *vcpu); +u64 vgic_v3_get_misr (struct kvm_vcpu *vcpu); +#define irqchip_in_kernel (k) (!!((k)->arch.vgic.in_kernel))#define vgic_initialized (k) ((k)->arch.vgic.initialized)#define vgic_ready (k) ((k)->arch.vgic.ready)
@@ -406,4 +422,6 @@ int vgic_v4_load(struct kvm_vcpu *vcpu);void vgic_v4_commit (struct kvm_vcpu *vcpu);int vgic_v4_put (struct kvm_vcpu *vcpu,bool need_db); +bool vgic_state_is_nested (struct kvm_vcpu *vcpu); +#endif /* __KVM_ARM_VGIC_H */
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 47/66 KVM: arm64: nv: Don't load the GICv4 context on entering a nested guest
In reply to Marc ZyngierWhen entering a nested guest (vgic_state_is_nested() == true), special care must be taken *not* to make the vPE resident, as these are interrupts targetting the L1 guest, and not any nested guest.
By not making the vPE resident, we guarantee that the delivery of an vLPI will result in a doorbell, forcing an exit from the nested guest and a switch to the L1 guest to handle the interrupt.
--- arch/arm64/kvm/vgic/vgic-v3.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/vgic/vgic-v3.c +8 -2
@@ -683,8 +683,8 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)if (vgic_state_is_nested (vcpu))vgic_v3_load_nested (vcpu); - -WARN_ON (vgic_v4_load (vcpu)); +else +WARN_ON (vgic_v4_load (vcpu)); }void vgic_v3_vmcr_sync (struct kvm_vcpu *vcpu)
@@ -702,6 +702,12 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) {struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; +/* +* vgic_v4_put will do nothing if we were not resident. This +* covers both the cases where we've blocked (we already have +* done a vgic_v4_put) and when running a nested guest (the +* vPE was never resident in order to generate a doorbell). +*/ WARN_ON (vgic_v4_put (vcpu, false));vgic_v3_vmcr_sync (vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 48/66 KVM: arm64: nv: vgic: Emulate the HW bit in software
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Should the guest hypervisor use the HW bit in the LRs, we need to emulate the deactivation from the L2 guest into the L1 distributor emulation, which is handled by L0.
It's all good fun.
--- arch/arm64/include/asm/kvm_hyp.h | 2 ++ arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/vgic/vgic-v3-nested.c | 32 ++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic.c | 6 ++++-- include/kvm/arm_vgic.h | 1 + 5 files changed, 40 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_hyp.h +2 -0
@@ -57,6 +57,8 @@ DECLARE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);int __vgic_v2_perform_cpuif_access (struct kvm_vcpu *vcpu); +u64 __gic_v3_get_lr (unsigned int lr); +void __vgic_v3_save_state (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_restore_state (struct vgic_v3_cpu_if *cpu_if);void __vgic_v3_activate_traps (struct vgic_v3_cpu_if *cpu_if);
arch/arm64/kvm/hyp/vgic-v3-sr.c +1 -1
@@ -18,7 +18,7 @@#define vtr_to_nr_pre_bits (v) ((((u32)(v) >> 26) & 7) + 1)#define vtr_to_nr_apr_regs (v) (1 << (vtr_to_nr_pre_bits(v) - 5)) -static u64 __gic_v3_get_lr (unsigned int lr) +u64 __gic_v3_get_lr (unsigned int lr) {switch (lr &0xf ) {case 0 :
arch/arm64/kvm/vgic/vgic-v3-nested.c +32 -0
@@ -138,6 +138,38 @@ static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) } } +void vgic_v3_sync_nested (struct kvm_vcpu *vcpu) +{ +struct vgic_v3_cpu_if *cpu_if =vcpu_nested_if (vcpu); +struct vgic_v3_cpu_if *s_cpu_if =vcpu_shadow_if (vcpu); +struct vgic_irq *irq; +int i; + +for (i =0 ; i < s_cpu_if->used_lrs; i++) { +u64 lr = cpu_if->vgic_lr[i]; +int l1_irq; + +if (!(lr &ICH_LR_HW ) || !(lr &ICH_LR_STATE )) +continue ; + +/* +* If we had a HW lr programmed by the guest hypervisor, we +* need to emulate the HW effect between the guest hypervisor +* and the nested guest. +*/ + l1_irq = (lr &ICH_LR_PHYS_ID_MASK ) >>ICH_LR_PHYS_ID_SHIFT ; + irq =vgic_get_irq (vcpu->kvm, vcpu, l1_irq); +if (!irq) +continue ;/* oh well, the guest hyp is broken */ + + lr =__gic_v3_get_lr (i); +if (!(lr &ICH_LR_STATE )) + irq->active = false; + +vgic_put_irq (vcpu->kvm, irq); + } +} +void vgic_v3_load_nested (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
arch/arm64/kvm/vgic/vgic.c +4 -2
@@ -876,9 +876,11 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) {int used_lrs; -/* If nesting, this is a load/put affair, not flush/sync. */ -if (vgic_state_is_nested (vcpu)) +/* If nesting, emulate the HW effect from L0 to L1 */ +if (vgic_state_is_nested (vcpu)) { +vgic_v3_sync_nested (vcpu);return ; + }/* An empty ap_list_head implies used_lrs == 0 */ if (list_empty (&vcpu->arch.vgic_cpu.ap_list_head))
include/kvm/arm_vgic.h +1 -0
@@ -371,6 +371,7 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu);void kvm_vgic_put (struct kvm_vcpu *vcpu);void kvm_vgic_vmcr_sync (struct kvm_vcpu *vcpu); +void vgic_v3_sync_nested (struct kvm_vcpu *vcpu);void vgic_v3_load_nested (struct kvm_vcpu *vcpu);void vgic_v3_put_nested (struct kvm_vcpu *vcpu);void vgic_v3_handle_nested_maint_irq (struct kvm_vcpu *vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 49/66 KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ
In reply to Marc ZyngierFrom: Andre Przywara <[email protected]>
The VGIC maintenance IRQ signals various conditions about the LRs, when the GIC's virtualization extension is used.
So far we didn't need it, but nested virtualization needs to know about this interrupt, so add a userland interface to setup the IRQ number. The architecture mandates that it must be a PPI, on top of that this code only exports a per-device option, so the PPI is the same on all VCPUs.
--- .../virt/kvm/devices/arm-vgic-v3.rst | 12 +++++++++- arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 +++++++++++++++++++ include/kvm/arm_vgic.h | 3 +++ tools/arch/arm/include/uapi/asm/kvm.h | 1 + 5 files changed, 38 insertions(+), 1 deletion(-)
Documentation/virt/kvm/devices/arm-vgic-v3.rst +11 -1
@@ -284,8 +284,18 @@ Groups: | Aff3 | Aff2 | Aff1 | Aff0 | Errors: - ======= ============================================= -EINVAL vINTID is not multiple of 32 or info field is not VGIC_LEVEL_INFO_LINE_LEVEL ======= ============================================= + + KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ + Attributes: + + The attr field of kvm_device_attr encodes the following values: + + bits: | 31 .... 5 | 4 .... 0 | + values: | RES0 | vINTID | + + The vINTID specifies which interrupt is generated when the vGIC + must generate a maintenance interrupt. This must be a PPI.
arch/arm64/include/uapi/asm/kvm.h +1 -0
@@ -335,6 +335,7 @@ struct kvm_vcpu_events {#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
arch/arm64/kvm/vgic/vgic-kvm-device.c +22 -0
@@ -241,6 +241,12 @@ static int vgic_get_common_attr(struct kvm_device *dev, VGIC_NR_PRIVATE_IRQS, uaddr);break ; } +case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ : { +u32 __user *uaddr = (u32 __user *)(long )attr->addr; + + r =put_user (dev->kvm->arch.vgic.maint_irq, uaddr); +break ; + } }return r;
@@ -627,6 +633,21 @@ static int vgic_v3_set_attr(struct kvm_device *dev, reg = tmp32;return vgic_v3_attr_regs_access (dev, attr, ®, true); } +case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ : { +u32 __user *uaddr = (u32 __user *)(long )attr->addr; +u32 val; + +if (get_user (val, uaddr)) +return -EFAULT ; + +/* Must be a PPI. */ +if ((val >=VGIC_NR_PRIVATE_IRQS ) || (val <VGIC_NR_SGIS )) +return -EINVAL ; + + dev->kvm->arch.vgic.maint_irq = val; + +return 0 ; + }case KVM_DEV_ARM_VGIC_GRP_CTRL : {int ret;
@@ -712,6 +733,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS :return vgic_v3_has_attr_regs (dev, attr);case KVM_DEV_ARM_VGIC_GRP_NR_IRQS : +case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ :return 0 ;case KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO : { if (((attr->attr & KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK) >>
include/kvm/arm_vgic.h +3 -0
@@ -219,6 +219,9 @@ struct vgic_dist {int nr_spis; +/* The GIC maintenance IRQ for nested hypervisors. */ +u32 maint_irq; +/* base addresses in guest physical address space: */ gpa_t vgic_dist_base;/* distributor */ union {
tools/arch/arm/include/uapi/asm/kvm.h +1 -0
@@ -246,6 +246,7 @@ struct kvm_vcpu_events {#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 50/66 KVM: arm64: nv: Implement maintenance interrupt forwarding
In reply to Marc ZyngierWhen we take a maintenance interrupt, we need to decide whether it is generated on an action from the guest, or if it is something that needs to be forwarded to the guest hypervisor.
--- arch/arm64/kvm/vgic/vgic-init.c | 30 ++++++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 25 +++++++++++++++++++---- 2 files changed, 51 insertions(+), 4 deletions(-)
arch/arm64/kvm/vgic/vgic-init.c +30 -0
@@ -6,10 +6,12 @@#include <linux/uaccess.h> #include <linux/interrupt.h> #include <linux/cpu.h> +#include <linux/irq.h> #include <linux/kvm_host.h> #include <kvm/arm_vgic.h> #include <asm/kvm_emulate.h> #include <asm/kvm_mmu.h> +#include <asm/kvm_nested.h> #include "vgic.h" /*
@@ -221,6 +223,16 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)if (!irqchip_in_kernel (vcpu->kvm))return 0 ; +if (nested_virt_in_use (vcpu)) { +/* FIXME: remove this hack */ +if (vcpu->kvm->arch.vgic.maint_irq ==0 ) + vcpu->kvm->arch.vgic.maint_irq = kvm_vgic_global_state.maint_irq; + ret =kvm_vgic_set_owner (vcpu, vcpu->kvm->arch.vgic.maint_irq, + vcpu); +if (ret) +return ret; + } +/* * If we are creating a VCPU with a GICv3 we must also register the * KVM io device for the redistributor that belongs to this VCPU.
@@ -462,12 +474,23 @@ static int vgic_init_cpu_dying(unsigned int cpu)static irqreturn_t vgic_maintenance_handler (int irq,void *data) { +struct kvm_vcpu *vcpu = *(struct kvm_vcpu **)data; +/* * We cannot rely on the vgic maintenance interrupt to be * delivered synchronously. This means we can only use it to * exit the VM, and we perform the handling of EOIed * interrupts on the exit path (see vgic_fold_lr_state). */ + +/* If not nested, deactivate */ +if (!vcpu || !vgic_state_is_nested (vcpu)) { +irq_set_irqchip_state (irq,IRQCHIP_STATE_ACTIVE , false); +return IRQ_HANDLED ; + } + +/* Assume nested from now */ +vgic_v3_handle_nested_maint_irq (vcpu);return IRQ_HANDLED ; }
@@ -538,6 +561,13 @@ int kvm_vgic_hyp_init(void)return ret; } + ret =irq_set_vcpu_affinity (kvm_vgic_global_state.maint_irq, +kvm_get_running_vcpus ()); +if (ret) { +kvm_err ("Error setting vcpu affinity\n" ); + goto out_free_irq; + } + ret =cpuhp_setup_state (CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING ,"kvm/arm/vgic:starting" , vgic_init_cpu_starting, vgic_init_cpu_dying);
arch/arm64/kvm/vgic/vgic-v3-nested.c +21 -4
@@ -173,10 +173,20 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu)void vgic_v3_load_nested (struct kvm_vcpu *vcpu) {struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; +struct vgic_irq *irq; +unsigned long flags; vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3;vgic_v3_create_shadow_lr (vcpu);__vgic_v3_restore_state (vcpu_shadow_if (vcpu)); + + irq =vgic_get_irq (vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); +raw_spin_lock_irqsave (&irq->irq_lock, flags); +if (irq->line_level || irq->active) +irq_set_irqchip_state (kvm_vgic_global_state.maint_irq, +IRQCHIP_STATE_ACTIVE , true); +raw_spin_unlock_irqrestore (&irq->irq_lock, flags); +vgic_put_irq (vcpu->kvm, irq); }void vgic_v3_put_nested (struct kvm_vcpu *vcpu)
@@ -191,11 +201,14 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu)*/ vgic_v3_fixup_shadow_lr_state (vcpu); vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; +irq_set_irqchip_state (kvm_vgic_global_state.maint_irq, +IRQCHIP_STATE_ACTIVE , false); }void vgic_v3_handle_nested_maint_irq (struct kvm_vcpu *vcpu) {struct vgic_v3_cpu_if *cpu_if =vcpu_nested_if (vcpu); +bool state;/* * If we exit a nested VM with a pending maintenance interrupt from the
@@ -203,8 +216,12 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu)* can re-sync the appropriate LRs and sample level triggered interrupts * again. */ -if (vgic_state_is_nested (vcpu) && - (cpu_if->vgic_hcr &ICH_HCR_EN ) && -vgic_v3_get_misr (vcpu)) -kvm_inject_nested_irq (vcpu); +if (!vgic_state_is_nested (vcpu)) +return ; + + state = cpu_if->vgic_hcr &ICH_HCR_EN ; + state &=vgic_v3_get_misr (vcpu); + +kvm_vgic_inject_irq (vcpu->kvm, vcpu->vcpu_id, + vcpu->kvm->arch.vgic.maint_irq, state, vcpu); }
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 51/66 KVM: arm64: nv: Add nested GICv3 tracepoints
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Adding tracepoints to be able to peek into the shadow LRs used when running a guest guest.
--- arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 13 ++- 2 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h
arch/arm64/kvm/vgic/vgic-nested-trace.h new file +137 -0
@@ -0,0 +1,137 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(_TRACE_VGIC_NESTED_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _TRACE_VGIC_NESTED_H + +#include <linux/tracepoint.h> + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM kvm + +#define SLR_ENTRY_VALS (x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT, \ + (__entry->orig_lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT (vgic_create_shadow_lrs, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs, u64 *orig_lrs), +TP_ARGS (vcpu, nr_lr, lrs, orig_lrs), + +TP_STRUCT__entry ( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) +__array ( u64, orig_lrs,16 ) + ), + +TP_fast_assign ( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + memcpy(__entry->orig_lrs, orig_lrs, 16 * sizeof(u64)); + ), + +TP_printk ("nr_lr: %d\n" +"%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" +"%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)" , + __entry->nr_lr, +SLR_ENTRY_VALS (0 ),SLR_ENTRY_VALS (1 ),SLR_ENTRY_VALS (2 ), +SLR_ENTRY_VALS (3 ),SLR_ENTRY_VALS (4 ),SLR_ENTRY_VALS (5 ), +SLR_ENTRY_VALS (6 ),SLR_ENTRY_VALS (7 ),SLR_ENTRY_VALS (8 ), +SLR_ENTRY_VALS (9 ),SLR_ENTRY_VALS (10 ),SLR_ENTRY_VALS (11 ), +SLR_ENTRY_VALS (12 ),SLR_ENTRY_VALS (13 ),SLR_ENTRY_VALS (14 ), +SLR_ENTRY_VALS (15 )) +); + +#define LR_ENTRY_VALS (x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT (vgic_put_nested, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs), +TP_ARGS (vcpu, nr_lr, lrs), + +TP_STRUCT__entry ( + __field( int, nr_lr ) +__array ( u64, lrs,16 ) + ), + +TP_fast_assign ( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + ), + +TP_printk ("nr_lr: %d\n" +"%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" +"%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu" , + __entry->nr_lr, +LR_ENTRY_VALS (0 ),LR_ENTRY_VALS (1 ),LR_ENTRY_VALS (2 ), +LR_ENTRY_VALS (3 ),LR_ENTRY_VALS (4 ),LR_ENTRY_VALS (5 ), +LR_ENTRY_VALS (6 ),LR_ENTRY_VALS (7 ),LR_ENTRY_VALS (8 ), +LR_ENTRY_VALS (9 ),LR_ENTRY_VALS (10 ),LR_ENTRY_VALS (11 ), +LR_ENTRY_VALS (12 ),LR_ENTRY_VALS (13 ),LR_ENTRY_VALS (14 ), +LR_ENTRY_VALS (15 )) +); + +TRACE_EVENT (vgic_nested_hw_emulate, + TP_PROTO(int lr, u64 lr_val, u32 l1_intid), +TP_ARGS (lr, lr_val, l1_intid), + +TP_STRUCT__entry ( + __field( int, lr ) + __field( u64, lr_val ) +__field ( u32, l1_intid ) + ), + +TP_fast_assign ( + __entry->lr = lr; + __entry->lr_val = lr_val; + __entry->l1_intid = l1_intid; + ), + +TP_printk ("lr: %d LR %llx L1 INTID: %u\n" , + __entry->lr, __entry->lr_val, __entry->l1_intid) +); + +#endif /* _TRACE_VGIC_NESTED_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH vgic/ +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE vgic-nested-trace + +/* This part must be outside protection */ +#include <trace/define_trace.h>
arch/arm64/kvm/vgic/vgic-v3-nested.c +12 -1
@@ -13,6 +13,9 @@#include "vgic.h" +#define CREATE_TRACE_POINTS +#include "vgic-nested-trace.h" +static inline struct vgic_v3_cpu_if *vcpu_nested_if (struct kvm_vcpu *vcpu) {return &vcpu->arch.vgic_cpu.nested_vgic_v3;
@@ -119,6 +122,9 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) used_lrs = i +1 ; } +trace_vgic_create_shadow_lrs (vcpu, kvm_vgic_global_state.nr_lr, + s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->used_lrs = used_lrs; }
@@ -163,8 +169,10 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu)continue ;/* oh well, the guest hyp is broken */ lr =__gic_v3_get_lr (i); -if (!(lr &ICH_LR_STATE )) +if (!(lr &ICH_LR_STATE )) { +trace_vgic_nested_hw_emulate (i, lr, l1_irq); irq->active = false; + }vgic_put_irq (vcpu->kvm, irq); }
@@ -195,6 +203,9 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu)__vgic_v3_save_state (vcpu_shadow_if (vcpu)); +trace_vgic_put_nested (vcpu, kvm_vgic_global_state.nr_lr, +vcpu_shadow_if (vcpu)->vgic_lr); +/* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one.
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 52/66 KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT
In reply to Marc ZyngierSince we're (almost) feature complete, let's allow userspace to request KVM_ARM_VCPU_NESTED_VIRT by bumping the KVM_VCPU_MAX_FEATURES up.
It's going to be great...
--- arch/arm64/include/asm/kvm_host.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -38,7 +38,7 @@#define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8#define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 53/66 KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation
In reply to Marc ZyngierSupport guest-provided information information to find out about the range of required invalidation.
--- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/nested.c | 57 +++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 79 +++++++++++++++++++---------- 3 files changed, 109 insertions(+), 28 deletions(-)
arch/arm64/include/asm/kvm_nested.h +1 -0
@@ -123,6 +123,7 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit);extern bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit);extern bool forward_nv_traps (struct kvm_vcpu *vcpu); +unsigned int ttl_to_size (u8 ttl);struct sys_reg_params ;struct sys_reg_desc ;
arch/arm64/kvm/nested.c +57 -0
63 lines not on this page · raw message
arch/arm64/kvm/sys_regs.c +51 -28
116 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 54/66 KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like information
In reply to Marc ZyngierIn order to be able to make S2 TLB invalidations more performant on NV, let's use a scheme derived from the ARMv8.4 TTL extension.
If bits [56:55] in the descriptor are non-zero, they indicate a level which can be used as an invalidation range.
--- arch/arm64/include/asm/kvm_nested.h | 4 ++ arch/arm64/kvm/nested.c | 98 +++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 13 ++-- 3 files changed, 110 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_nested.h +4 -0
@@ -123,6 +123,8 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit);extern bool forward_traps (struct kvm_vcpu *vcpu,u64 control_bit);extern bool forward_nv_traps (struct kvm_vcpu *vcpu); +u8 get_guest_mapping_ttl (struct kvm_vcpu *vcpu,struct kvm_s2_mmu *mmu, +u64 addr);unsigned int ttl_to_size (u8 ttl);struct sys_reg_params ;
@@ -131,4 +133,6 @@ struct sys_reg_desc;void access_nested_id_reg (struct kvm_vcpu *v,struct sys_reg_params *p,const struct sys_reg_desc *r); +#define KVM_NV_GUEST_MAP_SZ GENMASK_ULL(56, 55) +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +98 -0
116 lines not on this page · raw message
arch/arm64/kvm/sys_regs.c +8 -5
@@ -2584,10 +2584,13 @@ static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, u64 val) {unsigned long max_size; -u8 ttl =0 ; +u8 ttl; -if (cpus_have_const_cap (ARM64_HAS_ARMv8_4_TTL)) { - ttl =FIELD_GET (GENMASK_ULL (47 ,44 ), val); + ttl =FIELD_GET (GENMASK_ULL (47 ,44 ), val); + +if (!(cpus_have_const_cap (ARM64_HAS_ARMv8_4_TTL) && ttl)) { +u64 addr = (val &GENMASK_ULL (35 ,0 )) <<12 ; + ttl =get_guest_mapping_ttl (vcpu, mmu, addr); } max_size =ttl_to_size (ttl);
@@ -2628,6 +2631,8 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p,u64 base_addr;unsigned long max_size; +spin_lock (&vcpu->kvm->mmu_lock); +/* * We drop a number of things from the supplied value: *
@@ -2639,8 +2644,6 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p,*/ base_addr = (p->regval &GENMASK_ULL (35 ,0 )) <<12 ; -spin_lock (&vcpu->kvm->mmu_lock); - mmu =lookup_s2_mmu (vcpu->kvm, vttbr,HCR_VM );if (mmu) { max_size =compute_tlb_inval_range (vcpu, mmu, p->regval);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 55/66 KVM: arm64: Allow populating S2 SW bits
In reply to Marc ZyngierThe S2 page table code doesn't use the SW bits yet, but we are about to need them to encode some guest Stage-2 information (its mapping size in the form of the TTL encoding).
Propagate the SW bits specified by the caller, and store them into the corresponding entry.
--- arch/arm64/include/asm/kvm_pgtable.h | 10 ++++++++++ arch/arm64/kvm/hyp/pgtable.c | 6 ++++++ 2 files changed, 16 insertions(+)
arch/arm64/include/asm/kvm_pgtable.h +10 -0
@@ -35,6 +35,10 @@ struct kvm_pgtable {* @KVM_PGTABLE_PROT_W: Write permission. * @KVM_PGTABLE_PROT_R: Read permission. * @KVM_PGTABLE_PROT_DEVICE: Device attributes. +* @KVM_PGTABLE_PROT_S2_SW0: SW bit 0. +* @KVM_PGTABLE_PROT_S2_SW1: SW bit 1. +* @KVM_PGTABLE_PROT_S2_SW2: SW bit 2. +* @KVM_PGTABLE_PROT_S2_SW3: SW bit 3. */ enum kvm_pgtable_prot {KVM_PGTABLE_PROT_X =BIT (0 ),
@@ -42,6 +46,12 @@ enum kvm_pgtable_prot {KVM_PGTABLE_PROT_R =BIT (2 ),KVM_PGTABLE_PROT_DEVICE =BIT (3 ), + +/* Cunningly, this matches the PTE bits... */ +KVM_PGTABLE_PROT_S2_SW0 =BIT (55 ), +KVM_PGTABLE_PROT_S2_SW1 =BIT (56 ), +KVM_PGTABLE_PROT_S2_SW2 =BIT (57 ), + KVM_PGTABLE_PROT_S2_SW3 = BIT(58), };#define PAGE_HYP (KVM_PGTABLE_PROT_R | KVM_PGTABLE_PROT_W)
arch/arm64/kvm/hyp/pgtable.c +6 -0
@@ -44,6 +44,7 @@#define KVM_PTE_LEAF_ATTR_HI_S1_XN BIT(54)#define KVM_PTE_LEAF_ATTR_HI_S2_XN BIT(54) +#define KVM_PTE_LEAF_ATTR_HI_S2_SW GENMASK(58, 55)struct kvm_pgtable_walk_data {struct kvm_pgtable *pgt;
@@ -457,6 +458,7 @@ static int stage2_map_set_prot_attr(enum kvm_pgtable_prot prot, attr |=FIELD_PREP (KVM_PTE_LEAF_ATTR_LO_S2_SH , sh); attr |=KVM_PTE_LEAF_ATTR_LO_S2_AF ; + attr |= prot &KVM_PTE_LEAF_ATTR_HI_S2_SW ; data->attr = attr;return 0 ; }
@@ -805,6 +807,10 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,if (prot &KVM_PGTABLE_PROT_X ) clr |=KVM_PTE_LEAF_ATTR_HI_S2_XN ; +/* Always propagate the SW bits */ + clr |=FIELD_PREP (KVM_PTE_LEAF_ATTR_HI_S2_SW ,0xf ); + set |= prot &KVM_PTE_LEAF_ATTR_HI_S2_SW ; + ret =stage2_update_leaf_attrs (pgt, addr,1 , set, clr,NULL , &level);if (!ret)kvm_call_hyp (__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 56/66 KVM: arm64: nv: Tag shadow S2 entries with nested level
In reply to Marc ZyngierPopulate bits [56:55] of the leaf entry with the level provided by the guest's S2 translation.
--- arch/arm64/include/asm/kvm_nested.h | 8 ++++++++ arch/arm64/kvm/mmu.c | 6 ++++++ 2 files changed, 14 insertions(+)
arch/arm64/include/asm/kvm_nested.h +8 -0
@@ -5,6 +5,8 @@#include <linux/bitfield.h> #include <linux/kvm_host.h> +#include <asm/kvm_pgtable.h> +static inline bool nested_virt_in_use (const struct kvm_vcpu *vcpu) { return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
@@ -135,4 +137,10 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,#define KVM_NV_GUEST_MAP_SZ GENMASK_ULL(56, 55) +static inline u64 kvm_encode_nested_level (struct kvm_s2_trans *trans) +{ +return FIELD_PREP (KVM_PGTABLE_PROT_S2_SW1 |KVM_PGTABLE_PROT_S2_SW0 , + trans->level); +} +#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/mmu.c +6 -0
@@ -920,11 +920,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,* Potentially reduce shadow S2 permissions to match the guest's own * S2. For exec faults, we'd only reach this point if the guest * actually allowed it (see kvm_s2_handle_perm_fault). +* +* Also encode the level of the nested translation in the SW bits of +* the PTE/PMD/PUD. This will be retrived on TLB invalidation from +* the guest. */ if (kvm_is_shadow_s2_fault (vcpu)) { writable &=kvm_s2_trans_writable (nested);if (!kvm_s2_trans_readable (nested)) prot &= ~KVM_PGTABLE_PROT_R ; + + prot |=kvm_encode_nested_level (nested); }spin_lock (&kvm->mmu_lock);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 57/66 KVM: arm64: nv: Add include containing the VNCR_EL2 offsets
In reply to Marc ZyngierVNCR_EL2 points to a page containing a number of system registers accessed by a guest hypervisor when ARMv8.4-NV is enabled.
Let's document the offsets in that page, as we are going to use this layout.
--- arch/arm64/include/asm/vncr_mapping.h | 73 +++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 arch/arm64/include/asm/vncr_mapping.h
arch/arm64/include/asm/vncr_mapping.h new file +73 -0
@@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* System register offsets in the VNCR page +* All offsets are *byte* displacements! +*/ + +#ifndef __ARM64_VNCR_MAPPING_H__ +#define __ARM64_VNCR_MAPPING_H__ + +#define VNCR_VTTBR_EL2 0x020 +#define VNCR_VTCR_EL2 0x040 +#define VNCR_VMPIDR_EL2 0x050 +#define VNCR_CNTVOFF_EL2 0x060 +#define VNCR_HCR_EL2 0x078 +#define VNCR_HSTR_EL2 0x080 +#define VNCR_VPIDR_EL2 0x088 +#define VNCR_TPIDR_EL2 0x090 +#define VNCR_VNCR_EL2 0x0B0 +#define VNCR_CPACR_EL1 0x100 +#define VNCR_CONTEXTIDR_EL1 0x108 +#define VNCR_SCTLR_EL1 0x110 +#define VNCR_ACTLR_EL1 0x118 +#define VNCR_TCR_EL1 0x120 +#define VNCR_AFSR0_EL1 0x128 +#define VNCR_AFSR1_EL1 0x130 +#define VNCR_ESR_EL1 0x138 +#define VNCR_MAIR_EL1 0x140 +#define VNCR_AMAIR_EL1 0x148 +#define VNCR_MDSCR_EL1 0x158 +#define VNCR_SPSR_EL1 0x160 +#define VNCR_CNTV_CVAL_EL0 0x168 +#define VNCR_CNTV_CTL_EL0 0x170 +#define VNCR_CNTP_CVAL_EL0 0x178 +#define VNCR_CNTP_CTL_EL0 0x180 +#define VNCR_SCXTNUM_EL1 0x188 +#define VNCR_ZCR_EL1 0x1E0 +#define VNCR_TTBR0_EL1 0x200 +#define VNCR_TTBR1_EL1 0x210 +#define VNCR_FAR_EL1 0x220 +#define VNCR_ELR_EL1 0x230 +#define VNCR_SP_EL1 0x240 +#define VNCR_VBAR_EL1 0x250 +#define VNCR_ICH_LR0_EL2 0x400 +// VNCR_ICH_LRN_EL2(n) VNCR_ICH_LR0_EL2+8*((n) & 7) +#define VNCR_ICH_AP0R0_EL2 0x480 +// VNCR_ICH_AP0RN_EL2(n) VNCR_ICH_AP0R0_EL2+8*((n) & 3) +#define VNCR_ICH_AP1R0_EL2 0x4A0 +// VNCR_ICH_AP1RN_EL2(n) VNCR_ICH_AP1R0_EL2+8*((n) & 3) +#define VNCR_ICH_HCR_EL2 0x4C0 +#define VNCR_ICH_VMCR_EL2 0x4C8 +#define VNCR_VDISR_EL2 0x500 +#define VNCR_PMBLIMITR_EL1 0x800 +#define VNCR_PMBPTR_EL1 0x810 +#define VNCR_PMBSR_EL1 0x820 +#define VNCR_PMSCR_EL1 0x828 +#define VNCR_PMSEVFR_EL1 0x830 +#define VNCR_PMSICR_EL1 0x838 +#define VNCR_PMSIRR_EL1 0x840 +#define VNCR_PMSLATFR_EL1 0x848 +#define VNCR_TRFCR_EL1 0x880 +#define VNCR_MPAM1_EL1 0x900 +#define VNCR_MPAMHCR_EL2 0x930 +#define VNCR_MPAMVPMV_EL2 0x938 +#define VNCR_MPAMVPM0_EL2 0x940 +#define VNCR_MPAMVPM1_EL2 0x948 +#define VNCR_MPAMVPM2_EL2 0x950 +#define VNCR_MPAMVPM3_EL2 0x958 +#define VNCR_MPAMVPM4_EL2 0x960 +#define VNCR_MPAMVPM5_EL2 0x968 +#define VNCR_MPAMVPM6_EL2 0x970 +#define VNCR_MPAMVPM7_EL2 0x978 + +#endif /* __ARM64_VNCR_MAPPING_H__ */
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 58/66 KVM: arm64: Map VNCR-capable registers to a separate page
In reply to Marc ZyngierWith ARMv8.4-NV, registers that can be directly accessed in memory by the guest have to live at architected offsets in a special page.
Let's annotate the sysreg enum to reflect the offset at which they are in this page, whith a little twist:
If running on HW that doesn't have the ARMv8.4-NV feature, or even a VM that doesn't use NV, we store all the system registers in the usual sys_regs array. The only difference with the pre-8.4 situation is that VNCR-capable registers are at a "similar" offset as in the VNCR page (we can compute the actual offset at compile time), and that the sys_regs array is both bigger and sparse.
--- arch/arm64/include/asm/kvm_host.h | 97 ++++++++++++++++++++----------- 1 file changed, 63 insertions(+), 34 deletions(-)
arch/arm64/include/asm/kvm_host.h +63 -34
@@ -26,6 +26,7 @@#include <asm/kvm.h> #include <asm/kvm_asm.h> #include <asm/thread_info.h> +#include <asm/vncr_mapping.h> #define __KVM_HAVE_ARCH_INTC_INITIALIZED
@@ -171,31 +172,32 @@ struct kvm_vcpu_fault_info {u64 disr_el1;/* Deferred [SError] Status Register */ }; +/* +* VNCR() just places the VNCR_capable registers in the enum after +* __VNCR_START__, and the value (after correction) to be an 8-byte offset +* from the VNCR base. As we don't require the enum to be otherwise ordered, +* we need the terrible hack below to ensure that we correctly size the +* sys_regs array, no matter what. +* +* The __MAX__ macro has been lifted from Sean Eron Anderson's wonderful +* treasure trove of bit hacks: +* https://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax +*/ +#define __MAX__ (x,y) ((x) ^ (((x) ^ (y)) & -((x) < (y)))) +#define VNCR (r) \ + __before_##r, \ + r = __VNCR_START__ + ((VNCR_ ## r) / 8), \ + __after_##r = __MAX__(__before_##r - 1, r) +enum vcpu_sysreg { __INVALID_SYSREG__,/* 0 is reserved as an invalid value */ MPIDR_EL1 ,/* MultiProcessor Affinity Register */ CSSELR_EL1 ,/* Cache Size Selection Register */ -SCTLR_EL1 ,/* System Control Register */ -ACTLR_EL1 ,/* Auxiliary Control Register */ -CPACR_EL1 ,/* Coprocessor Access Control */ -ZCR_EL1 ,/* SVE Control */ -TTBR0_EL1 ,/* Translation Table Base Register 0 */ -TTBR1_EL1 ,/* Translation Table Base Register 1 */ -TCR_EL1 ,/* Translation Control Register */ -ESR_EL1 ,/* Exception Syndrome Register */ -AFSR0_EL1 ,/* Auxiliary Fault Status Register 0 */ -AFSR1_EL1 ,/* Auxiliary Fault Status Register 1 */ -FAR_EL1 ,/* Fault Address Register */ -MAIR_EL1 ,/* Memory Attribute Indirection Register */ -VBAR_EL1 ,/* Vector Base Address Register */ -CONTEXTIDR_EL1 ,/* Context ID Register */ TPIDR_EL0 ,/* Thread ID, User R/W */ TPIDRRO_EL0 ,/* Thread ID, User R/O */ TPIDR_EL1 ,/* Thread ID, Privileged */ -AMAIR_EL1 ,/* Aux Memory Attribute Indirection Register */ CNTKCTL_EL1 ,/* Timer Control Register (EL1) */ PAR_EL1 ,/* Physical Address Register */ -MDSCR_EL1 ,/* Monitor Debug System Control Register */ MDCCINT_EL1 ,/* Monitor Debug Comms Channel Interrupt Enable Reg */ DISR_EL1 ,/* Deferred Interrupt Status Register */
@@ -226,16 +228,6 @@ enum vcpu_sysreg {APGAKEYLO_EL1 ,APGAKEYHI_EL1 , -ELR_EL1 , -SP_EL1 , -SPSR_EL1 , - -CNTVOFF_EL2 , -CNTV_CVAL_EL0 , -CNTV_CTL_EL0 , -CNTP_CVAL_EL0 , -CNTP_CTL_EL0 , -/* 32bit specific registers. */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */
@@ -243,20 +235,14 @@ enum vcpu_sysreg {DBGVCR32_EL2 ,/* Debug Vector Catch Register */ /* EL2 registers */ -VPIDR_EL2 ,/* Virtualization Processor ID Register */ -VMPIDR_EL2 ,/* Virtualization Multiprocessor ID Register */ SCTLR_EL2 ,/* System Control Register (EL2) */ ACTLR_EL2 ,/* Auxiliary Control Register (EL2) */ -HCR_EL2 ,/* Hypervisor Configuration Register */ MDCR_EL2 ,/* Monitor Debug Configuration Register (EL2) */ CPTR_EL2 ,/* Architectural Feature Trap Register (EL2) */ -HSTR_EL2 ,/* Hypervisor System Trap Register */ HACR_EL2 ,/* Hypervisor Auxiliary Control Register */ TTBR0_EL2 ,/* Translation Table Base Register 0 (EL2) */ TTBR1_EL2 ,/* Translation Table Base Register 1 (EL2) */ TCR_EL2 ,/* Translation Control Register (EL2) */ -VTTBR_EL2 ,/* Virtualization Translation Table Base Register */ -VTCR_EL2 ,/* Virtualization Translation Control Register */ SPSR_EL2 ,/* EL2 saved program status register */ ELR_EL2 ,/* EL2 exception link register */ AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */
@@ -270,7 +256,6 @@ enum vcpu_sysreg {RVBAR_EL2 ,/* Reset Vector Base Address Register */ RMR_EL2 ,/* Reset Management Register */ CONTEXTIDR_EL2 ,/* Context ID Register (EL2) */ -TPIDR_EL2 ,/* EL2 Software Thread ID Register */ CNTHCTL_EL2 ,/* Counter-timer Hypervisor Control register */ SP_EL2 ,/* EL2 Stack Pointer */ CNTHP_CTL_EL2,
@@ -278,6 +263,41 @@ enum vcpu_sysreg {CNTHV_CTL_EL2 ,CNTHV_CVAL_EL2 , + __VNCR_START__,/* Any VNCR-capable reg goes after this point */ + +VNCR (SCTLR_EL1 ),/* System Control Register */ +VNCR (ACTLR_EL1 ),/* Auxiliary Control Register */ +VNCR (CPACR_EL1 ),/* Coprocessor Access Control */ +VNCR (ZCR_EL1 ),/* SVE Control */ +VNCR (TTBR0_EL1 ),/* Translation Table Base Register 0 */ +VNCR (TTBR1_EL1 ),/* Translation Table Base Register 1 */ +VNCR (TCR_EL1 ),/* Translation Control Register */ +VNCR (ESR_EL1 ),/* Exception Syndrome Register */ +VNCR (AFSR0_EL1 ),/* Auxiliary Fault Status Register 0 */ +VNCR (AFSR1_EL1 ),/* Auxiliary Fault Status Register 1 */ +VNCR (FAR_EL1 ),/* Fault Address Register */ +VNCR (MAIR_EL1 ),/* Memory Attribute Indirection Register */ +VNCR (VBAR_EL1 ),/* Vector Base Address Register */ +VNCR (CONTEXTIDR_EL1 ),/* Context ID Register */ +VNCR (AMAIR_EL1 ),/* Aux Memory Attribute Indirection Register */ +VNCR (MDSCR_EL1 ),/* Monitor Debug System Control Register */ +VNCR (ELR_EL1 ), +VNCR (SP_EL1 ), +VNCR (SPSR_EL1 ), +VNCR (VPIDR_EL2 ),/* Virtualization Processor ID Register */ +VNCR (VMPIDR_EL2 ),/* Virtualization Multiprocessor ID Register */ +VNCR (HCR_EL2 ),/* Hypervisor Configuration Register */ +VNCR (HSTR_EL2 ),/* Hypervisor System Trap Register */ +VNCR (VTTBR_EL2 ),/* Virtualization Translation Table Base Register */ +VNCR (VTCR_EL2 ),/* Virtualization Translation Control Register */ +VNCR (TPIDR_EL2 ),/* EL2 Software Thread ID Register */ + +VNCR (CNTVOFF_EL2 ), +VNCR (CNTV_CVAL_EL0 ), +VNCR (CNTV_CTL_EL0 ), +VNCR (CNTP_CVAL_EL0 ), +VNCR (CNTP_CTL_EL0 ), +NR_SYS_REGS /* Nothing after this line! */ };
@@ -294,6 +314,9 @@ struct kvm_cpu_context {u64 sys_regs[NR_SYS_REGS ];struct kvm_vcpu *__hyp_running_vcpu; + +/* This pointer has to be 4kB aligned. */ +u64 *vncr_array; };struct kvm_pmu_events {
@@ -489,7 +512,13 @@ struct kvm_vcpu_arch {* for system registers that are never context switched, but only * emulated. */ -#define __ctxt_sys_reg (c,r) (&(c)->sys_regs[(r)]) +static inline u64 *__ctxt_sys_reg (const struct kvm_cpu_context *ctxt,int r) +{ +if (unlikely (r >= __VNCR_START__ && ctxt->vncr_array)) +return &ctxt->vncr_array[r - __VNCR_START__]; + +return (u64 *)&ctxt->sys_regs[r]; +}#define ctxt_sys_reg (c,r) (*__ctxt_sys_reg(c,r))
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 59/66 KVM: arm64: nv: Move nested vgic state into the sysreg file
In reply to Marc ZyngierThe vgic nested state needs to be accessible from the VNCR page, and thus needs to be part of the normal sysreg file. Let's move it there.
--- arch/arm64/include/asm/kvm_host.h | 9 +++ arch/arm64/kvm/sys_regs.c | 53 +++++++++++------ arch/arm64/kvm/vgic/vgic-v3-nested.c | 88 ++++++++++++++-------------- arch/arm64/kvm/vgic/vgic-v3.c | 17 ++++-- arch/arm64/kvm/vgic/vgic.h | 10 ++++ include/kvm/arm_vgic.h | 7 --- 6 files changed, 110 insertions(+), 74 deletions(-)
arch/arm64/include/asm/kvm_host.h +9 -0
@@ -298,6 +298,15 @@ enum vcpu_sysreg {VNCR (CNTP_CVAL_EL0 ),VNCR (CNTP_CTL_EL0 ), +VNCR (ICH_LR0_EL2 ), +ICH_LR15_EL2 =ICH_LR0_EL2 +15 , +VNCR (ICH_AP0R0_EL2 ), +ICH_AP0R3_EL2 =ICH_AP0R0_EL2 +3 , +VNCR (ICH_AP1R0_EL2 ), +ICH_AP1R3_EL2 =ICH_AP1R0_EL2 +3 , +VNCR (ICH_HCR_EL2 ), +VNCR (ICH_VMCR_EL2 ), +NR_SYS_REGS /* Nothing after this line! */ };
arch/arm64/kvm/sys_regs.c +35 -18
93 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic-v3-nested.c +45 -43
190 lines not on this page · raw message
arch/arm64/kvm/vgic/vgic-v3.c +11 -6
@@ -292,10 +292,11 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) ICC_SRE_EL1_SRE);/* * If nesting is allowed, force GICv3 onto the nested -* guests as well. +* guests as well by setting the shadow state to the +* same value. */ if (nested_virt_in_use (vcpu)) - vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; + vcpu->arch.vgic_cpu.shadow_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser =INITIAL_PENDBASER_VALUE ; }else { vgic_v3->vgic_sre =0 ;
@@ -662,11 +663,15 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;/* -* vgic_v3_load_nested only affects the LRs in the shadow -* state, so it is fine to pass the nested state around. +* If the vgic is in nested state, populate the shadow state +* from the guest's nested state. As vgic_v3_load_nested() +* will only load LRs, let's deal with the rest of the state +* here as if it was a non-nested state. Cunning. */ -if (vgic_state_is_nested (vcpu)) - cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; +if (vgic_state_is_nested (vcpu)) { +vgic_v3_create_shadow_state (vcpu); + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + }/* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen
arch/arm64/kvm/vgic/vgic.h +10 -0
@@ -318,4 +318,14 @@ int vgic_v4_init(struct kvm *kvm);void vgic_v4_teardown (struct kvm *kvm);void vgic_v4_configure_vsgis (struct kvm *kvm); +void vgic_v3_sync_nested (struct kvm_vcpu *vcpu); +void vgic_v3_create_shadow_state (struct kvm_vcpu *vcpu); +void vgic_v3_load_nested (struct kvm_vcpu *vcpu); +void vgic_v3_put_nested (struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq (struct kvm_vcpu *vcpu); + +#define ICH_LRN (n) (ICH_LR0_EL2 + (n)) +#define ICH_AP0RN (n) (ICH_AP0R0_EL2 + (n)) +#define ICH_AP1RN (n) (ICH_AP1R0_EL2 + (n)) +#endif
include/kvm/arm_vgic.h +0 -7
@@ -309,9 +309,6 @@ struct vgic_cpu {struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS ]; -/* CPU vif control registers for the virtual GICH interface */ -struct vgic_v3_cpu_if nested_vgic_v3; -/* * The shadow vif control register loaded to the hardware when * running a nested L2 guest with the virtual IMO/FMO bit set.
@@ -374,10 +371,6 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu);void kvm_vgic_put (struct kvm_vcpu *vcpu);void kvm_vgic_vmcr_sync (struct kvm_vcpu *vcpu); -void vgic_v3_sync_nested (struct kvm_vcpu *vcpu); -void vgic_v3_load_nested (struct kvm_vcpu *vcpu); -void vgic_v3_put_nested (struct kvm_vcpu *vcpu); -void vgic_v3_handle_nested_maint_irq (struct kvm_vcpu *vcpu);u16 vgic_v3_get_eisr (struct kvm_vcpu *vcpu);u16 vgic_v3_get_elrsr (struct kvm_vcpu *vcpu);u64 vgic_v3_get_misr (struct kvm_vcpu *vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 60/66 KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature
In reply to Marc ZyngierAdd the detection code for the ARMv8.4-NV feature.
--- arch/arm64/include/asm/cpucaps.h | 1 + arch/arm64/include/asm/kvm_nested.h | 6 ++++++ arch/arm64/kernel/cpufeature.c | 10 ++++++++++ 3 files changed, 17 insertions(+)
arch/arm64/include/asm/cpucaps.h +1 -0
@@ -17,6 +17,7 @@#define ARM64_WORKAROUND_834220 7#define ARM64_HAS_NO_HW_PREFETCH 8#define ARM64_HAS_NESTED_VIRT 9 +#define ARM64_HAS_ENHANCED_NESTED_VIRT 10#define ARM64_HAS_VIRT_HOST_EXTN 11#define ARM64_WORKAROUND_CAVIUM_27456 12#define ARM64_HAS_32BIT_EL0 13
arch/arm64/include/asm/kvm_nested.h +6 -0
@@ -14,6 +14,12 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)); } +static inline bool enhanced_nested_virt_in_use (const struct kvm_vcpu *vcpu) +{ +return cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT ) && +nested_virt_in_use (vcpu); +} +/* Translation helpers from non-VHE EL2 to EL1 */ static inline u64 tcr_el2_ips_to_tcr_el1_ps (u64 tcr_el2) {
arch/arm64/kernel/cpufeature.c +10 -0
@@ -1815,6 +1815,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .field_pos = ID_AA64MMFR2_NV_SHIFT, .min_field_value = 1, }, + { + .desc = "Enhanced Nested Virtualization Support", + .capability = ARM64_HAS_ENHANCED_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_NV_SHIFT, + .min_field_value = 2, + }, #endif /* CONFIG_ARM64_VHE */ { .desc = "32-bit EL0 Support",
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 61/66 KVM: arm64: nv: Synchronize PSTATE early on exit
In reply to Marc ZyngierThe NV code relies on predicates such as is_hyp_ctxt() being reliable. In turn, is_hyp_ctxt() relies on things like PSTATE and the virtual HCR_EL2 being accurate.
But with ARMv8.4-NV removing trapping for a large part of the EL2 system registers (among which HCR_EL2), we can't use such trapping to synchronize the rest of the state.
Let's look at the following sequence for a VHE guest:
(1) enter guest in host EL0
(2) guest traps to guest vEL2 (no hypervisor intervention)
(3) guest clears virtual HCR_EL2.TGE (no trap either)
(4) host interrupt fires, exit
(5) is_hyp_ctxt() now says "guest" (PSTATE.M==EL1 and TGE==0)
It is obvious that such behaviour would be rather unfortunate, and lead to interesting, difficult to catch bugs specially if preemption kicks in (yes, I wasted a whole week chasing this one).
In order to preserve the invariant that a guest entered in host context must exit in the same context, we must make sure that is_hyp_ctxt() works correctly. Since we can always observe the guest value of HCR_EL2.{E2H,TGE} in the VNCR_EL2 page, we solely need to synchronize PSTATE as early as possible.
This basically amounts to moving from_hw_pstate() as close as possible to the guest exit point, and fixup_guest_exit() seems as good a place as any.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 16 ++++-- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 26 ++-------- arch/arm64/kvm/hyp/nvhe/switch.c | 8 ++- arch/arm64/kvm/hyp/vhe/switch.c | 57 +++++++++++++++++++++- 4 files changed, 78 insertions(+), 29 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +12 -4
@@ -407,11 +407,11 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu) }/* -* Return true when we were able to fixup the guest exit and should return to -* the guest, false when we should restore the host state and return to the -* main run loop. +* Prologue for the guest fixup, populating ESR_EL2 and fixing up PC +* if required. */ -static inline bool fixup_guest_exit (struct kvm_vcpu *vcpu,u64 *exit_code) +static inline void fixup_guest_exit_prologue (struct kvm_vcpu *vcpu, +u64 *exit_code) {if (ARM_EXCEPTION_CODE (*exit_code) !=ARM_EXCEPTION_IRQ ) vcpu->arch.fault.esr_el2 =read_sysreg_el2 (SYS_ESR );
@@ -430,7 +430,15 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)if (esr_ec == ESR_ELx_EC_HVC32 || esr_ec == ESR_ELx_EC_HVC64)write_sysreg_el2 (read_sysreg_el2 (SYS_ELR ) -4 ,SYS_ELR ); } +} +/* +* Return true when we were able to fixup the guest exit and should return to +* the guest, false when we should restore the host state and return to the +* main run loop. +*/ +static inline bool fixup_guest_exit (struct kvm_vcpu *vcpu,u64 *exit_code) +{/* * We're using the raw exception code in order to only process * the trap if no SError is pending. We will come back to the
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +3 -23
@@ -51,32 +51,12 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)ctxt_sys_reg (ctxt,SPSR_EL1 ) =read_sysreg_el1 (SYS_SPSR ); } -static inline u64 from_hw_pstate (const struct kvm_cpu_context *ctxt) -{ -u64 reg =read_sysreg_el2 (SYS_SPSR ); - -if (__is_hyp_ctxt (ctxt)) { -u64 mode = reg & (PSR_MODE_MASK |PSR_MODE32_BIT ); - -switch (mode) { -case PSR_MODE_EL1t: - mode = PSR_MODE_EL2t; -break ; -case PSR_MODE_EL1h: - mode = PSR_MODE_EL2h; -break ; - } - -return (reg & ~(PSR_MODE_MASK |PSR_MODE32_BIT )) | mode; - } - -return reg; -} -static inline void __sysreg_save_el2_return_state (struct kvm_cpu_context *ctxt) { +/* On VHE, PSTATE is saved in fixup_guest_exit_vhe() */ +if (!has_vhe ()) + ctxt->regs.pstate =read_sysreg_el2 (SYS_SPSR ); ctxt->regs.pc =read_sysreg_el2 (SYS_ELR ); - ctxt->regs.pstate =from_hw_pstate (ctxt);if (cpus_have_final_cap (ARM64_HAS_RAS_EXTN ))ctxt_sys_reg (ctxt,DISR_EL1 ) =read_sysreg_s (SYS_VDISR_EL2 );
arch/arm64/kvm/hyp/nvhe/switch.c +7 -1
@@ -166,6 +166,12 @@ static void __pmu_switch_to_host(struct kvm_cpu_context *host_ctxt)write_sysreg (pmu->events_host, pmcntenset_el0); } +static bool fixup_guest_exit_nvhe (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +fixup_guest_exit_prologue (vcpu, exit_code); +return fixup_guest_exit (vcpu, exit_code); +} +/* Switch to the guest for legacy non-VHE systems */ int __kvm_vcpu_run (struct kvm_vcpu *vcpu) {
@@ -219,7 +225,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) exit_code =__guest_enter (vcpu);/* And we're baaack! */ - }while (fixup_guest_exit (vcpu, &exit_code)); + }while (fixup_guest_exit_nvhe (vcpu, &exit_code));__sysreg_save_state_nvhe (guest_ctxt);__sysreg32_save_state (vcpu);
arch/arm64/kvm/hyp/vhe/switch.c +56 -1
80 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 62/66 KVM: arm64: nv: Sync nested timer state with ARMv8.4
In reply to Marc ZyngierFrom: Christoffer Dall <[email protected]>
Emulating the ARMv8.4-NV timers is a bit odd, as the timers can be reconfigured behind our back without the hypervisor even noticing. In the VHE case, that's an actual regression in the architecture...
--- arch/arm64/kvm/arch_timer.c | 37 ++++++++++++++++++++++++++++++++++++ arch/arm64/kvm/arm.c | 3 +++ include/kvm/arm_arch_timer.h | 1 + 3 files changed, 41 insertions(+)
arch/arm64/kvm/arch_timer.c +37 -0
@@ -778,6 +778,43 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu)set_cntvoff (0 ); } +void kvm_timer_sync_nested (struct kvm_vcpu *vcpu) +{ +if (!is_hyp_ctxt (vcpu)) +return ; + +/* +* Guest hypervisors using ARMv8.4 enhanced nested virt support have +* their EL1 timer register accesses redirected to the VNCR page. +*/ +if (!vcpu_el2_e2h_is_set (vcpu)) { +/* +* For a non-VHE guest hypervisor, we update the hardware +* timer registers with the latest value written by the guest +* to the VNCR page and let the hardware take care of the +* rest. +*/ +write_sysreg_el0 (__vcpu_sys_reg (vcpu,CNTV_CTL_EL0 ),SYS_CNTV_CTL ); +write_sysreg_el0 (__vcpu_sys_reg (vcpu,CNTV_CVAL_EL0 ),SYS_CNTV_CVAL ); +write_sysreg_el0 (__vcpu_sys_reg (vcpu,CNTP_CTL_EL0 ),SYS_CNTP_CTL ); +write_sysreg_el0 (__vcpu_sys_reg (vcpu,CNTP_CVAL_EL0 ),SYS_CNTP_CVAL ); + }else { +/* +* For a VHE guest hypervisor, the emulated state (which +* is stored in the VNCR page) could have been updated behind +* our back, and we must reset the emulation of the timers. +*/ + +struct timer_map map; +get_timer_map (vcpu, &map); + +soft_timer_cancel (&map.emul_vtimer->hrtimer); +soft_timer_cancel (&map.emul_ptimer->hrtimer); +timer_emulate (map.emul_vtimer); +timer_emulate (map.emul_ptimer); + } +} +/* * With a userspace irqchip we have to check if the guest de-asserted the * timer and if so, unmask the timer irq signal on the host interrupt
arch/arm64/kvm/arm.c +3 -0
@@ -849,6 +849,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)if (static_branch_unlikely (&userspace_irqchip_in_use))kvm_timer_sync_user (vcpu); +if (enhanced_nested_virt_in_use (vcpu)) +kvm_timer_sync_nested (vcpu); +kvm_arch_vcpu_ctxsync_fp (vcpu);/*
include/kvm/arm_arch_timer.h +1 -0
@@ -68,6 +68,7 @@ int kvm_timer_hyp_init(bool);int kvm_timer_enable (struct kvm_vcpu *vcpu);int kvm_timer_vcpu_reset (struct kvm_vcpu *vcpu);void kvm_timer_vcpu_init (struct kvm_vcpu *vcpu); +void kvm_timer_sync_nested (struct kvm_vcpu *vcpu);void kvm_timer_sync_user (struct kvm_vcpu *vcpu);bool kvm_timer_should_notify_user (struct kvm_vcpu *vcpu);void kvm_timer_update_run (struct kvm_vcpu *vcpu);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 63/66 KVM: arm64: nv: Allocate VNCR page when required
In reply to Marc ZyngierIf running a NV guest on an ARMv8.4-NV capable system, let's allocate an additional page that will be used by the hypervisor to fulfill system register accesses.
--- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/nested.c | 8 ++++++++ arch/arm64/kvm/reset.c | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +2 -1
@@ -523,7 +523,8 @@ struct kvm_vcpu_arch {*/ static inline u64 *__ctxt_sys_reg (const struct kvm_cpu_context *ctxt,int r) { -if (unlikely (r >= __VNCR_START__ && ctxt->vncr_array)) +if (unlikely (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT ) && + r >= __VNCR_START__ && ctxt->vncr_array))return &ctxt->vncr_array[r - __VNCR_START__];return (u64 *)&ctxt->sys_regs[r];
arch/arm64/kvm/nested.c +8 -0
@@ -47,6 +47,12 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)if (!cpus_have_final_cap (ARM64_HAS_NESTED_VIRT ))return -EINVAL ; +if (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT )) { + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page (GFP_KERNEL | __GFP_ZERO); +if (!vcpu->arch.ctxt.vncr_array) +return -ENOMEM ; + } +mutex_lock (&kvm->lock);/*
@@ -64,6 +70,8 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) { kvm_free_stage2_pgd(&tmp[num_mmus - 1]); kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); + vcpu->arch.ctxt.vncr_array = NULL; } else { kvm->arch.nested_mmus_size = num_mmus; ret =0 ;
arch/arm64/kvm/reset.c +1 -0
@@ -150,6 +150,7 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu)void kvm_arm_vcpu_destroy (struct kvm_vcpu *vcpu) {kfree (vcpu->arch.sve_state); +free_page ((unsigned long )vcpu->arch.ctxt.vncr_array); }static void kvm_vcpu_reset_sve (struct kvm_vcpu *vcpu)
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 63/66] KVM: arm64: nv: Allocate VNCR page when required
Quoted 31 lines · Marc Zyngier
If running a NV guest on an ARMv8.4-NV capable system, let's allocate an additional page that will be used by the hypervisor to fulfill system register accesses.
Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/nested.c | 8 ++++++++ arch/arm64/kvm/reset.c | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)arch/arm64/include/asm/kvm_host.h +2
-1@@ -523,7 +523,8 @@ struct kvm_vcpu_arch {*/ static inline u64 *__ctxt_sys_reg (const struct kvm_cpu_context *ctxt,int r) { -if (unlikely (r >= __VNCR_START__ && ctxt->vncr_array)) +if (unlikely (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT ) && + r >= __VNCR_START__ && ctxt->vncr_array))return &ctxt->vncr_array[r - __VNCR_START__];return (u64 *)&ctxt->sys_regs[r];arch/arm64/kvm/nested.c +0
-0
arch/arm64/kvm/nested.c int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) · line 55
Quoted 9 lines
arch/arm64/kvm/nested.c +6
-0@@ -47,6 +47,12 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)if (!cpus_have_final_cap (ARM64_HAS_NESTED_VIRT ))return -EINVAL ; +if (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT )) { + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page (GFP_KERNEL | __GFP_ZERO); +if (!vcpu->arch.ctxt.vncr_array) +return -ENOMEM ; + } +
If KVM_ARM_VCPU_INIT was called multiple times, the above codes would try to allocate a new page
without free-ing the previous one. Besides that, the following kvm_free_stage2_pgd() call would fail in the
second call with the error message "kvm_arch already initialized?". I think a possible fix is to add a new flag to indicate whether the NV related meta data have been initialized,
and only initialize them for the first call.
Quoted 27 lines
arch/arm64/kvm/nested.c +2
-0mutex_lock (&kvm->lock);/* @@ -64,6 +70,8 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) {kvm_free_stage2_pgd (&tmp[num_mmus -1 ]);kvm_free_stage2_pgd (&tmp[num_mmus -2 ]); +free_page ((unsigned long )vcpu->arch.ctxt.vncr_array); + vcpu->arch.ctxt.vncr_array =NULL ; } else { kvm->arch.nested_mmus_size = num_mmus; ret =0 ;arch/arm64/kvm/reset.c +1
-1@@ -150,6 +150,7 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu)void kvm_arm_vcpu_destroy (struct kvm_vcpu *vcpu) {kfree (vcpu->arch.sve_state); +free_page ((unsigned long )vcpu->arch.ctxt.vncr_array); }static void kvm_vcpu_reset_sve (struct kvm_vcpu *vcpu) --2.29.2
Quoted 4 lines
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 63/66] KVM: arm64: nv: Allocate VNCR page when required
Quoted 36 lines · Haibo Xu
Quoted 34 lines · Marc Zyngier
If running a NV guest on an ARMv8.4-NV capable system, let's allocate an additional page that will be used by the hypervisor to fulfill system register accesses.
Signed-off-by Marc Zyngier <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/nested.c | 8 ++++++++ arch/arm64/kvm/reset.c | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)arch/arm64/include/asm/kvm_host.h +2
-1@@ -523,7 +523,8 @@ struct kvm_vcpu_arch {*/ static inline u64 *__ctxt_sys_reg (const struct kvm_cpu_context *ctxt,int r) { -if (unlikely (r >= __VNCR_START__ && ctxt->vncr_array)) +if (unlikely (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT ) && + r >= __VNCR_START__ && ctxt->vncr_array))return &ctxt->vncr_array[r - __VNCR_START__];return (u64 *)&ctxt->sys_regs[r];arch/arm64/kvm/nested.c +0
-0@@ -47,6 +47,12 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)if (!cpus_have_final_cap (ARM64_HAS_NESTED_VIRT ))return -EINVAL ;
Quoted 10 lines
arch/arm64/kvm/nested.c int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) · line 55
Quoted 6 lines
arch/arm64/kvm/nested.c +6
-0+if (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT )) { + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page (GFP_KERNEL | __GFP_ZERO); +if (!vcpu->arch.ctxt.vncr_array) +return -ENOMEM ; + } +If
KVM_ARM_VCPU_INITwas called multiple times, the above codes would try to allocate a new page without free-ing the previous one. Besides that, the followingkvm_free_stage2_pgd()call would
I assume you mean kvm_init_stage2_mmu() here.
Quoted 4 lines
fail in the second call with the error message "
kvm_archalready initialized?". I think a possible fix is to add a new flag to indicate whether the NV related meta data have been initialized, and only initialize them for the first call.
Good catch. But I think we have all the data we need at this stage to avoid this issue:
arch/arm64/kvm/mmu.c +13 -0
@@ -390,7 +390,20 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)int cpu, err;struct kvm_pgtable *pgt; +/* +* If we already have our page tables in place, and that the +* MMU context is the canonical one, we have a bug somewhere, +* as this is only supposed to ever happen once per VM. +* +* Otherwise, we're building nested page tables, and that's +* probably because userspace called KVM_ARM_VCPU_INIT more +* than once on the same vcpu. Since that's actually legal, +* don't kick a fuss and leave gracefully. +*/ if (mmu->pgt !=NULL ) { +if (&kvm->arch.mmu != mmu) +return 0 ; +kvm_err ("kvm_arch already initialized?\n" );return -EINVAL ; }
arch/arm64/kvm/nested.c +3 -1
@@ -48,7 +48,9 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)return -EINVAL ;if (cpus_have_final_cap (ARM64_HAS_ENHANCED_NESTED_VIRT )) { - vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page (GFP_KERNEL | __GFP_ZERO); +if (!vcpu->arch.ctxt.vncr_array) + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page (GFP_KERNEL | __GFP_ZERO); +if (!vcpu->arch.ctxt.vncr_array)return -ENOMEM ; }
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 64/66 KVM: arm64: nv: Enable ARMv8.4-NV support
In reply to Marc ZyngierAs all the VNCR-capable system registers are nicely separated from the rest of the crowd, let's set HCR_EL2.NV2 on and let the ball rolling.
--- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_emulate.h | 23 +++++++++++++---------- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/hyp/vhe/switch.c | 14 +++++++++++++- 4 files changed, 28 insertions(+), 11 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
@@ -14,6 +14,7 @@/* Hyp Configuration Register (HCR) bits */ #define HCR_ATA (UL(1) << 56)#define HCR_FWB (UL(1) << 46) +#define HCR_NV2 (UL(1) << 45)#define HCR_AT (UL(1) << 44)#define HCR_NV1 (UL(1) << 43)#define HCR_NV (UL(1) << 42)
arch/arm64/include/asm/kvm_emulate.h +13 -10
@@ -242,21 +242,24 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)static inline u64 __fixup_spsr_el2_write (struct kvm_cpu_context *ctxt,u64 val) { -if (!__vcpu_el2_e2h_is_set (ctxt)) { -/* -* Clear the .M field when writing SPSR to the CPU, so that we -* can detect when the CPU clobbered our SPSR copy during a -* local exception. -*/ - val &= ~0xc ; - } + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + +if (enhanced_nested_virt_in_use (vcpu) ||__vcpu_el2_e2h_is_set (ctxt)) +return val; -return val; +/* +* Clear the .M field when writing SPSR to the CPU, so that we +* can detect when the CPU clobbered our SPSR copy during a +* local exception. +*/ +return val &= ~0xc ; }static inline u64 __fixup_spsr_el2_read (const struct kvm_cpu_context *ctxt,u64 val) { -if (__vcpu_el2_e2h_is_set (ctxt)) + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + +if (enhanced_nested_virt_in_use (vcpu) ||__vcpu_el2_e2h_is_set (ctxt))return val;/*
arch/arm64/include/asm/sysreg.h +1 -0
@@ -487,6 +487,7 @@#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2)#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0)#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0)#define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0)
arch/arm64/kvm/hyp/vhe/switch.c +13 -1
@@ -48,7 +48,13 @@ static void __activate_traps(struct kvm_vcpu *vcpu)* the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |=HCR_TVM |HCR_TRVM |HCR_AT |HCR_TTLB |HCR_NV1 ; +if (enhanced_nested_virt_in_use (vcpu)) { + hcr &= ~HCR_TVM ; + }else { + hcr |=HCR_TVM |HCR_TRVM |HCR_TTLB ; + } + + hcr |=HCR_AT |HCR_NV1 ; }else {/* * For a guest hypervisor on v8.1 (VHE), allow to
@@ -80,6 +86,12 @@ static void __activate_traps(struct kvm_vcpu *vcpu)if (!vcpu_el2_tge_is_set (vcpu)) hcr |=HCR_AT |HCR_TTLB ; } + +if (enhanced_nested_virt_in_use (vcpu)) { + hcr |=HCR_AT |HCR_TTLB |HCR_NV2 ; +write_sysreg_s (vcpu->arch.ctxt.vncr_array, +SYS_VNCR_EL2 ); + } }else if (nested_virt_in_use (vcpu)) { hcr |=__vcpu_sys_reg (vcpu,HCR_EL2 ); }
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 65/66 KVM: arm64: nv: Fast-track 'InHost' exception returns
In reply to Marc ZyngierA significant part of the ARMv8.3-NV extension is to trap ERET instructions so that the hypervisor gets a chance to switch from a vEL2 L1 guest to an EL1 L2 guest.
But this also has the unfortunate consequence of trapping ERET in unsuspecting circumstances, such as staying at vEL2 (interrupt handling while being in the guest hypervisor), or returning to host userspace in the case of a VHE guest.
Although we already make some effort to handle these ERET quicker by not doing the put/load dance, it is still way too far down the line for it to be efficient enough.
For these cases, it would ideal to ERET directly, no question asked. Of course, we can't do that. But the next best thing is to do it as early as possible, in fixup_guest_exit(), much as we would handle FPSIMD exceptions.
--- arch/arm64/kvm/emulate-nested.c | 26 ++-------------- arch/arm64/kvm/hyp/vhe/switch.c | 53 ++++++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 25 deletions(-)
arch/arm64/kvm/emulate-nested.c +2 -24
@@ -52,8 +52,7 @@ bool forward_nv_traps(struct kvm_vcpu *vcpu)void kvm_emulate_nested_eret (struct kvm_vcpu *vcpu) { -u64 spsr, elr, mode; -bool direct_eret; +u64 spsr, elr;/* * Forward this trap to the virtual EL2 if the virtual
@@ -62,31 +61,10 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)if (forward_nv_traps (vcpu))return ; -/* -* Going through the whole put/load motions is a waste of time -* if this is a VHE guest hypervisor returning to its own -* userspace, or the hypervisor performing a local exception -* return. No need to save/restore registers, no need to -* switch S2 MMU. Just do the canonical ERET. -*/ - spsr =vcpu_read_sys_reg (vcpu,SPSR_EL2 ); - mode = spsr & (PSR_MODE_MASK |PSR_MODE32_BIT ); - - direct_eret = (mode == PSR_MODE_EL0t && -vcpu_el2_e2h_is_set (vcpu) && -vcpu_el2_tge_is_set (vcpu)); - direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); - -if (direct_eret) { - *vcpu_pc (vcpu) =vcpu_read_sys_reg (vcpu,ELR_EL2 ); - *vcpu_cpsr (vcpu) = spsr; -trace_kvm_nested_eret (vcpu, *vcpu_pc (vcpu), spsr); -return ; - } -preempt_disable ();kvm_arch_vcpu_put (vcpu); + spsr =__vcpu_sys_reg (vcpu,SPSR_EL2 ); elr =__vcpu_sys_reg (vcpu,ELR_EL2 );trace_kvm_nested_eret (vcpu, elr, spsr);
arch/arm64/kvm/hyp/vhe/switch.c +52 -1
65 lines not on this page · raw message
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerPATCH v3 66/66 KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests
In reply to Marc ZyngierDue to the way ARMv8.4-NV suppresses traps when accessing EL2 system registers, we can't track when the guest changes its HCR_EL2.TGE setting. This means we always trap EL1 TLBIs, even if they don't affect any guest.
This obviously has a huge impact on performance, as we handle TLBI traps as a normal exit, and a normal VHE host issues thousands of TLBIs when booting (and quite a few when running userspace).
A cheap way to reduce the overhead is to handle the limited case of {E2H,TGE}=={1,1} as a guest fixup, as we already have the right mmu configuration in place. Just execute the decoded instruction right away and return to the guest.
--- arch/arm64/kvm/hyp/vhe/switch.c | 36 +++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/vhe/tlb.c | 6 ++++-- arch/arm64/kvm/sys_regs.c | 25 ++++++++--------------- 3 files changed, 48 insertions(+), 19 deletions(-)
arch/arm64/kvm/hyp/vhe/switch.c +36 -0
@@ -168,6 +168,39 @@ void deactivate_traps_vhe_put(void)__deactivate_traps_common (); } +static bool __hyp_handle_tlbi_el1 (struct kvm_vcpu *vcpu) +{ +u32 instr; +u64 val; + +/* +* Ideally, we would never trap on EL1 TLB invalidations when the +* guest's HCR_EL2.{E2H,TGE} == {1,1}. But "thanks" to ARMv8.4, we +* don't trap writes to HCR_EL2, meaning that we can't track +* changes to the virtual TGE bit. So we leave HCR_EL2.TTLB set on +* the host. Oopsie... +* +* In order to speed-up EL1 TLBIs from the vEL2 guest when TGE is +* set, try and handle these invalidation as quickly as possible, +* without fully exiting (unless this needs forwarding). +*/ +if (kvm_vcpu_trap_get_class (vcpu) != ESR_ELx_EC_SYS64 || + !vcpu_mode_el2 (vcpu) || + (__vcpu_sys_reg (vcpu,HCR_EL2 ) & (HCR_E2H |HCR_TGE )) != (HCR_E2H |HCR_TGE )) +return false; + + instr =esr_sys64_to_sysreg (kvm_vcpu_get_esr (vcpu)); +if (sys_reg_Op0 (instr) != TLBI_Op0 || +sys_reg_Op1 (instr) != TLBI_Op1_EL1) +return false; + + val =vcpu_get_reg (vcpu,kvm_vcpu_sys_get_rt (vcpu)); +__kvm_tlb_el1_instr (NULL , val, instr); +__kvm_skip_instr (vcpu); + +return true; +} +static bool __hyp_handle_eret (struct kvm_vcpu *vcpu) {struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt;
@@ -261,6 +294,9 @@ static bool fixup_guest_exit_vhe(struct kvm_vcpu *vcpu, u64 *exit_code,if (*exit_code ==ARM_EXCEPTION_TRAP ) {if (__hyp_handle_eret (vcpu))return true; + +if (__hyp_handle_tlbi_el1 (vcpu)) +return true; }return fixup_guest_exit (vcpu, exit_code);
arch/arm64/kvm/hyp/vhe/tlb.c +4 -2
@@ -200,7 +200,8 @@ void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding)dsb (ishst);/* Switch to requested VMID */ -__tlb_switch_to_guest (mmu, &cxt); +if (mmu) +__tlb_switch_to_guest (mmu, &cxt);/* * Execute the same instruction as the guest hypervisor did,
@@ -239,5 +240,6 @@ void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding)dsb (ish);isb (); -__tlb_switch_to_host (&cxt); +if (mmu) +__tlb_switch_to_host (&cxt); }
arch/arm64/kvm/sys_regs.c +8 -17
@@ -2684,6 +2684,8 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) {u32 sys_encoding =sys_insn (p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); +u64 virtual_vttbr =vcpu_read_sys_reg (vcpu,VTTBR_EL2 ); +struct kvm_s2_mmu *mmu;/* * If we're here, this is because we've trapped on a EL1 TLBI
@@ -2702,24 +2704,13 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p,mutex_lock (&vcpu->kvm->lock); -if ((__vcpu_sys_reg (vcpu,HCR_EL2 ) & (HCR_E2H |HCR_TGE )) != (HCR_E2H |HCR_TGE )) { -u64 virtual_vttbr =vcpu_read_sys_reg (vcpu,VTTBR_EL2 ); -struct kvm_s2_mmu *mmu; - - mmu =lookup_s2_mmu (vcpu->kvm, virtual_vttbr,HCR_VM ); -if (mmu) -__kvm_tlb_el1_instr (mmu, p->regval, sys_encoding); + mmu =lookup_s2_mmu (vcpu->kvm, virtual_vttbr,HCR_VM ); +if (mmu) +__kvm_tlb_el1_instr (mmu, p->regval, sys_encoding); - mmu =lookup_s2_mmu (vcpu->kvm, virtual_vttbr,0 ); -if (mmu) -__kvm_tlb_el1_instr (mmu, p->regval, sys_encoding); - }else { -/* -* ARMv8.4-NV allows the guest to change TGE behind -* our back, so we always trap EL1 TLBIs from vEL2... -*/ -__kvm_tlb_el1_instr (&vcpu->kvm->arch.mmu, p->regval, sys_encoding); - } + mmu =lookup_s2_mmu (vcpu->kvm, virtual_vttbr,0 ); +if (mmu) +__kvm_tlb_el1_instr (mmu, p->regval, sys_encoding);mutex_unlock (&vcpu->kvm->lock);
Signature · 6 lines
2.29.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 26 lines · Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
Quoted 6 lines
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
Hi Marc,
I got a kernel BUG message when booting the L2 guest kernel with the kvmtool on a FVP setup.
Could you help have a look about the BUG message as well as my environment configuration?
I think It probably caused by some local configurations of the FVP setup.
Thanks,
Haibo
------------------------------------------------------------------------------------------------------------------------------
1 # lkvm run -k ./Image -m 320 -c 2 --name guest-229
2 [ 77.714334] kernel BUG at arch/arm64/kernel/traps.c:407!
3 [ 77.715758] Internal error: Oops - BUG: 0 [#1] SMP
4 [ 77.716900] Modules linked in:
5 [ 77.717754] CPU: 0 PID: 229 Comm: lkvm Not tainted 5.11.0-rc1+ #2
6 [ 77.719193] Hardware name: linux,dummy-virt (DT)
7 [ 77.720300] pstate: 00400009 (nzcv daif +PAN -UAO -TCO BTYPE=--)
8 [ 77.722006] pc : do_undefinstr+0x4d0/0x5b0
9 [ 77.722900] lr : do_undefinstr+0x380/0x5b0
10 [ 77.723800] sp : ffff8000122ab8c0
11 [ 77.724674] x29: ffff8000122ab8c0 x28: ffff0000048c49c0
12 [ 77.725824] x27: 0000000000000000 x26: 0000000000000000
13 [ 77.727050] x25: 0000000000000000 x24: ffff000004b9d520
14 [ 77.728217] x23: 0000000080400009 x22: ffff8000100e42d0
15 [ 77.729700] x21: ffff8000122abaa0 x20: ffff0000048c49c0
16 [ 77.730856] x19: ffff8000122ab950 x18: 0000000000000000
17 [ 77.732047] x17: 0000000000000000 x16: 0000000000000000
18 [ 77.733276] x15: 0000000000000000 x14: 0000000000000000
19 [ 77.734423] x13: 0000000000000000 x12: 0000000000000000
20 [ 77.735629] x11: 0000000000000000 x10: 0000000000000000
21 [ 77.737200] x9 : ffff8000102fb0a0 x8 : ffff000004b9d730
22 [ 77.738369] x7 : 0000000000000000 x6 : 0000000080000000
23 [ 77.739533] x5 : 0000000000000000 x4 : 000000000000001f
24 [ 77.740700] x3 : 0000000000000000 x2 : ffff800011a05b80
25 [ 77.741963] x1 : ffff800011c09110 x0 : 0000000080400009
26 [ 77.743142] Call trace:
27 [ 77.743700] do_undefinstr+0x4d0/0x5b0
28 [ 77.744662] el1_undef+0x2c/0x48
29 [ 77.745832] el1_sync_handler+0x80/0xb0
30 [ 77.746736] el1_sync+0x74/0x100
31 [ 77.747612] reset_pmcr+0x8/0x88
32 [ 77.748471] kvm_reset_vcpu+0x128/0x290
33 [ 77.749425] kvm_arch_vcpu_ioctl+0x698/0x6c8
34 [ 77.750561] kvm_vcpu_ioctl+0x3c8/0x5f0
35 [ 77.751472] __arm64_sys_ioctl+0xa8/0xe8
36 [ 77.752375] el0_svc_common.constprop.0+0x78/0x188
37 [ 77.753920] do_el0_svc+0x28/0x88
38 [ 77.754821] el0_svc+0x1c/0x28
39 [ 77.755723] el0_sync_handler+0xa8/0xb0
40 [ 77.756678] el0_sync+0x160/0x180
41 [ 77.757582] Code: d2801400 17ffffa9 a9025bf5 f9001bf7 (d4210000)
42 [ 77.758970] ---[ end trace a8cdeac7ff43b5a5 ]---
43 [ 77.763012] ------------[ cut here ]------------
44 [ 77.769730] WARNING: CPU: 0 PID: 0 at kernel/rcu/tree.c:632
rcu_eqs_enter.isra.0+0x68/0x70
45 [ 77.771552] Modules linked in:
46 [ 77.772420] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D
5.11.0-rc1+ #2
47 [ 77.774155] Hardware name: linux,dummy-virt (DT)
48 [ 77.775301] pstate: 204003c9 (nzCv DAIF +PAN -UAO -TCO BTYPE=--)
49 [ 77.776809] pc : rcu_eqs_enter.isra.0+0x68/0x70
50 [ 77.777999] lr : rcu_idle_enter+0x14/0x20
51 [ 77.778905] sp : ffff8000119f3e80
52 [ 77.779759] x29: ffff8000119f3e80 x28: ffff00001fefca00
53 [ 77.785541] x27: ffff800011a02ec0 x26: 0000000000000000
54 [ 77.786700] x25: 0000000000000000 x24: ffff8000119f9528
55 [ 77.787915] x23: ffff800011a02ec0 x22: ffff8000115837b8
56 [ 77.789166] x21: ffff8000119f9500 x20: 0000000000000000
57 [ 77.790345] x19: ffff80001156e000 x18: 0000000000000010
58 [ 77.791560] x17: 0000000000000000 x16: 0000000000000000
59 [ 77.792775] x15: 0000000000000000 x14: 0000000000000000
60 [ 77.793978] x13: 0000000000000002 x12: 0000000000000000
61 [ 77.795153] x11: 0000000000000001 x10: 0000000000000ae0
62 [ 77.805799] x9 : ffff800010db2d94 x8 : ffff800011a03a00
63 [ 77.813399] x7 : 0000000000000000 x6 : 000000166ce4e201
64 [ 77.821100] x5 : 00ffffffffffffff x4 : ffff80000e93e000
65 [ 77.828958] x3 : 4000000000000002 x2 : 4000000000000000
66 [ 77.836659] x1 : ffff800011585ac0 x0 : ffff00001fec3ac0
67 [ 77.844360] Call trace:
68 [ 77.847910] rcu_eqs_enter.isra.0+0x68/0x70
69 [ 77.851519] rcu_idle_enter+0x14/0x20
70 [ 77.854989] default_idle_call+0x3c/0x16c
71 [ 77.858460] do_idle+0x214/0x260
72 [ 77.862200] cpu_startup_entry+0x2c/0x90
73 [ 77.866046] rest_init+0xc4/0xd0
74 [ 77.869654] arch_call_rest_init+0x14/0x1c
75 [ 77.873334] start_kernel+0x80c/0x844
76 [ 77.877065] ---[ end trace a8cdeac7ff43b5a6 ]---
---------------------------------------------------------------------------------------------------------------Quoted 144 lines
And to be clear: although Jintack and Christoffer have written tons of the stuff originaly, I'm the one responsible for breaking it!
[1] https://lore.kernel.org/r/[email protected]
[2]git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.gitkvm-arm64/nv-5.11.-WIP
[3]git://linux-arm.org/kvmtool.gitnv/nv-wip-5.2-rc5Andre Przywara (1):
KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQChristoffer Dall (15):
KVM: arm64: nv: Introduce nested virtualization VCPU feature
KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
KVM: arm64: nv: Allow userspace to setPSR_MODE_EL2x
KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
KVM: arm64: nv: ResetVMPIDR_EL2andVPIDR_EL2to sane values
KVM: arm64: nv: Handle trapped ERET from virtual EL2
KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
KVM: arm64: nv: Only toggle cache for virtual EL2 whenSCTLR_EL2
changes
KVM: arm64: nv: Implement nested Stage-2 page table walk logic
KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
KVM: arm64: nv:arch_timer: Support hyp timer emulation
KVM: arm64: nv: vgic: Emulate the HW bit in software
KVM: arm64: nv: Add nested GICv3 tracepoints
KVM: arm64: nv: Sync nested timer state with ARMv8.4Jintack Lim (19):
arm64: AddARM64_HAS_NESTED_VIRTcpufeature
KVM: arm64: nv: Handle HCR_EL2.NV system register traps
KVM: arm64: nv: Support virtual EL2 exceptions
KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
KVM: arm64: nv: TrapSPSR_EL1,ELR_EL1andVBAR_EL1from virtual EL2
KVM: arm64: nv: TrapCPACR_EL1access in virtual EL2
KVM: arm64: nv: Handle PSCI call via smc from the guest
KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
KVM: arm64: nv: ConfigureHCR_EL2for nested virtualization
KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
KVM: arm64: nv: Set a handler for the system instruction traps
KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
KVM: arm64: nv: Nested GICv3 SupportMarc Zyngier (31):
KVM: arm64: nv: Add EL2 system registers to vcpu context
KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
KVM: arm64: nv: Handle virtual EL2 registers in
vcpu_read/write_sys_reg()
KVM: arm64: nv: HandleSPSR_EL2specially
KVM: arm64: nv: Handle HCR_EL2.E2H specially
KVM: arm64: nv: Save/Restore vEL2 sysregs
KVM: arm64: nv: Forward debug traps to the nested guest
KVM: arm64: nv: Filter out unsupported features from ID regs
KVM: arm64: nv: Hide RAS from nested guests
KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
KVM: arm64: nv: Handle shadow stage 2 page faults
KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
KVM: arm64: nv: Fold guest'sHCR_EL2configuration into the host's
KVM: arm64: nv: Add handling of EL2-specific timer registers
KVM: arm64: nv: Load timer before the GIC
KVM: arm64: nv: Don't load the GICv4 context on entering a nested
guest
KVM: arm64: nv: Implement maintenance interrupt forwarding
KVM: arm64: nv: Allow userspace to requestKVM_ARM_VCPU_NESTED_VIRT
KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation
KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like
information
KVM: arm64: Allow populating S2 SW bits
KVM: arm64: nv: Tag shadow S2 entries with nested level
KVM: arm64: nv: Add include containing theVNCR_EL2offsets
KVM: arm64: Map VNCR-capable registers to a separate page
KVM: arm64: nv: Move nested vgic state into the sysreg file
KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature
KVM: arm64: nv: Synchronize PSTATE early on exit
KVM: arm64: nv: Allocate VNCR page when required
KVM: arm64: nv: Enable ARMv8.4-NV support
KVM: arm64: nv: Fast-track 'InHost' exception returns
KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests.../admin-guide/kernel-parameters.txt | 4 + .../virt/kvm/devices/arm-vgic-v3.rst | 12 +- arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/esr.h | 6 + arch/arm64/include/asm/kvm_arm.h | 28 +- arch/arm64/include/asm/kvm_asm.h | 4 + arch/arm64/include/asm/kvm_emulate.h | 145 +- arch/arm64/include/asm/kvm_host.h | 175 ++- arch/arm64/include/asm/kvm_hyp.h | 2 + arch/arm64/include/asm/kvm_mmu.h | 17 +- arch/arm64/include/asm/kvm_nested.h | 152 ++ arch/arm64/include/asm/kvm_pgtable.h | 10 + arch/arm64/include/asm/sysreg.h | 104 +- arch/arm64/include/asm/vncr_mapping.h | 73 + arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kernel/cpufeature.c | 35 + arch/arm64/kvm/Makefile | 4 +- arch/arm64/kvm/arch_timer.c | 189 ++- arch/arm64/kvm/arm.c | 34 +- arch/arm64/kvm/at.c | 231 ++++ arch/arm64/kvm/emulate-nested.c | 186 +++ arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/handle_exit.c | 81 +- arch/arm64/kvm/hyp/exception.c | 44 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 31 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 28 +- arch/arm64/kvm/hyp/nvhe/switch.c | 10 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 6 + arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 207 ++- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 +- arch/arm64/kvm/hyp/vhe/tlb.c | 83 ++ arch/arm64/kvm/inject_fault.c | 62 +- arch/arm64/kvm/mmu.c | 183 ++- arch/arm64/kvm/nested.c | 908 ++++++++++++ arch/arm64/kvm/reset.c | 14 +- arch/arm64/kvm/sys_regs.c | 1226 ++++++++++++++++- arch/arm64/kvm/sys_regs.h | 6 + arch/arm64/kvm/trace_arm.h | 65 +- arch/arm64/kvm/vgic/vgic-init.c | 30 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 + arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 240 ++++ arch/arm64/kvm/vgic/vgic-v3.c | 39 +- arch/arm64/kvm/vgic/vgic.c | 44 + arch/arm64/kvm/vgic/vgic.h | 10 + include/kvm/arm_arch_timer.h | 7 + include/kvm/arm_vgic.h | 16 + tools/arch/arm/include/uapi/asm/kvm.h | 1 + 50 files changed, 4890 insertions(+), 160 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/vncr_mapping.h create mode 100644 arch/arm64/kvm/at.c create mode 100644 arch/arm64/kvm/emulate-nested.c create mode 100644 arch/arm64/kvm/nested.c create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c
Quoted 7 lines
--
2.29.2_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
1 #!/bin/bash
2
3 PLATFORM=Base_RevC-2xAEMv8A
4 #PLATFORM=Base_AEMv8A
5 #FVP_PATH=/opt/arm/developmentstudio_platinum-2020.a/bin/ 6 FVP_PATH=/home/haibo/Downloads/FVP_6196/
7 FVP_GCCV=Linux64_GCC-6.4
8 FW_PATH=/home/haibo/nv_test
9
10 MODEL=${FVP_PATH}/FVP_${PLATFORM}
11 DTB_IMG=${FW_PATH}/fvp-base-aemv8a-aemv8a.dtb 12 BL1_IMG=${FW_PATH}/bl1.bin
13 FIP_IMG=${FW_PATH}/fip.bin
14 KERNEL=${FW_PATH}/Image-5.11
15 INITRD=${FW_PATH}/ramdisk.img
16 DISK_IMG0=${FW_PATH}/rootfs.img
17
18 CLUSTER0_NUM_CORES=${CLUSTER0_NUM_CORES:-1}
19 CLUSTER1_NUM_CORES=${CLUSTER1_NUM_CORES:-1}
20 SECURE_MEMORY=${SECURE_MEMORY:-0}
21 CACHE_STATE_MODELLED=${CACHE_STATE_MODELLED:=0} 22
23 KERN_ADDR=0x80080000
24 DTB_ADDR=0x82000000
25 INITRD_ADDR=0x84000000
26
27 ${MODEL} \
28 -C pctl.startup=0.0.0.0 \
29 -C bp.secure_memory=$SECURE_MEMORY \
30 -C cache_state_modelled=$CACHE_STATE_MODELLED \
31 -C bp.pl011_uart0.untimed_fifos=1 \
32 -C bp.pl011_uart0.unbuffered_output=1 \
33 -C cluster0.NUM_CORES=$CLUSTER0_NUM_CORES \
34 -C cluster0.has_nested_virtualization=1 \
35 -C cluster0.has_arm_v8-1=1 \
36 -C cluster0.has_arm_v8-2=1 \
37 -C cluster0.has_arm_v8-3=1 \
38 -C cluster0.has_arm_v8-4=1 \
39 -C cluster1.NUM_CORES=$CLUSTER1_NUM_CORES \
40 -C cluster1.has_nested_virtualization=1 \
41 -C cluster1.has_arm_v8-1=1 \
42 -C cluster1.has_arm_v8-2=1 \
43 -C cluster1.has_arm_v8-3=1 \
44 -C cluster1.has_arm_v8-4=1 \
45 -C bp.secureflashloader.fname=${BL1_IMG} \
46 -C bp.flashloader0.fname=${FIP_IMG} \
47 --data cluster0.cpu0=${KERNEL}@${KERN_ADDR} \
48 --data cluster0.cpu0=${INITRD}@${INITRD_ADDR} \
49 --data cluster0.cpu0=${DTB_IMG}@${DTB_ADDR} \
50 -C bp.virtioblockdevice.image_path=${DISK_IMG0} \
51 -C bp.smsc_91c111.enabled=1 \
52 -C bp.hostbridge.interfaceName="arm_tap0" \_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerHi Haibo,
Quoted 34 lines · Haibo Xu
Quoted 26 lines · Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
Quoted 6 lines
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
Quoted 7 lines
Hi Marc,
I got a kernel BUG message when booting the L2 guest kernel with the kvmtool on a FVP setup.
Could you help have a look about the BUG message as well as my environment configuration?
I think It probably caused by some local configurations of the FVP setup.
No, this is likely a bug in your L1 guest, which was fixed in -rc3:
2a5f1b67ec57 ("KVM: arm64: Don't access PMCR_EL0 when no PMU is available")
and was found in the exact same circumstances. Alternatively, and if you don't want to change your L1 guest, you can just pass the --pmu option to kvmtool when starting the L1 guest.
Hope this helps,
M.
Signature · 5 lines
Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 47 lines · Marc Zyngier
Hi Haibo,
Quoted 34 lines · Haibo Xu
Quoted 26 lines · Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
Quoted 6 lines
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
Quoted 8 lines
Hi Marc,
I got a kernel BUG message when booting the L2 guest kernel with the kvmtool on a FVP setup.
Could you help have a look about the BUG message as well as my environment configuration?
I think It probably caused by some local configurations of the FVP setup.
Quoted 8 lines
No, this is likely a bug in your L1 guest, which was fixed in -rc3:
2a5f1b67ec57("KVM: arm64: Don't accessPMCR_EL0when no PMU is available")and was found in the exact same circumstances. Alternatively, and if you don't want to change your L1 guest, you can just pass the --pmu option to kvmtool when starting the L1 guest.
After passing --pmu when starting a L1 guest, I can successfully run a L2 guest now!
Thanks so much for the help!
Haibo
Quoted 6 lines
Hope this helps,
M.--
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
On 10/12/2020 15:58, Marc Zyngier wrote:
Hi,
Quoted 1 line
This is a rework of the NV series that I posted 10 months ago[1], as a
....
Quoted 4 lines
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2.
I have updated the branch, as the headers changed as well:
https://gitlab.arm.com/linux-arm/kvmtool/-/commits/nv-5.11-WIP/
Cheers,
Andre
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 32 lines · Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
Quoted 5 lines
And to be clear: although Jintack and Christoffer have written tons of the stuff originaly, I'm the one responsible for breaking it!
[1] https://lore.kernel.org/r/[email protected]
[2]git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git
kvm-arm64/nv-5.11.-WIP
Quoted 1 line
[3]
git://linux-arm.org/kvmtool.gitnv/nv-wip-5.2-rc5
Hi Marc,
I have tried to enable the NV support in Qemu, and now I can successfully boot a L2 guest
in Qemu KVM mode.
This patch series looks good from the Qemu side except for two minor requirements:
(1) Qemu will check whether a feature was supported by the KVM cap when the user tries
to enable it in the command line, so a new capability was prefered for
the NV(KVM_CAP_ARM_NV?).
(2) According to the Documentation/virt/kvm/api.rst
<https://github.com/torvalds/linux/blob/master/Documentation/virt/kvm/api.rst#482-kvm_arm_vcpu_init>,
userspace can call KVM_ARM_VCPU_INIT
multiple times for a given vcpu, but the kvm_vcpu_init_nested() do
have some issue when
called multiple times(please refer to the detailed comments in patch
63)
Regards,
Haibo
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re-send in case the previous email was blocked for the inlined hyper-link.
Hi Marc,
I have tried to enable the NV support in Qemu, and now I can successfully boot a L2 guest
in Qemu KVM mode.
This patch series looks good from the Qemu side except for two minor requirements:
(1) Qemu will check whether a feature was supported by the KVM cap when the user tries
to enable it in the command line, so a new capability was
prefered for the NV(KVM_CAP_ARM_NV?).
(2) According to the Documentation/virt/kvm/api.rst, userspace can call KVM_ARM_VCPU_INIT
multiple times for a given vcpu, but the kvm_vcpu_init_nested()
do have some issue when
called multiple times(please refer to the detailed comments in patch 63)
Regards,
Haibo
Quoted 176 lines · Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
And to be clear: although Jintack and Christoffer have written tons of the stuff originaly, I'm the one responsible for breaking it!
[1] https://lore.kernel.org/r/[email protected]
[2]git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.gitkvm-arm64/nv-5.11.-WIP
[3]git://linux-arm.org/kvmtool.gitnv/nv-wip-5.2-rc5Andre Przywara (1):
KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQChristoffer Dall (15):
KVM: arm64: nv: Introduce nested virtualization VCPU feature
KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
KVM: arm64: nv: Allow userspace to setPSR_MODE_EL2x
KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
KVM: arm64: nv: ResetVMPIDR_EL2andVPIDR_EL2to sane values
KVM: arm64: nv: Handle trapped ERET from virtual EL2
KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
KVM: arm64: nv: Only toggle cache for virtual EL2 whenSCTLR_EL2
changes
KVM: arm64: nv: Implement nested Stage-2 page table walk logic
KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
KVM: arm64: nv:arch_timer: Support hyp timer emulation
KVM: arm64: nv: vgic: Emulate the HW bit in software
KVM: arm64: nv: Add nested GICv3 tracepoints
KVM: arm64: nv: Sync nested timer state with ARMv8.4Jintack Lim (19):
arm64: AddARM64_HAS_NESTED_VIRTcpufeature
KVM: arm64: nv: Handle HCR_EL2.NV system register traps
KVM: arm64: nv: Support virtual EL2 exceptions
KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
KVM: arm64: nv: TrapSPSR_EL1,ELR_EL1andVBAR_EL1from virtual EL2
KVM: arm64: nv: TrapCPACR_EL1access in virtual EL2
KVM: arm64: nv: Handle PSCI call via smc from the guest
KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
KVM: arm64: nv: ConfigureHCR_EL2for nested virtualization
KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
KVM: arm64: nv: Set a handler for the system instruction traps
KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
KVM: arm64: nv: Nested GICv3 SupportMarc Zyngier (31):
KVM: arm64: nv: Add EL2 system registers to vcpu context
KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
KVM: arm64: nv: Handle virtual EL2 registers in
vcpu_read/write_sys_reg()
KVM: arm64: nv: HandleSPSR_EL2specially
KVM: arm64: nv: Handle HCR_EL2.E2H specially
KVM: arm64: nv: Save/Restore vEL2 sysregs
KVM: arm64: nv: Forward debug traps to the nested guest
KVM: arm64: nv: Filter out unsupported features from ID regs
KVM: arm64: nv: Hide RAS from nested guests
KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
KVM: arm64: nv: Handle shadow stage 2 page faults
KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
KVM: arm64: nv: Fold guest'sHCR_EL2configuration into the host's
KVM: arm64: nv: Add handling of EL2-specific timer registers
KVM: arm64: nv: Load timer before the GIC
KVM: arm64: nv: Don't load the GICv4 context on entering a nested
guest
KVM: arm64: nv: Implement maintenance interrupt forwarding
KVM: arm64: nv: Allow userspace to requestKVM_ARM_VCPU_NESTED_VIRT
KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation
KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like
information
KVM: arm64: Allow populating S2 SW bits
KVM: arm64: nv: Tag shadow S2 entries with nested level
KVM: arm64: nv: Add include containing theVNCR_EL2offsets
KVM: arm64: Map VNCR-capable registers to a separate page
KVM: arm64: nv: Move nested vgic state into the sysreg file
KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature
KVM: arm64: nv: Synchronize PSTATE early on exit
KVM: arm64: nv: Allocate VNCR page when required
KVM: arm64: nv: Enable ARMv8.4-NV support
KVM: arm64: nv: Fast-track 'InHost' exception returns
KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests.../admin-guide/kernel-parameters.txt | 4 + .../virt/kvm/devices/arm-vgic-v3.rst | 12 +- arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/esr.h | 6 + arch/arm64/include/asm/kvm_arm.h | 28 +- arch/arm64/include/asm/kvm_asm.h | 4 + arch/arm64/include/asm/kvm_emulate.h | 145 +- arch/arm64/include/asm/kvm_host.h | 175 ++- arch/arm64/include/asm/kvm_hyp.h | 2 + arch/arm64/include/asm/kvm_mmu.h | 17 +- arch/arm64/include/asm/kvm_nested.h | 152 ++ arch/arm64/include/asm/kvm_pgtable.h | 10 + arch/arm64/include/asm/sysreg.h | 104 +- arch/arm64/include/asm/vncr_mapping.h | 73 + arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kernel/cpufeature.c | 35 + arch/arm64/kvm/Makefile | 4 +- arch/arm64/kvm/arch_timer.c | 189 ++- arch/arm64/kvm/arm.c | 34 +- arch/arm64/kvm/at.c | 231 ++++ arch/arm64/kvm/emulate-nested.c | 186 +++ arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/handle_exit.c | 81 +- arch/arm64/kvm/hyp/exception.c | 44 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 31 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 28 +- arch/arm64/kvm/hyp/nvhe/switch.c | 10 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 6 + arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 207 ++- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 +- arch/arm64/kvm/hyp/vhe/tlb.c | 83 ++ arch/arm64/kvm/inject_fault.c | 62 +- arch/arm64/kvm/mmu.c | 183 ++- arch/arm64/kvm/nested.c | 908 ++++++++++++ arch/arm64/kvm/reset.c | 14 +- arch/arm64/kvm/sys_regs.c | 1226 ++++++++++++++++- arch/arm64/kvm/sys_regs.h | 6 + arch/arm64/kvm/trace_arm.h | 65 +- arch/arm64/kvm/vgic/vgic-init.c | 30 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 + arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 240 ++++ arch/arm64/kvm/vgic/vgic-v3.c | 39 +- arch/arm64/kvm/vgic/vgic.c | 44 + arch/arm64/kvm/vgic/vgic.h | 10 + include/kvm/arm_arch_timer.h | 7 + include/kvm/arm_vgic.h | 16 + tools/arch/arm/include/uapi/asm/kvm.h | 1 + 50 files changed, 4890 insertions(+), 160 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/vncr_mapping.h create mode 100644 arch/arm64/kvm/at.c create mode 100644 arch/arm64/kvm/emulate-nested.c create mode 100644 arch/arm64/kvm/nested.c create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c
Quoted 7 lines
--
2.29.2_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Kindly ping!
Quoted 21 lines · Haibo Xu
Re-send in case the previous email was blocked for the inlined hyper-link.
Hi Marc,
I have tried to enable the NV support in Qemu, and now I can successfully boot a L2 guest
in Qemu KVM mode.This patch series looks good from the Qemu side except for two minor requirements:
(1) Qemu will check whether a feature was supported by the KVM cap when the user tries
to enable it in the command line, so a new capability was
prefered for theNV(KVM_CAP_ARM_NV?).
(2) According to theDocumentation/virt/kvm/api.rst, userspace can callKVM_ARM_VCPU_INIT
multiple times for a given vcpu, but thekvm_vcpu_init_nested()
do have some issue when
called multiple times(please refer to the detailed comments in patch 63)
Quoted 187 lines
Regards,
HaiboQuoted 183 lines · Marc Zyngier
This is a rework of the NV series that I posted 10 months ago[1], as a lot of the KVM code has changed since, and the series apply anymore (not that anybody really cares as the the HW is, as usual, made of unobtainium...).
From the previous version:
- Integration with the new page-table code
- New exception injection code
- No more messing with the nVHE code
- No AArch32!!!!
- Rebased on v5.10-rc4 + kvmarm/next for 5.11From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression.
The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it.
For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-).
And to be clear: although Jintack and Christoffer have written tons of the stuff originaly, I'm the one responsible for breaking it!
[1] https://lore.kernel.org/r/[email protected]
[2]git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.gitkvm-arm64/nv-5.11.-WIP
[3]git://linux-arm.org/kvmtool.gitnv/nv-wip-5.2-rc5Andre Przywara (1):
KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQChristoffer Dall (15):
KVM: arm64: nv: Introduce nested virtualization VCPU feature
KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
KVM: arm64: nv: Allow userspace to setPSR_MODE_EL2x
KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
KVM: arm64: nv: ResetVMPIDR_EL2andVPIDR_EL2to sane values
KVM: arm64: nv: Handle trapped ERET from virtual EL2
KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
KVM: arm64: nv: Only toggle cache for virtual EL2 whenSCTLR_EL2
changes
KVM: arm64: nv: Implement nested Stage-2 page table walk logic
KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
KVM: arm64: nv:arch_timer: Support hyp timer emulation
KVM: arm64: nv: vgic: Emulate the HW bit in software
KVM: arm64: nv: Add nested GICv3 tracepoints
KVM: arm64: nv: Sync nested timer state with ARMv8.4Jintack Lim (19):
arm64: AddARM64_HAS_NESTED_VIRTcpufeature
KVM: arm64: nv: Handle HCR_EL2.NV system register traps
KVM: arm64: nv: Support virtual EL2 exceptions
KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
KVM: arm64: nv: TrapSPSR_EL1,ELR_EL1andVBAR_EL1from virtual EL2
KVM: arm64: nv: TrapCPACR_EL1access in virtual EL2
KVM: arm64: nv: Handle PSCI call via smc from the guest
KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
KVM: arm64: nv: ConfigureHCR_EL2for nested virtualization
KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
KVM: arm64: nv: Set a handler for the system instruction traps
KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
KVM: arm64: nv: Nested GICv3 SupportMarc Zyngier (31):
KVM: arm64: nv: Add EL2 system registers to vcpu context
KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
KVM: arm64: nv: Handle virtual EL2 registers in
vcpu_read/write_sys_reg()
KVM: arm64: nv: HandleSPSR_EL2specially
KVM: arm64: nv: Handle HCR_EL2.E2H specially
KVM: arm64: nv: Save/Restore vEL2 sysregs
KVM: arm64: nv: Forward debug traps to the nested guest
KVM: arm64: nv: Filter out unsupported features from ID regs
KVM: arm64: nv: Hide RAS from nested guests
KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
KVM: arm64: nv: Handle shadow stage 2 page faults
KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
KVM: arm64: nv: Fold guest'sHCR_EL2configuration into the host's
KVM: arm64: nv: Add handling of EL2-specific timer registers
KVM: arm64: nv: Load timer before the GIC
KVM: arm64: nv: Don't load the GICv4 context on entering a nested
guest
KVM: arm64: nv: Implement maintenance interrupt forwarding
KVM: arm64: nv: Allow userspace to requestKVM_ARM_VCPU_NESTED_VIRT
KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation
KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like
information
KVM: arm64: Allow populating S2 SW bits
KVM: arm64: nv: Tag shadow S2 entries with nested level
KVM: arm64: nv: Add include containing theVNCR_EL2offsets
KVM: arm64: Map VNCR-capable registers to a separate page
KVM: arm64: nv: Move nested vgic state into the sysreg file
KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature
KVM: arm64: nv: Synchronize PSTATE early on exit
KVM: arm64: nv: Allocate VNCR page when required
KVM: arm64: nv: Enable ARMv8.4-NV support
KVM: arm64: nv: Fast-track 'InHost' exception returns
KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests.../admin-guide/kernel-parameters.txt | 4 + .../virt/kvm/devices/arm-vgic-v3.rst | 12 +- arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/esr.h | 6 + arch/arm64/include/asm/kvm_arm.h | 28 +- arch/arm64/include/asm/kvm_asm.h | 4 + arch/arm64/include/asm/kvm_emulate.h | 145 +- arch/arm64/include/asm/kvm_host.h | 175 ++- arch/arm64/include/asm/kvm_hyp.h | 2 + arch/arm64/include/asm/kvm_mmu.h | 17 +- arch/arm64/include/asm/kvm_nested.h | 152 ++ arch/arm64/include/asm/kvm_pgtable.h | 10 + arch/arm64/include/asm/sysreg.h | 104 +- arch/arm64/include/asm/vncr_mapping.h | 73 + arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kernel/cpufeature.c | 35 + arch/arm64/kvm/Makefile | 4 +- arch/arm64/kvm/arch_timer.c | 189 ++- arch/arm64/kvm/arm.c | 34 +- arch/arm64/kvm/at.c | 231 ++++ arch/arm64/kvm/emulate-nested.c | 186 +++ arch/arm64/kvm/guest.c | 6 + arch/arm64/kvm/handle_exit.c | 81 +- arch/arm64/kvm/hyp/exception.c | 44 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 31 +- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 28 +- arch/arm64/kvm/hyp/nvhe/switch.c | 10 +- arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 2 +- arch/arm64/kvm/hyp/pgtable.c | 6 + arch/arm64/kvm/hyp/vgic-v3-sr.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 207 ++- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 125 +- arch/arm64/kvm/hyp/vhe/tlb.c | 83 ++ arch/arm64/kvm/inject_fault.c | 62 +- arch/arm64/kvm/mmu.c | 183 ++- arch/arm64/kvm/nested.c | 908 ++++++++++++ arch/arm64/kvm/reset.c | 14 +- arch/arm64/kvm/sys_regs.c | 1226 ++++++++++++++++- arch/arm64/kvm/sys_regs.h | 6 + arch/arm64/kvm/trace_arm.h | 65 +- arch/arm64/kvm/vgic/vgic-init.c | 30 + arch/arm64/kvm/vgic/vgic-kvm-device.c | 22 + arch/arm64/kvm/vgic/vgic-nested-trace.h | 137 ++ arch/arm64/kvm/vgic/vgic-v3-nested.c | 240 ++++ arch/arm64/kvm/vgic/vgic-v3.c | 39 +- arch/arm64/kvm/vgic/vgic.c | 44 + arch/arm64/kvm/vgic/vgic.h | 10 + include/kvm/arm_arch_timer.h | 7 + include/kvm/arm_vgic.h | 16 + tools/arch/arm/include/uapi/asm/kvm.h | 1 + 50 files changed, 4890 insertions(+), 160 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/vncr_mapping.h create mode 100644 arch/arm64/kvm/at.c create mode 100644 arch/arm64/kvm/emulate-nested.c create mode 100644 arch/arm64/kvm/nested.c create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c--
2.29.2_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerQuoted 18 lines · Haibo Xu
Kindly ping!
Quoted 14 lines · Haibo Xu
Re-send in case the previous email was blocked for the inlined hyper-link.
Hi Marc,
I have tried to enable the NV support in Qemu, and now I can successfully boot a L2 guest
in Qemu KVM mode.This patch series looks good from the Qemu side except for two minor requirements:
(1) Qemu will check whether a feature was supported by the KVM cap when the user tries to enable it in the command line, so a new capability was prefered for theNV(KVM_CAP_ARM_NV?).
I have added KVM_CAP_ARM_EL2 (rather than NV) to that effect.
Quoted 4 lines
Quoted 4 lines
(2) According to the
Documentation/virt/kvm/api.rst, userspace can callKVM_ARM_VCPU_INITmultiple times for a given vcpu, but thekvm_vcpu_init_nested()do have some issue when called multiple times(please refer to the detailed comments in patch 63)
This is now fixed, I believe.
I have pushed out a branch [1] that addresses all the reported issues, though it currently lack some testing. Please let me know if it works for you.
Thanks,
M.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-5.12-WIP
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Quoted 29 lines · Marc Zyngier
Quoted 18 lines · Haibo Xu
Kindly ping!
Quoted 14 lines · Haibo Xu
Re-send in case the previous email was blocked for the inlined hyper-link.
Hi Marc,
I have tried to enable the NV support in Qemu, and now I can successfully boot a L2 guest
in Qemu KVM mode.This patch series looks good from the Qemu side except for two minor requirements:
(1) Qemu will check whether a feature was supported by the KVM cap when the user tries to enable it in the command line, so a new capability was prefered for theNV(KVM_CAP_ARM_NV?).I have added
KVM_CAP_ARM_EL2(rather than NV) to that effect.Quoted 4 lines
Quoted 4 lines
(2) According to the
Documentation/virt/kvm/api.rst, userspace can callKVM_ARM_VCPU_INITmultiple times for a given vcpu, but thekvm_vcpu_init_nested()do have some issue when called multiple times(please refer to the detailed comments in patch 63)
Quoted 6 lines
This is now fixed, I believe.
I have pushed out a branch [1] that addresses all the reported issues, though it currently lack some testing. Please let me know if it works for you.
Hi Marc,
I have verified the fix, and it works well with Qemu.
thanks,
Haibo
Quoted 8 lines
Thanks,
M.[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-5.12-WIP
--
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm